For context, I am trying to build a chess bot that uses a both RL(Reinforcement Learning) and ML(Machine Learning, training on dataset of games from the better chess players) models with the ultimate goal of comparing them. I made a custom Chess app complete with its own GUI and sprites, etc and I want to add these bots to be able to play against each other as well as human players.
I found out about the Barracuda library which seemed the perfect tool for the job. I decided to start with the RL bot first (it seemed like the harder of the two). However, I’ve run into a few issues.
What I did is I have this python code which I used to create a “simple” RL model for playing chess
import gym
from gym import spaces
import numpy as np
import chess
from stable_baselines3 import PPO
import torch
import onnx
# Custom Chess Environment
class ChessEnv(gym.Env):
def __init__(self):
super(ChessEnv, self).__init__()
self.board = chess.Board()
self.action_space = spaces.Discrete(4672) # Total possible moves in chess
self.observation_space = spaces.Box(low=0, high=1, shape=(8, 8, 12), dtype=np.float32)
def reset(self):
self.board.reset()
return self._get_observation() # Return the numerical representation of the board
def step(self, action):
legal_moves = list(self.board.legal_moves)
if action >= len(legal_moves):
action = action % len(legal_moves) # Map action to a valid legal move
self.board.push(legal_moves[action])
done = self.board.is_game_over()
reward = 1 if self.board.is_checkmate() else 0 # Reward for checkmate (you can adjust logic here)
return self._get_observation(), reward, done, {}
def render(self, mode="human"):
print(self.board) # Print the board for a human-readable representation
def _get_observation(self):
# Convert the board to a numerical representation
board_state = np.zeros((8, 8, 12), dtype=np.float32)
piece_map = self.board.piece_map()
for square, piece in piece_map.items():
piece_type = piece.piece_type - 1
color = int(piece.color)
row, col = divmod(square, 8)
board_state[row, col, piece_type + 6 * color] = 1
return board_state
# Create the environment
env = ChessEnv()
# Define the RL model using PPO
model = PPO("MlpPolicy", env, verbose=1)
# Train the model
games = 5 # small number for testing
model.learn(total_timesteps=games) # will increase timesteps for better training
# Save the trained model
model.save("chess_rl_bot")
# Load the trained model (for later use)
model = PPO.load("chess_rl_bot")
# Extract the policy (neural network)
policy = model.policy
# Create a dummy input matching the model’s observation space
dummy_input = torch.zeros(1, *policy.observation_space.shape) # Adjust shape if needed
# Export the model to ONNX format with a specific opset version
onnx_model_path = "ppo_chess_model.onnx"
torch.onnx.export(policy, dummy_input, onnx_model_path, opset_version=9) # Use opset_version=9
# Save the ONNX model
print(f"ONNX model saved to {onnx_model_path}")
I save the output model as a .onnx file which I add to my Unity under Assets/Models folder I created the Models folder. So would be something like Assets/Models/rlmodel.onnx
I try adding the model and get these errots
OnnxImportException: Unknown type ReduceLogSumExp encountered while parsing layer /ReduceLogSumExp_output_0.
Unity.Barracuda.ONNX.ONNXModelConverter.Err (Unity.Barracuda.Model model, System.String layerName, System.String message, System.String extendedMessage, System.String debugMessage) (at Library/PackageCache/com.unity.barracuda@2.0.0/Barracuda/Runtime/ONNX/ONNXModelConverter.cs:3298)
Asset import failed, “Assets/Models/ppo_chess_model.onnx” > OnnxImportException: Unknown type ReduceLogSumExp encountered while parsing layer /ReduceLogSumExp_output_0. Unity.Barracuda.ONNX.ONNXModelConverter.Err (Unity.Barracuda.Model model, System.String layerName, System.String message, System.String extendedMessage, System.String debugMessage) (at Library/PackageCache/com.unity.barracuda@2.0.0/Barracuda/Runtime/ONNX/ONNXModelConverter.cs:3298)
Unity.Barracuda.ONNX.ONNXModelConverter.ConvertOnnxModel (Onnx.ModelProto onnxModel) (at Library/PackageCache/com.unity.barracuda@2.0.0/Barracuda/Runtime/ONNX/ONNXModelConverter.cs:2818)
Unity.Barracuda.ONNX.ONNXModelConverter.Convert (Google.Protobuf.CodedInputStream inputStream) (at Library/PackageCache/com.unity.barracuda@2.0.0/Barracuda/Runtime/ONNX/ONNXModelConverter.cs:155)
Unity.Barracuda.ONNX.ONNXModelConverter.Convert (System.String filePath) (at Library/PackageCache/com.unity.barracuda@2.0.0/Barracuda/Runtime/ONNX/ONNXModelConverter.cs:83) Unity.Barracuda.ONNXModelImporter.OnImportAsset (UnityEditor.AssetImporters.AssetImportContext ctx) (at Library/PackageCache/com.unity.barracuda@2.0.0/Barracuda/Editor/ONNXModelImporter.cs:58)
UnityEditor.AssetImporters.ScriptedImporter.GenerateAssetData (UnityEditor.AssetImporters.AssetImportContext ctx) (at <06837d428b6d46f581d93f9597703696>:0)
UnityEditorInternal.InternalEditorUtility:ProjectWindowDrag(HierarchyProperty, Boolean)
UnityEngine.GUIUtility:ProcessEvent(Int32, IntPtr, Boolean&)
And then they go away BUT I get this error when I try to call the bot
NotSupportedException: Format version not supported: 242
Unity.Barracuda.ModelLoader.Load (System.IO.BinaryReader fileReader, System.Boolean verbose, System.Boolean applyPatching, System.Boolean skipWeights) (at Library/PackageCache/com.unity.barracuda@2.0.0/Barracuda/Runtime/Core/ModelLoader.cs:111)
Unity.Barracuda.ModelLoader.Load (System.Byte stream, System.Boolean verbose, System.Boolean skipWeights) (at Library/PackageCache/com.unity.barracuda@2.0.0/Barracuda/Runtime/Core/ModelLoader.cs:68)
WhiteState…ctor (System.String playerName, System.Boolean isWhite) (at Assets/Scripts/Players/Bots/Trained/White.cs:20)
This is the Unity C# Code for the Bot. It does inherit from some other things but I believe the other files arent necessary because other bots(basic algorithm based bots) inherit from those other files and have been working ok
using UnityEngine;
using Unity.Barracuda;
using System.IO;
public class White : Bot {
}
public class WhiteState : BotState
{
private Model model;
private IWorker worker;
private string modelFilePath = "C:/Users/cruzmart/Daniel/web/Chess-Unity/Assets/Models/ppo_chess_model.onnx"; // Update this path to match your model file location
private string outputTensorName = "action"; // Replace with your model's output tensor name
public WhiteState(string playerName, bool isWhite) : base(playerName, isWhite)
{
// Load the model from the file path and initialize a worker to run inference
model = ModelLoader.Load(File.ReadAllBytes(modelFilePath));
worker = WorkerFactory.CreateWorker(WorkerFactory.Type.ComputePrecompiled, model);
}
public WhiteState(WhiteState original) : base(original) { }
public override PlayerState Clone() => new WhiteState(this);
public override Vector2Int GetMove()
{
// Get the observation from the game (this will be a numerical representation of the board state)
float[] observation = GetCurrentBoardObservation();
// Create a tensor from the observation array
Tensor inputTensor = new Tensor(1, 8, 8, 12, observation);
worker.Execute(inputTensor);
// Extract the action from the output tensor
Tensor outputTensor = worker.PeekOutput(outputTensorName);
// Convert the model's output to a move (this depends on how the model is structured)
int action = outputTensor.ArgMax()[0]; // Assuming the action is a discrete move, you may need to adjust this
Vector2Int move = ConvertActionToMove(action);
// Clean up tensors
inputTensor.Dispose();
outputTensor.Dispose();
return move;
}
Vector2Int ConvertActionToMove(int action)
{
// Convert the action (a move number) to a chess move
int fromSquare = action / 64;
int toSquare = action % 64;
return new Vector2Int(fromSquare, toSquare);
}
private float[] GetCurrentBoardObservation()
{
// Convert the board state to a 768-element array (8x8x12)
float[] observation = new float[768];
char[,] board = CurrentGame.StringBoard();
for (int y = 0; y < 8; y++)
{
for (int x = 0; x < 8; x++)
{
char piece = board[y, x];
int channel = GetChannelForPiece(piece);
if (channel != -1)
{
observation[y * 96 + x * 12 + channel] = 1f;
}
}
}
return observation;
}
private int GetChannelForPiece(char piece)
{
switch (piece)
{
case 'P': return 0; // White Pawn
case 'N': return 1; // White Knight
case 'B': return 2; // White Bishop
case 'R': return 3; // White Rook
case 'Q': return 4; // White Queen
case 'K': return 5; // White King
case 'p': return 6; // Black Pawn
case 'n': return 7; // Black Knight
case 'b': return 8; // Black Bishop
case 'r': return 9; // Black Rook
case 'q': return 10; // Black Queen
case 'k': return 11; // Black King
default: return -1; // Empty square
}
}
void OnDestroy()
{
// Clean up the worker when done
worker.Dispose();
}
}
The file is basically supposed to get the trained RL/ML model, convert to the Model type using Barracuda, and then read input(observations, I basically convert the current board to a form the model can read) and give output in the form of best moves.
I have no idea if the actual code is working as intended but I cant even check this because I am stuck at the above error(s).
System info:
the info for how the model was made:
- OS: Linux-5.15.0-130-generic-x86_64-with-glibc2.29 # 140~20.04.1-Ubuntu SMP Wed Dec 18 21:35:34 UTC 2024
- Python: 3.8.10
- Stable-Baselines3: 2.4.1
- PyTorch: 2.4.1+cu121
- GPU Enabled: False
- Numpy: 1.24.4
- Cloudpickle: 3.1.1
- Gymnasium: 1.0.0
- OpenAI Gym: 0.26.2
General
- Unity 2021.3.33f1 LTS
- Windows(The model was created using Linux system but I am working on a Windows. I dont think this should be an issue but I could be wrong.)
Does anyone have any experience with this/can offer some knowledge on either what I can do or if there are other alternative for integrating RL and/or ML models into this project?
Thank you for reading all that and for your time.