Barracuda for RL/ML Chess Bot

For context, I am trying to build a chess bot that uses a both RL(Reinforcement Learning) and ML(Machine Learning, training on dataset of games from the better chess players) models with the ultimate goal of comparing them. I made a custom Chess app complete with its own GUI and sprites, etc and I want to add these bots to be able to play against each other as well as human players.

I found out about the Barracuda library which seemed the perfect tool for the job. I decided to start with the RL bot first (it seemed like the harder of the two). However, I’ve run into a few issues.

What I did is I have this python code which I used to create a “simple” RL model for playing chess

import gym
from gym import spaces
import numpy as np
import chess
from stable_baselines3 import PPO
import torch
import onnx

# Custom Chess Environment
class ChessEnv(gym.Env):
    def __init__(self):
        super(ChessEnv, self).__init__()
        self.board = chess.Board()
        self.action_space = spaces.Discrete(4672)  # Total possible moves in chess
        self.observation_space = spaces.Box(low=0, high=1, shape=(8, 8, 12), dtype=np.float32)

    def reset(self):
        self.board.reset()
        return self._get_observation()  # Return the numerical representation of the board

    def step(self, action):
        legal_moves = list(self.board.legal_moves)
        if action >= len(legal_moves):
            action = action % len(legal_moves)  # Map action to a valid legal move
        self.board.push(legal_moves[action])
        done = self.board.is_game_over()
        reward = 1 if self.board.is_checkmate() else 0  # Reward for checkmate (you can adjust logic here)
        return self._get_observation(), reward, done, {}

    def render(self, mode="human"):
        print(self.board)  # Print the board for a human-readable representation

    def _get_observation(self):
        # Convert the board to a numerical representation
        board_state = np.zeros((8, 8, 12), dtype=np.float32)
        piece_map = self.board.piece_map()
        for square, piece in piece_map.items():
            piece_type = piece.piece_type - 1
            color = int(piece.color)
            row, col = divmod(square, 8)
            board_state[row, col, piece_type + 6 * color] = 1
        return board_state

# Create the environment
env = ChessEnv()

# Define the RL model using PPO
model = PPO("MlpPolicy", env, verbose=1)

# Train the model
games = 5  # small number for testing
model.learn(total_timesteps=games)  # will increase timesteps for better training

# Save the trained model
model.save("chess_rl_bot")

# Load the trained model (for later use)
model = PPO.load("chess_rl_bot")

# Extract the policy (neural network)
policy = model.policy

# Create a dummy input matching the model’s observation space
dummy_input = torch.zeros(1, *policy.observation_space.shape)  # Adjust shape if needed

# Export the model to ONNX format with a specific opset version
onnx_model_path = "ppo_chess_model.onnx"
torch.onnx.export(policy, dummy_input, onnx_model_path, opset_version=9)  # Use opset_version=9

# Save the ONNX model
print(f"ONNX model saved to {onnx_model_path}")

I save the output model as a .onnx file which I add to my Unity under Assets/Models folder I created the Models folder. So would be something like Assets/Models/rlmodel.onnx

I try adding the model and get these errots

OnnxImportException: Unknown type ReduceLogSumExp encountered while parsing layer /ReduceLogSumExp_output_0.
Unity.Barracuda.ONNX.ONNXModelConverter.Err (Unity.Barracuda.Model model, System.String layerName, System.String message, System.String extendedMessage, System.String debugMessage) (at Library/PackageCache/com.unity.barracuda@2.0.0/Barracuda/Runtime/ONNX/ONNXModelConverter.cs:3298)

Asset import failed, “Assets/Models/ppo_chess_model.onnx” > OnnxImportException: Unknown type ReduceLogSumExp encountered while parsing layer /ReduceLogSumExp_output_0. Unity.Barracuda.ONNX.ONNXModelConverter.Err (Unity.Barracuda.Model model, System.String layerName, System.String message, System.String extendedMessage, System.String debugMessage) (at Library/PackageCache/com.unity.barracuda@2.0.0/Barracuda/Runtime/ONNX/ONNXModelConverter.cs:3298)
Unity.Barracuda.ONNX.ONNXModelConverter.ConvertOnnxModel (Onnx.ModelProto onnxModel) (at Library/PackageCache/com.unity.barracuda@2.0.0/Barracuda/Runtime/ONNX/ONNXModelConverter.cs:2818)
Unity.Barracuda.ONNX.ONNXModelConverter.Convert (Google.Protobuf.CodedInputStream inputStream) (at Library/PackageCache/com.unity.barracuda@2.0.0/Barracuda/Runtime/ONNX/ONNXModelConverter.cs:155)
Unity.Barracuda.ONNX.ONNXModelConverter.Convert (System.String filePath) (at Library/PackageCache/com.unity.barracuda@2.0.0/Barracuda/Runtime/ONNX/ONNXModelConverter.cs:83) Unity.Barracuda.ONNXModelImporter.OnImportAsset (UnityEditor.AssetImporters.AssetImportContext ctx) (at Library/PackageCache/com.unity.barracuda@2.0.0/Barracuda/Editor/ONNXModelImporter.cs:58)
UnityEditor.AssetImporters.ScriptedImporter.GenerateAssetData (UnityEditor.AssetImporters.AssetImportContext ctx) (at <06837d428b6d46f581d93f9597703696>:0)
UnityEditorInternal.InternalEditorUtility:ProjectWindowDrag(HierarchyProperty, Boolean)
UnityEngine.GUIUtility:ProcessEvent(Int32, IntPtr, Boolean&)

And then they go away BUT I get this error when I try to call the bot

NotSupportedException: Format version not supported: 242
Unity.Barracuda.ModelLoader.Load (System.IO.BinaryReader fileReader, System.Boolean verbose, System.Boolean applyPatching, System.Boolean skipWeights) (at Library/PackageCache/com.unity.barracuda@2.0.0/Barracuda/Runtime/Core/ModelLoader.cs:111)
Unity.Barracuda.ModelLoader.Load (System.Byte stream, System.Boolean verbose, System.Boolean skipWeights) (at Library/PackageCache/com.unity.barracuda@2.0.0/Barracuda/Runtime/Core/ModelLoader.cs:68)
WhiteState…ctor (System.String playerName, System.Boolean isWhite) (at Assets/Scripts/Players/Bots/Trained/White.cs:20)

This is the Unity C# Code for the Bot. It does inherit from some other things but I believe the other files arent necessary because other bots(basic algorithm based bots) inherit from those other files and have been working ok

using UnityEngine;
using Unity.Barracuda;
using System.IO;

public class White : Bot {

}

public class WhiteState : BotState
{
    private Model model;
    private IWorker worker;

    private string modelFilePath = "C:/Users/cruzmart/Daniel/web/Chess-Unity/Assets/Models/ppo_chess_model.onnx";  // Update this path to match your model file location
    private string outputTensorName = "action";  // Replace with your model's output tensor name

    public WhiteState(string playerName, bool isWhite) : base(playerName, isWhite)
    {
        // Load the model from the file path and initialize a worker to run inference
        model = ModelLoader.Load(File.ReadAllBytes(modelFilePath));
        worker = WorkerFactory.CreateWorker(WorkerFactory.Type.ComputePrecompiled, model);
    }

    public WhiteState(WhiteState original) : base(original) { }

    public override PlayerState Clone() => new WhiteState(this);

    public override Vector2Int GetMove()
    {
        // Get the observation from the game (this will be a numerical representation of the board state)
        float[] observation = GetCurrentBoardObservation();

        // Create a tensor from the observation array
        Tensor inputTensor = new Tensor(1, 8, 8, 12, observation);
        worker.Execute(inputTensor);

        // Extract the action from the output tensor
        Tensor outputTensor = worker.PeekOutput(outputTensorName);

        // Convert the model's output to a move (this depends on how the model is structured)
        int action = outputTensor.ArgMax()[0];  // Assuming the action is a discrete move, you may need to adjust this
        Vector2Int move = ConvertActionToMove(action);

        // Clean up tensors
        inputTensor.Dispose();
        outputTensor.Dispose();

        return move;
    }

    Vector2Int ConvertActionToMove(int action)
    {
        // Convert the action (a move number) to a chess move
        int fromSquare = action / 64;
        int toSquare = action % 64;
        return new Vector2Int(fromSquare, toSquare);
    }

    private float[] GetCurrentBoardObservation()
    {
        // Convert the board state to a 768-element array (8x8x12)
        float[] observation = new float[768];
        char[,] board = CurrentGame.StringBoard();

        for (int y = 0; y < 8; y++)
        {
            for (int x = 0; x < 8; x++)
            {
                char piece = board[y, x];
                int channel = GetChannelForPiece(piece);
                if (channel != -1)
                {
                    observation[y * 96 + x * 12 + channel] = 1f;
                }
            }
        }

        return observation;
    }

    private int GetChannelForPiece(char piece)
    {
        switch (piece)
        {
            case 'P': return 0;  // White Pawn
            case 'N': return 1;  // White Knight
            case 'B': return 2;  // White Bishop
            case 'R': return 3;  // White Rook
            case 'Q': return 4;  // White Queen
            case 'K': return 5;  // White King
            case 'p': return 6;  // Black Pawn
            case 'n': return 7;  // Black Knight
            case 'b': return 8;  // Black Bishop
            case 'r': return 9;  // Black Rook
            case 'q': return 10; // Black Queen
            case 'k': return 11; // Black King
            default: return -1;  // Empty square
        }
    }

    void OnDestroy()
    {
        // Clean up the worker when done
        worker.Dispose();
    }
}

The file is basically supposed to get the trained RL/ML model, convert to the Model type using Barracuda, and then read input(observations, I basically convert the current board to a form the model can read) and give output in the form of best moves.

I have no idea if the actual code is working as intended but I cant even check this because I am stuck at the above error(s).

System info:
the info for how the model was made:

  • OS: Linux-5.15.0-130-generic-x86_64-with-glibc2.29 # 140~20.04.1-Ubuntu SMP Wed Dec 18 21:35:34 UTC 2024
  • Python: 3.8.10
  • Stable-Baselines3: 2.4.1
  • PyTorch: 2.4.1+cu121
  • GPU Enabled: False
  • Numpy: 1.24.4
  • Cloudpickle: 3.1.1
  • Gymnasium: 1.0.0
  • OpenAI Gym: 0.26.2

General

  • Unity 2021.3.33f1 LTS
  • Windows(The model was created using Linux system but I am working on a Windows. I dont think this should be an issue but I could be wrong.)

Does anyone have any experience with this/can offer some knowledge on either what I can do or if there are other alternative for integrating RL and/or ML models into this project?

Thank you for reading all that and for your time.

Nice project!
I recommend that you upgrade to Unity 6 with Sentis 2.1.
Barracuda has been renamed Sentis for over a year now, and now supports more operators. For instance in your log you have “Unknown type ReduceLogSumExp”, which is now supported.
You might want to look at Boardgame AI Sample for an other example of a board game.
Good luck!

1 Like