Implementing custom code for model training in ML-Agents package

Unity Engine
, ,
1

I have a model named PredNet, which is a deep convolutional recurrent neural network, and I wish to actually use the functionalities of the CameraSensor agent from ML-Agents to train this model of mine. I know that the ML-Agents package is solely for using RL algorithms, however, I was curious whether it provides any functionality to add your own custom training algorithm or model to train your Unity simulation on. If not, are there any other Unity packages that might help me with this task? Currently, I have the model (PredNet) fully coded in Python too, so additionally, is there any API/package by Unity to run external, Python-based models/scripts?

2

The ml agents package is primarily targeting reinforcement learning tasks as you’ve identified and would probably not be suited for what you’re trying to do. It is absolutely possible to use a custom algorithm with ml agents but I think you would be swimming against the current.

Have you checked out unity’s perception package? GitHub - Unity-Technologies/com.unity.perception: Perception toolkit for sim2real training and validation in Unity

I feel like that would give you a better starting point to train from unity output without having to deal with RL by default. The marketing for unity perception makes it look like just a segmentation tool but it has alot more capability than just that. Here’s an example of using it for robotic pose estimation - GitHub - Unity-Technologies/Robotics-Object-Pose-Estimation: A complete end-to-end demonstration in which we collect training data in Unity and use that data to train a deep neural network to predict the pose of a cube. This model is then deployed in a simulated robotic pick-and-place task.

1 Like
3

Thanks a lot for the suggestion! While it does seem really promising, in my project I don’t have a lot of disk space to work with, so I was hoping to use a more “online” training approach, in which I send the data from my running Unity simulation to my model, use it for training simultaneously, and then get the output and basically discard the data.