Running a PyTorch (or even a TensorFlow) Model on Unity MLAgent

Unity Engine
,
1

MLAgents Version: 0.15.1

I’ve begun to dive into the unknown territory of trying to get a PyTorch model into an MLAgent within Unity. Looking at whatever forum threads I can find it seems that converting the model into a Barracuda model might do the trick. I was successful in converting PyTorch > Onnx > Barracuda and in the inspector, the model looks like this

5883374--626795--upload_2020-5-21_22-9-14.png

but I am not able to drag the model(“brain”) into the MLAgent via inspector. If I try to drag the model into the inspector I see the following error log.

NullReferenceException: Object reference not set to an instance of an object
MLAgents.Inference.BarracudaModelParamLoader.CheckModel (Barracuda.Model model, MLAgents.Policies.BrainParameters brainParameters, MLAgents.Sensors.SensorComponent[] sensorComponents, MLAgents.Policies.BehaviorType behaviorType) (at C:/Users/Janimator0/Desktop/UnityMLAgent_Project/ml-agents/com.unity.ml-agents/Runtime/Inference/BarracudaModelParamLoader.cs:149)
MLAgents.Editor.BehaviorParametersEditor.DisplayFailedModelChecks () (at C:/Users/Janimator0/Desktop/UnityMLAgent_Project/ml-agents/com.unity.ml-agents/Editor/BehaviorParametersEditor.cs:108)
MLAgents.Editor.BehaviorParametersEditor.OnInspectorGUI () (at C:/Users/Janimator0/Desktop/UnityMLAgent_Project/ml-agents/com.unity.ml-agents/Editor/BehaviorParametersEditor.cs:68)
UnityEditor.UIElements.InspectorElement+<>c__DisplayClass55_0.<CreateIMGUIInspectorFromEditor>b__0 () (at <887a438259064e89922419bc7b8810ab>:0)
UnityEngine.GUIUtility:ProcessEvent(Int32, IntPtr)

I’ve read on another forum something about naming the consts correctly might do the trick but I’m unsure what the steps are for that? Can't load external model to agent (from .pth convert to .onnx and then to .nn) · Issue #3398 · Unity-Technologies/ml-agents · GitHub ← mentioned in this thread.

My questions are is it at all possible to get my PyTorch model to work on an MLAgent? Is it even possible to get a TensorFlow model to work on a MLAgent? What can be done to achieve this? How can I ensure that the “consts” are named correctly?

2

Hi!!!Do you have find the solution ???

3

sorryy for the mistake…do you have found the solution??

4

Interested in this thread as well.

5

We’ve clarified our policy around this a bit here. In short, you’re welcome to try using models that you trained outside of ml-agents, but we’re not able to provide much support for you if you do.

In this case, that part of the code is looking for some constants stored in tensors with specific names. Since the original post, the error handling there has been improved a bit.

The naming conventions for the expected tensors isn’t documented, but you can at least find the names in TensorNames.cs

1 Like