Ideas how to make ML-agents easier

This is a early morning thought :sunrise_over_mountains: how to make ML-agents a bit easier to use.

The idea would be to by-pass any use of python like this:

(0) Install ML-agents package
(1) Design your model in the usual way inside Unity
(2) Click “generate model”
(3) It uploads the details (inputs/outputs/model-type/etc.) to the cloud server and sends back a training ONNX back to Unity.
(4) Use this training ONNX inside Unity to train your model to get the weights using Sentis. Could even do this at run-time.
(5) Save the model with weights into an asset.

Therefor you wouldn’t have to mess around with installing the correct version of python/torch etc. In the future it’d be nice to skip step 3 but that would require backprop inside unity which is possible but a lot of work I expect. (Unless ML-agents just has a fixed set of models in which case that wouldn’t be needed). Maybe even training could be done in the cloud but that might require uploading your whole game there.

Anyone got any other thoughts? Is anything holding you back from using ML-agents?

1 Like