ONNX How To ... Create and use a generated ONNX model

Unity Engine
, , ,
1

I created a ONNX model from the FaceBook/MusicGen-Small model at HuggingFace using optimum-cli in a Terminal.

This resulted in a ONNX folder with the following .onnx files:

build_delay_pattern_mask
decoder_model
decoder_model_merged
decoder_with_past_model
encodec_decode
text_encoder

After that I created the following code to load the model and set a prompt.

    [RequireComponent(typeof(AudioSource))]
    public class MusicGenExample : MonoBehaviour
    {
        [SerializeField]
        private ModelAsset _ModelAsset;

        private AudioSource _audioSource;
        private Model _model;
        private string _prompt;

        private void Awake()
        {
            Initialize();
        }

        private void Initialize()
        {
            _model = ModelLoader.Load(_ModelAsset);
            _prompt = "A happy, upbeat pop song";
        }
    }

But now I am lost on how to proceed further. How can I now add the prompt to the model and generate the desired output? And which of the generated .onnx files should I use to generate the output (a song using the desired prompt). So basically I’d like to know how I can figure out what .onnx file(s) I need to use, how to access them and what input and output parameters I should use. If I can get help to get this one running, converting other models from Hugging Face should hopefully be easier, once I understand the workflow. Thanks in advance!

1 Like
2

You’ll have to check the python code of how they chain the models together I’m afraid :confused:
In the meantime, you could try this one

1 Like
3

Interesting, I had not seen the .sentis extension before. Is this something I could do myself, creating .sentis versions of models, or is this done by Unity employees only? If so, is there a manual/tutorial somewhere, explaining how I could do that?

1 Like
4

You can do it yourself! Check out the “Serialize a model” page in the docs.

https://docs.unity3d.com/Packages/com.unity.sentis@2.1/manual/serialize-a-model.html

You can serialize a model from code using the ModelWriter.Save(path, model); method. This lets you serialize a model you have built or adapted with the functional api, or a quantized model.

2 Likes
5

To clarify a bit:

  • .sentis file is the serialized format for a neural network asset. It is how we handle serliazliation in unity and allows you to bundle it as a regular asset in a assetbundle and all that.
    Having a serialized format allows to skip a lot of data loading and processing linked to unity import pipeline. Which makes it ideal for assetbundle/streaming. Similar to how jpeg are not loaded dynamically in asset bundles, but the serialized BTC textures.
  • how to create a .sentis, you can follow what @Giles-Coope said.
  • You can also use the functional API to build your own graph of operations and serialize that.
    (cf ~\Samples\Use the functional API with an existing model)
1 Like