Help with modifying model input in Unity Inference Engine

AI & Machine Learning AI Inference Engine
, ,
1

Hi, I’m just getting started with the Inference Engine 2.2.0 and ran into an issue. I’m trying to use a movement model from Kaggle called movenet, which predicts human joint positions from an image. I’ve converted the model to ONNX, and its input shape is (1, 192, 192, 3) — where the last dimension is the RGB channels, and the values are in int format (0–255).

However, Unity’s Inference Engine expects input tensors to meet the following criteria:

  • The data type must be float.
  • The layout must be NCHW (batch, channels, height, width). For example, 1 × 3 × 192 × 192 represents a single RGB image.

How can I modify my model so it:

  1. Accepts floats instead of ints as input
  2. Reorders the input shape from NHWC to NCHW

I’ve only found an example in the official documentation for modifying the output of a model — nothing about changing the input layout or data type. Also, AI tools haven’t been much help :sweat_smile:

Is there any tutorial or example out there that shows how to properly handle this in Unity?

Thanks in advance!

2

Hi there,

Inference Engine doesn’t have any limitations on the input shape and can accept integer tensors as input. In this case however as you are using an image and will probably want to use the TextureConverter API (which outputs float tensors in the range [0, 1]), you will want to use the functional API for the model inputs.

There is a sample in the package that should be useful reference called ‘Use the functional API with an existing model’, this can be accessed via the package manager. In your case the code will be something like this:

var sourceModel = ModelLoader.Load(sourceModelAsset);

// Declare a functional graph.
var graph = new FunctionalGraph();

// Create input from image
var imageInput = graph.AddInput<float>(new TensorShape(1, 192, 192, 3));

// Scale and cast to integer range [0, 255]
var modelInput = (255f * imageInput).ToInt();

// Apply the forward method of the source model to the transformed functional input and add the outputs to the graph.
var outputs = Functional.Forward(sourceModel, modelInput);
graph.AddOutputs(outputs);

// Compile the graph to return the final model.
m_RuntimeModel = graph.Compile();

Then make sure you call the TextureConverter with the correct shape and tensor layout, in your case ‘TensorLayout.NHWC’. The ‘Convert textures to tensors’ sample contains many examples.

Hope this helps.

3

Thanks a lot gilescoope :slight_smile: I never noticed the examples in the package helped me a lot.

There is no ToInt() implementation You have to use the Functional methods but i managed to convert the model to use my textures to generate the output, couldnt done it withouth You :smiley:
This is my code that now works:

public Texture2D inputTexture;
public ModelAsset modelAsset;
BackendType backendType = BackendType.GPUCompute;
Model convertedModel;
Worker m_Worker;
float[] results;

void Start()
{
    // Modifies the model input from int to float and normalizes the input values to the range [0, 1].
    Model sourceModel = ModelLoader.Load(modelAsset);
    FunctionalGraph graph = new FunctionalGraph();
    FunctionalTensor imageInput = graph.AddInput<float>(new TensorShape(1, 192, 192, 3));
    FunctionalTensor modelInput = Functional.Div(imageInput,Functional.Constant(255));
    FunctionalTensor[] outputs = Functional.Forward(sourceModel, modelInput);
    graph.AddOutputs(outputs);
    convertedModel = graph.Compile();
}

private void Update()
{
    if (Input.GetKeyDown(KeyCode.Space))
    {
        // Create worker to run the model.
        m_Worker = new Worker(convertedModel, backendType);

        // Create the input from our input texture.
        using var input = new Tensor<float>(new TensorShape(1, 192, 192, 3));

        // Convert the texture to a tensor of the shape that fitts the model input.
        using Tensor<float> tensorNHWC = new Tensor<float>(new TensorShape(1, inputTexture.height, inputTexture.width, 3));
        TextureConverter.ToTensor(inputTexture, tensorNHWC, new TextureTransform().SetTensorLayout(TensorLayout.NHWC));

        // Execute the model.
        m_Worker.Schedule(input);

        var output = m_Worker.PeekOutput() as Tensor<float>;
        results = output.DownloadToArray();

        // Print the results to the console
        for (int i = 0; i < results.Length; i++)
        {
            Debug.Log($"{results[i]}");
        }
    }
}