API Suggestions

AI Beta Sentis
1

First, I must admit - Sentis handles onnx model import and inference much better than Barracuda. Thank you for the hard work! The comparison of the results with ONNX Runtime is also very good. The differences are usually small, although there are exceptions from time to time. I can share some inference comparison with ORT, if you’re interested.

And here are some suggestions for improvements, based on my experiments so far, if you don’t mind:

  1. It would be good if there is API to serialize & deserialize the imported model to file, or to Unity asset (for easier and faster loading, with option to compress the content).
  2. It would be good if there is API to serialize & deserialize a tensor to file or to Unity asset.
  3. When a model is selected in the Editor, there are several lists in Inspector.
    • usually the lists contain long messages (e.g. in errors, warnings or layers) that are hard to see, because the list is not horizontally scrollable. It would be good, when the mouse hovers over a line, to have the full text of that line displayed as tooltip.
    • the names of the layers in the Layers-list are not the same as the node names in Netron. I wonder, which ones are correct.
    • usually in the Errors-list the messages are labeled as ImporterWarning. If it’s just a warning, why is it in the Errors-section? Or, maybe it should be ImporterError instead.
2 Likes
2

Thanks for the feedback!

  • for differences with ORT typically we are in the ~10e-5 absolute/relative ballpark.
    But if you find bigger differences do flag them out we might be doing things wrong

    1. we have implemented that and it’ll come in the upcoming version
    1. we’ll think on that one, but it’s actually quite easy to do that with a custom [Serialize] + float array
    1. thanks the the UI tips, we’ll update. For names we often create new layers/merge them and thus create new names for them, so it might not match exactly with Netron
1 Like
3

Regarding serialization of tensors - yes, you are right. But I meant to have the shape serialized/deserialized as well.

4

Correct + the datatype too, TensorShape is marked as [Serializable] too
so you can have something like

[Serializable]
public class TensorFloatS
{
    public TensorShape shape;
    public float[] data;
}

serializedT = new TensorFloatS();
serializedT.shape = tensor.shape;
serializedT.data = tensor.ToReadOnlyArray();
1 Like
5

Yes, it would be nice to have this as utility API in Sentis.

6

One more “nice to have” feature would be a static method like ‘WorkerFactory.GetBestBackendForDevice()’. I think this existed in Barracuda, and would free the developers from the task to update backend types, when a better backend type (e.g. for a specific mobile device) becomes available.

7

FYI we have this in our backlog to fix, known as issues 77 and 78.

1 Like