Coverting sentis-whisper-tiny to Sentis 2.0.0

Unity Engine
, , , ,
1

Hi everyone,

I have managed to convert most of the code for the whisper example (as downloaded from HuggingFace) but I am having no success with the following code in RunWhisper.cs at about line 80:

 Model decoderWithArgMax = Functional.Compile(
         (tokens, audio) => Functional.ArgMax(decoder.Forward(tokens, audio)[0], 2),
         (decoder.inputs[0], decoder.inputs[1])
 );

I have tried a few things to get this working but with no luck, the Compile function has changed, there is no “Forward” on the model anymore and it doesn’t seem to like the audio input. Learning as I am going along so no expert!

Does anyone have any pointers on how to update this?

Thanks
Liam

2

cf Edit a model | Sentis | 2.0.0

var graph = new FunctionalGraph();
var tokens = graph.AddInput(model, 0);
var tokens_audio = Functional.Forward(model, tokens);
var output = Functional.ArgMaX(tokens_audio[0], 2);
var decoderWithArgmax = graph.Compile(output);
3

Thanks for this. The script will now compile but I am getting this error:

AssertionException: ModelOutputs.ValueError: inputs length does not equal model input count 1, 2
Assertion failure. Value was False
Expected: True

Is it something to do with the “audio” variable?

4

Got it working, just had to make one minor change:
FunctionalTensor[] tokens = graph.AddInputs(decoder);
Thanks for your help!

5

Ah yes sorry for that~ good that you figured it out :slight_smile:

6

Would someone be able to help me with a similar issue of converting sentis-MiniLM-v6 to Sentis 2.0? I’m in the same boat of having been able to convert the rest of the code but I am currently stuck on the two Functional.Compile() calls in the sample script:

    Model modelWithMeanPooling = Functional.Compile(
      (input_ids, attention_mask, token_type_ids) =>
      {
          var tokenEmbeddings = model.Forward(input_ids, attention_mask, token_type_ids)[0];
          return MeanPooling(tokenEmbeddings, attention_mask);
      },
      (model.inputs[0], model.inputs[1], model.inputs[2])
    );

and

    Model dotScoreModel = Functional.Compile(
        (input1, input2) => Functional.ReduceSum(input1 * input2, 1),
        (InputDef.Float(new TensorShape(1, FEATURES)),
        InputDef.Float(new TensorShape(1, FEATURES)))
    );

I’m not sure how to get the inputs/outputs to match up with the new FunctionalGraph approach. Any help/guidance would be sincerely appreciated!

8

Hi there, here’s how to rewrite those:

var graph = new FunctionalGraph();
var input_ids = graph.AddInput(model, 0);
var attention_mask = graph.AddInput(model, 1);
var token_type_ids = graph.AddInput(model, 2);
var tokenEmbeddings = Functional.Forward(model, input_ids, attention_mask, token_type_ids)[0];
var meanPooling = MeanPooling(tokenEmbeddings, attention_mask);
Model modelWithMeanPooling = graph.Compile(meanPooling);

and

var graph = new FunctionalGraph();
var input1 = graph.AddInput(DataType.Float, new TensorShape(1, FEATURES));
var input2 = graph.AddInput(DataType.Float, new TensorShape(1, FEATURES));
var output = Functional.ReduceSum(input1 * input2, 1);
Model dotScoreModel = graph.Compile(output);

Hopefully that’s instructional and clear.

1 Like
9

Thank you so much for helping with this! Unfortunately this code isn’t quite working, as now I am getting this error when I try to run the model:

AssertionException: Cannot get the rank of a shape with a dynamic rank.
Assertion failure. Value was False
Expected: True
UnityEngine.Assertions.Assert.Fail (System.String message, System.String userMessage) (at <e509afeff7384f24a8f0ac30527ff01c>:0)
UnityEngine.Assertions.Assert.IsTrue (System.Boolean condition, System.String message) (at <e509afeff7384f24a8f0ac30527ff01c>:0)
Unity.Sentis.Logger.AssertIsTrue (System.Boolean condition, System.String msg) (at ./Library/PackageCache/com.unity.sentis@2.0.0/Runtime/Core/Logger.cs:48)
Unity.Sentis.DynamicTensorShape.get_rank () (at ./Library/PackageCache/com.unity.sentis@2.0.0/Runtime/Core/ShapeInference/DynamicTensorShape.cs:41)
Unity.Sentis.Layers.Select.InferPartial (Unity.Sentis.PartialInferenceContext ctx) (at ./Library/PackageCache/com.unity.sentis@2.0.0/Runtime/Core/Layers/Layer.Transformation.cs:1017)
Unity.Sentis.Functional.Forward (Unity.Sentis.Model model, Unity.Sentis.FunctionalTensor[] inputs) (at ./Library/PackageCache/com.unity.sentis@2.0.0/Runtime/Core/Functional/Functional.Model.cs:47)
MiniLM.CreateMLModel () (at Assets/Scripts/_inport/AI/MiniLMv6.cs:105)
MiniLM.GradeSimilarity (System.String sentence, System.String input) (at Assets/Scripts/_inport/AI/MiniLMv6.cs:45)
MiniLM.Start () (at Assets/Scripts/_inport/AI/MiniLMv6.cs:32)

This error happens when Functional.Forward() is called in the CreateMLModel() method.

10

Thanks for the feedback, this is a bug, I will create a ticket and we will try and get a fix in the next update.

The only thing I can think of that might fix this would be to try and make sure the inputs to the model are static shapes.

So do

var input_ids = graph.AddInput(___, new TensorShape(_____));
var attention_mask = graph.AddInput(___, new TensorShape(_____));
var token_type_ids = graph.AddInput(___, new TensorShape(_____));

You can inspect the model in Unity to see the data types and input shapes, and if they have dynamic shapes (they will look like “d0, d1…”) you can write those out with static values that you will feed in with your input tensors.

I can’t 100% guarantee this will fix the issue you have described, as it will depend on your model. But in general if your input tensors have fixed sizes then it’s good practice to do this, as we can optimize the model better in this case.

Let me know if this works. As I say we will try and fix the underlying issue too.

This is known internally as SENTIS-800.