Performance problem with Sentis Whisper Tiny

Unity Engine
,
1

Hi, according to the post here, it’s said that the main thread should only blocks for the job/graphics scheduling.

I use the Sentis Whisper Tiny demo to test on Android. Here’s the time of each part.
It takes 23s for padding. It’s horrible. Is it run in a sub thread?
And the other parts costs tens to hundreds of milliseconds.
If the tensor is never makeReadable and execute the next model again, the data is not read back. Right?
How can I make it not blocking the main thread?
Thank you.

before new TensorFloat(new TensorShape(1, numSamples), data)
TensorFloat time: 00:00:00.0108040
before ops.Pad(input, new int[] { 0, 0, 0, maxSamples - numSamples })
Pad time: 00:00:23.5512980
before spectroEngine.Execute(input30seconds)
Spectro time: 00:00:00.0336710
before encoderEngine.Execute(spectroOutput)
Encoder time: 00:00:00.4862310
before decoderEngine.Execute(inputs)
Decoder time: 00:00:00.8611970
before decoderEngine.Execute(inputs)
Decoder time: 00:00:00.0252620
before decoderEngine.Execute(inputs)
Decoder time: 00:00:00.0634260
before decoderEngine.Execute(inputs)
Decoder time: 00:00:00.0153540
before decoderEngine.Execute(inputs)
Decoder time: 00:00:00.0302340
before decoderEngine.Execute(inputs)
Decoder time: 00:00:00.0139590
和命中有什么
2

None of that code on HF is optimized.
It is meant as a example to get started, so we do take shortcuts that have some overhead which you could optimize away depending on your application

3

I can help out with different part of the sentis execution that you feel are not performant enough and give you ways to minimize such costs.
But do break it down into individual issues

4

Thank you. I updated to 1.4.0. And take the latest RunWhisper.cs code as a reference. It improves a lot. No pad op is needed now. And the 23s cost of it is removed.
But the other part still costs tens to hundreds of milliseconds. Any other suggestion to improve it?

5

which part in question is taking 100ms?
could you share a profiling capture.
Be sure to put the editor in release mode when profiling

6

Thank you for your reply.

Here are the captures.
For the 1st time. It costs 19s for one frame. It’s horrible experience.
For the 2nd & 3rd time, it’s better. But it still costs hundred milliseconds.
It’s running on the RayNeo X2 XR glasses. It’s not on the desktop.

Is it possible to totally run it in the sub thread like some native plugin does? It blocks the main thread would be with a poor experience.

https://drive.google.com/drive/folders/1xDR_yHlZsfMwq9WoeOiBiZMKHrIWzAo1?usp=sharing

7

Checking the log I see that sentis takes 16ms and there is a logtoconsole that takes 15ms too… so you might want to get rid of that
Sentis scheduling time is a bit high… are you sure you are not running in debug mode?
It shouldn’t cost that much unless it’s somehow doing new buffer allocations when running inference

8

Thank you. What do you mean by debug mode? Does it refer to developement build? If I don’t build development build, I can’t use profiler to capture.

9

image
Bottom right of the editor.

10

Thank you. I confirmed that it’s in mode release.

11

Gotcha.
Could you try one last thing?
Profile one inference with a given shape.
Then profile a second inference with the same shape.

input = Tensor(n, s, l) // specific shape
// profile call 1
worker.Execute(input);
// profile call 2
worker.Execute(input)

Tell me if execute2 is faster than execute1

12

Yes. The execute 2 is faster.

13

What’s the difference in ms between the two?