High performance inferencing

[alexandreribard_unity]

First point, this thread should help

AsyncReadbackRequest

Maybe I’m missing something, but isn’t this how I currently have the code structured? Aside from while (!outputTensor.IsAsyncReadbackRequestDone()) if (shouldPrint) { UnityEngine.Debug.Log("Waiting on async process to finish..."); shouldPrint = false; }; if (!isMainThread()) UnityEngine.Debug.Log($"Is on MainThread: {isMainThread()}"); the code should for the most part match the code from the resource you shared for read-output-async.

model.Execute schedules the work to GPU/Burst. There might have some layers that needs to run on the CPU (shape calculations…) too.
Could very well be that the cumulative of those two cost 12ms…
To aleviate this, I would try two things:

• fix your input shape of your model as much as possible to make our shape inference more able to bake things down.
• schedule N layers at a time
  Run a model | Sentis | 1.2.0-exp.2
• avoid blocking read from tensor (MakeReadable/CompletePendingTransactions)
  Read output from a model asynchronously | Sentis | 1.2.0-exp.2

If you think the 12ms is excessive, do file a bug report