Question Sentis training possible?

AI & Machine Learning AI Inference Engine
,
1

In the docu I see only inference, no output to ONX and no API to modulate weights or API to build model from scratch.

Sentis has no training possible? Only consume ONX?

Beta discussion should be in the main forum, with the rest, IMO. Although the SNR is quite a bit higher here than on the forum (…working on it ;))

2

If you can express your backward pass as a model then training is possible :wink:

3

The purpose of Unity is to simplify our work, isn’t it.

4

Model training we added the functional API ~Samples\Use the functional API with an existing model\UseTheFunctionalAPI.cs
https://docs.unity3d.com/Packages/com.unity.sentis@1.6/manual/create-a-new-model.html
I hope that is is easy for you, if not feedback is always welcomed.

In so far as training support we didn’t see compelling run-time use-case that would justify it. But if you have a exciting use for it, please do share!

5

Thats to create model at runtime, right? I don’t really see a use case for it but I don’t do research. Do you have a practical example?

It’s niche. I deal a lot with ecosystem sims so replacing BT with neural net makes sense to get emergent behavior at scale. Maybe your experience doesn’t support that, I’m open to ideas.

6

It doesn’t need to be at runtime. You can re-serialize your model back to file if you need it.
So you’d do that offline and save your new model.
Practical example spans from

  • adding a Softmax at the end of your model
  • adding a topp to a llm
var model_tokenize = Functional.Compile(
forward: inputs =>
{
    var logits = FunctionalTensor.FromModel(model, inputs)[0];
    logits = logits[.., ^1,..];
    var scores = Functional.Softmax(logits / temp, -1);

    var probsAndIndices = Functional.TopK(scores, top_k, -1);
    var sortedProbs = probsAndIndices[0];
    var sortedIndices = probsAndIndices[1];
    var cumSum = Functional.CumSum(sortedProbs, -1);

    var mask = (cumSum - sortedProbs) <= top_p;
    sortedProbs *= mask;
    sortedProbs /= Functional.ReduceSum(sortedProbs, -1);

    return new[] { sortedProbs };  },
                InputDef.FromModel(model)
            );
  • or you can even have fun and write complete graph that can run on the gpu/cpu.
    I have a neat example that shows you can get some physics simulation as a forward pass inference which runs on the GPU that I meant to release as a sample.
7

Now you’re talking my language :grin:
There was a guy who made a galaxy sim, using sentis as a math box.
(over)Thinking in the future: can sentis computebuffer be accessed via shaders, when using the GPU backend? I’d hate to do a roundtrip to the CPU with that much data.

8

Yeah yeah of course you can.
GPU tensors are computebuffers
https://docs.unity3d.com/Packages/com.unity.sentis@1.6/manual/access-tensor-data-directly.html
And ~\Samples\Use a compute buffer

Give me 1 or 2 days and I’ll release the upgraded sample of the galaxy simulation.
Where everything is on the gpu and the resulting position is sampled in the vertex shader to update the particle positions

9

@laurentlavigne here you go

10

Was going to wrap up this deadline but instead… down the rabbit hole I go :wink:

This looks super well thought out :+1:
Got this Burst vibe to it, simpler than jobs, I like.

Thanks! This is really some of the most concise and cleanest sim code I’ve seen!

Now for something a bit closer to actual gameplay, do you think this would work well with game stuff like influence maps?
https://github.com/laurentopia/AI-Influence-Map-using-COMPUTE-SHADER-and-ASYNC

I noodled a bit with the samples and found compile time to be quite high, will you guys eventually work on that?

Will you drop the version requirement? 2023.3 is very steep.
I’d like to use it in prod but that’s too high.
Most prods I know use 2022-2020, and on Quest2 we’re pretty much stuck with 2020 due to a string of performance regressions.

11

Unfortunately, for now the official min version is 2023.
But go try out 2022 it just so happens to still work wink wink