Tensor Operations with 1.4

AI & Machine Learning AI Inference Engine
,
1

I’m currently checking out the new 1.4 version of Sentis. Some promising stuff in there!

So far I’ve heavily used Ops to process tensor data that I’m now looking to port to IBackend.

issue with operations with scalar values?

Regarding the new docs page for Tensor Operations:

The example code in the documentation uses backend.Mul(randomTensor, mulTensor, 255f); but there doesn’t seem to be an overload that takes a scalar on IBackend. Is that a documentation issue, missing implementation or am I using it incorrectly?

question on tensor management

Other than that, the docs page gives a good overview of how to use it but focuses on operations where the result tensor has a different shape/size.

I’m wondering, what the best practice is with common operations that could be done in-place.
Let’s say I want to calculate:
x = a * b;

Following the docs I’d do

using TensorFloat x = TensorFloat.AllocNoData(a.shape);
backend.Mul(a, b, x);

Assuming a and b have the same shape, could I also reuse a as the output tensor and do

backend.Mul(a, b, a);

General thoughts on this? Good idea or no?

2

Hi @julienkay . I will leave it to another member of the team to answer the question about the backend API as I’m not an expert in it. (I can’t immediately see anything wrong with what you’re doing - though I don’t know if it was designed to be used like this. Of course that would fail with things like convolutions because of race conditions).

But before porting all your ops to the IBackend API. I would recommend you first take a look at the new Functional API examples here and here. And take a look at the 1.4.0 code on Hugging Face. The Functional API is much simpler in terms of memory management and has advantages such as compile time shape checking. It may take a bit of restructuring your code to take advantage of this way of doing things.

I have managed to convert code with lots of ops into the functional API and found you can do pretty much everything you can do with ops. And in many cases it is much faster too.

Of course, if you prefer to use the lower level backend code for extra optimisation, that’s fine too.

We would appreciate your feedback on the new API, for example how easy/hard you find the updating process.

3

Hi @julienkay ! I’m just starting to get in touch with the update and I have the same issue regarding operations on tensors with scalars. I eventually overcame this issue by creating a scalar tensor like so (I just copied your example and corrected it but I replaced the scalar value by the scalar tensor):

using TensorFloat scalar = new TensorFloat(255f);
backend.Mul(input1, scalar, output);

Again I’m not sure if it’s the correct solution so feel free to correct me if I’m wrong.

As for the second issue, I was also interested if it was possible to do calculations in place as it makes things shorter and easier so I tested it:

        float[] srcData1 = { 1, 2, 3 };
        float[] srcData2 = { 2, 2, 2 };
        TensorShape shape = new TensorShape(3);
        using TensorFloat a = new TensorFloat(shape, srcData1);
        using TensorFloat b = new TensorFloat(shape, srcData2);
        IBackend backend = WorkerFactory.CreateBackend(BackendType.CPU);
        using TensorFloat c = TensorFloat.AllocNoData(shape);
        // backend.Mul(a, b, a);
        // a.PrintDataPart(3);
        backend.Mul(a, b, c); 
        c.PrintDataPart(3);
        backend.Dispose();

The output with the operation in place resulted in an error:

AssertionException: BurstCPU.ScheduleJob.ValueError: output must be different from inputX
Assertion failure. Value was False
Expected: True

Then I tried your method and it correctly printed 2, 4, 6.
I guess we cannot do calculations in place unless mentionned in the documentation.

4

Hi, I’ll just also show how you would do this using the Functional API. (This is really a simplified example as in most cases you’d probably want to do more than one operation per model or combine the operations with another model.) If you keep hold of the worker you can use execute the model as many times as you like.

        var shape = new TensorShape(3);

        var multiplyModel = Functional.Compile( (a, b) => a * b,  (InputDef.Float(shape), InputDef.Float(shape)));

        float[] srcData1 = { 1, 2, 3 };
        float[] srcData2 = { 2, 2, 2 };
        using var a = new TensorFloat(shape, srcData1);
        using var b = new TensorFloat(shape, srcData2);
        var worker = WorkerFactory.CreateWorker(BackendType.GPUCompute, multiplyModel);
        worker.Execute(new Dictionary<string, Tensor> { { "input_0", a }, { "input_1", b } });
        var c = worker.PeekOutput() as TensorFloat;
        c.CompleteOperationsAndDownload(); //(if using GPUCOmpute)

        c.PrintDataPart(3);

        worker?.Dispose();

For “in-place” operations. Unless specified (such as the .Reshape()) operation, most operations are not in-place. But please feel free to tell us your use cases for this, as this was something we have talked about.

5

@PaulBUnity Thanks for these explanations these are helpful.
My initial questions are solved (thank you @MS-Fove for the tests, guess I should have done these myself)

I do have some feedback.
I appreciate having the functional API to create models, but I think there are cases where the processing is a bit more complex (many small processing steps between multiple models, loops, branching, …) where it’s not an ideal choice.

The backend API is appreciated as well, but I do have to say having to manage the memory of all tensors yourself does make the code a lot more verbose (especially coming from / porting code from python). I do miss the Ops API for it’s simplicity and I find myself gravitating towards implementing a wrapper around the backend API that allows to use it similarly.

I can go more into detail, but this Is roughly where I’m at right now.

6

Write me an email directly with the details. I’m curious for feedback