While playing around with different models on CPU and GPU, I noticed that in tensors where one dimension is 0, the tensorOnDevice value is sometimes null.
I couldn’t reproduce the error reliably (so it looks like a memory leak to me), but it seems to occur more often in the GPU runtime than in the CPU runtime. I guess, a tensor of dimension 0 should contain an empty array and not null (If I create a 0-dimensional tensor via TensorInt.Zeros(new TensorShape(0)); create tensorOnDevice contains an empty array on both CPU and GPU.)
Some operations like “Concat” don’t seem to have any problems. However, I noticed it with “Gather” and maybe some other operations are affected as well.
Minimal code to reproduce the behaviour:
ITensorAllocator allocator = new TensorCachingAllocator();
Ops gpuOps = WorkerFactory.CreateOps(BackendType.GPUCompute, allocator);
TensorInt input = new TensorInt(new TensorShape(3,1),new int[] { 0,1,2});
TensorInt indices = TensorInt.Zeros(new TensorShape(0));
Debug.Log(indices.tensorOnDevice); //Works fine on CPU
ComputeTensorData.Pin(indices);
Debug.Log(indices.tensorOnDevice); //Works fine on GPU
Tensor result = gpuOps.Gather(input, indices, 0);
Debug.Log("Result: " + result.shape);
//This one sometimes evaluates to true -> branches of my models cannot be completed
Debug.Log("Result: " + (result.tensorOnDevice == null));
I observed a similar behavior some time ago with an onnx “Shape” operation. I couldn’t perform any direct “Add” operations on the output of the “Shape” node. However, when I used a Gather operation in between (“Shape” → “Gather (0)” → “Add”) it worked.
Another strange observation, which is not necessarily related to the previous mentions, is:
If I want to run a model on the GPU and then iterate over all layers and output the deviceType, most of the outputs are still on the CPU (maybe this is due to my device as I use a Mac).
(...)
//Add all layers to outputs
foreach(Unity.Sentis.Layers.Layer l in modelLoader.layers)
{
modelLoader.AddOutput(l.name);
}
IWorker worker = WorkerFactory.CreateWorker(BackendType.GPUCompute, modelLoader);
//Execute the model
(...)
foreach (Unity.Sentis.Layers.Layer l in modelLoader.layers)
{
Tensor testTensor = worker.PeekOutput(l.name);
Debug.Log(l.name + ": " + testTensor.tensorOnDevice.deviceType);
}
I would expect that all tensors that are output of an (GPU supported according to Supported ONNX operators | Sentis | 1.2.0-exp.2 ) operation are also on the GPU after executing the model.
However, that is not the case.
Maybe others have already stumbled across these issues and have hints or explanations 
.
