Using Sentis with WebcamTexture

AI & Machine Learning AI Inference Engine
,
1

Hi, is it possible to use Unitys WebcamTexture with Sentis? I tried to convert it to a Texture2D, or use the WebcamTexture directly, but I’ll always get the following errors:

  • Given input shape: (1, 3, 720, 1280) has different dimension from model input shape: (1, 3, 416, 416) for input: input.1 at axis: 2
  • Given input shape: (1, 3, 720, 1280) has different dimension from model input shape: (1, 3, 416, 416) for input: input.1 at axis: 3
  • AssertionException: ValueError: reshaped length does not match with input length
    Assertion failure. Value was False

I tried to follow the TextureToTensor samples, but

My code for converting the WebcamTexture to Texture2D looks like this:

private void OnRenderImage(RenderTexture source, RenderTexture destination)
{    
    // convert webcamtexture to texture2D
    cameraTexture2D = Texture2D.CreateExternalTexture(webcamTexture.width, webcamTexture.height, 
    TextureFormat.RGB24, false, false, webcamTexture.GetNativeTexturePtr());

    using TensorFloat tensorRGB = TextureConverter.ToTensor(cameraTexture2D, channels: 3);
    worker.Execute(tensorRGB);

    // Get the result
    using TensorFloat outputTensor = worker.PeekOutput() as TensorFloat;

    // Create a tensor operation to divide the output values by 255, to remap to the (0-1) color range
    Ops ops = WorkerFactory.CreateOps(BackendType.GPUCompute, new TensorCachingAllocator());
    TensorFloat tensorScaled = ops.Div(outputTensor, 255f);

    // Copy the rescaled tensor to the screen as a texture
    TextureConverter.RenderToScreen(tensorScaled);
}

br,
Max

2

Hi there, so as the error mentions, your tensor has a different shape to the expected model input shape.
(1, 3, 720, 1280) vs (1, 3, 416, 416)

If you don’t tell the TextureConverter methods otherwise then output tensor will take its dimensions from the input image, in this case the webcam resolution of 720 x 1280. If you want to resize to a different size when converting you can provide ‘width’ and ‘height’ parameters to the ‘TextureConverter.ToTensor’ method, or for more options use the TextureTransform parameter.

There’s more information and examples in the docs and samples.

3

TextureConverter.ToTensor(image, width: 416, height: 416, channels: 3) is the method in action.