Extracting depth map from a texture outputs a red image instead of black&white

AI Beta Sentis
1

Hi!

I’m extracting a depth map image from a Texture. The extraction works, but the output texture is red instead of black and white.

The model output format seems to be R16.

Here is the code I’m using. I’ve tried looking for black&white-conversion by settings Color(sourceImage.r, sourceImage.r, sourceImage.r), but this produces a completely red image instead.

What should I do to get the black&white output from the model output instead of a red image?

        runtimeModel = ModelLoader.Load(modelAsset);
        worker = WorkerFactory.CreateWorker(BackendType.GPUCompute, runtimeModel);

        if (inputTexture != null) {
            TensorFloat inputTensor = TextureConverter.ToTensor(inputTexture);
            
            worker.Execute(inputTensor);

            TensorFloat outputTensor = worker.PeekOutput() as TensorFloat;

            // Create a tensor operation to divide the output values by 255, to remap to the (0-1) color range
            // I'm not sure if this should be done and if this is the proper way?
            Ops ops = WorkerFactory.CreateOps(BackendType.GPUCompute, new TensorCachingAllocator());
            TensorFloat divideAmount = new TensorFloat(255f);
            TensorFloat tensorScaled = ops.Div(outputTensor, divideAmount);

            // Copy the rescaled tensor to a RenderTexture            
            var rt = TextureConverter.ToTexture(tensorScaled);

            // Convert RenderTexture to Texture2D
            extractedDepthMap = ToTexture2D(rt);
            
            // Show results on canvas
            sourceCanvas.texture = inputTexture;
            resultCanvas.texture = extractedDepthMap;
            
            targetShader.SetTexture("_Image", inputTexture);
            targetShader.SetTexture("_Depth_map", extractedDepthMap);
            
            worker.Dispose();
        } else {
            Debug.Log("No source image defined.");
        }

The code I’m using to convert a RenderTexture to Texture2D is this:

    private Texture2D ToTexture2D(RenderTexture rTex)
    {
        Texture2D dest = new Texture2D(rTex.width, rTex.height, TextureFormat.R16, false);
        dest.Apply(false);
        Graphics.CopyTexture(rTex, dest);
        return dest;
    }

Here’s an example of the texture this code produces:

Näyttökuva 2023-9-20 kello 15.59.27
Näyttökuva 2023-9-20 kello 15.59.27840×472 106 KB

3

In this line:

var rt = TextureConverter.ToTexture(tensorScaled);

You are creating a texture with 1 channel because the tensor has this shape. If you want a 3 or 4 channel tensor then you can use the ‘TextureTransform’ optional parameter with ‘.SetDimensions’, ‘.SetBroadcastChannels’, ‘.SetChannelSwizzle’ etc to control exactly how you want the tensor to be transformed to a texture. It’s also possible to create the RenderTexture with the format you want then use ‘TextureConvert.RenderToTexture’ (again with the TextureTransform to broadcast the channels).

Use of the TextureConverter, TextureTransform and associated methods are covered in more detail in the docs and samples. I hope this helps.

2 Likes
4

Thank you so much! This helped very much.