GPUPixel for Sentis 2 Blaze Templates

Unity Engine
1

In the Sentis 2 templates for Blaze Pose/Hand/Face (Hugging Face), the BlazeUtils class uses a compute shader for image affine transformations. Are there plans to include a pixel shader to support GPUPixel (for example for WebGL support)? I tried making one to no avail.

Thank you.

2

Yeah it’s on our list, either as a functional op or as an override for the TextureConverter methods, I can’t promise when this will land though. Have you considered using WebGPU? That supports compute shaders and will generally given better performance than the pixel shader backend.

3

We’re using WebGPU but we also consider using WebGL for increased accessibility, especially that WebGPU’s reach for devices and browsers is still in the process of maturation. Do you have any rough ETA for GPUPixel support?

4

You can easily replace the ComputeShader that transforms the image with a Graphics.Blit.
In your material,

float4 frag(v2f i, UNITY_VPOS_TYPE screenPos : VPOS) : SV_Target
{
     uint2 O_pos = (uint2)(screenPos.xy - 0.5f);
     float4 uv = mul(affineMatrix, float4(O_pos.x, O_pos.y, 1, 1)) / float4(X_width, X_height, 1, 1);

     bool mask = uv.x >= 0 && uv.x <= 1 && uv.y >= 0 && uv.y <= 1;
     float4 c = mask * X_tex2D.SampleLevel(LinearClampSampler, uv.xy, 0);
     // SRGB conversion
     bool3 maskRGB = c.rgb > 0.0031308f;
     c.rgb = lerp(12.92f * c.rgb, 1.055f * (pow(abs(c.rgb), 0.41666666666f) - 0.055f), maskRGB);

     return c;
}

Be sure to bind all the textures and matrices

5

Thanks for the reply. I tried implementing this shader and feeding the texture as input by converting it to a tensor like so:

// Setup material properties
Material material = new Material(pixelShader);
material.SetTexture("_MainTex", texture);
material.SetMatrix("affineMatrix", M.ToFloat4x4());
material.SetFloat("X_width", texture.width);
material.SetFloat("X_height", texture.height);
material.SetVector("O_pos", new Vector4(m_DetectorInput.shape[2], m_DetectorInput.shape[1]));

Graphics.Blit(texture, material);

// Feed texture as input tensor to the detector
TextureTransform textureTransform = new TextureTransform();
textureTransform.SetDimensions(width: 192, height: 192, channels: 3);
textureTransform.SetTensorLayout(0, 3, 1, 2);
m_DetectorInput = TextureConverter.ToTensor(texture, textureTransform);

m_HandDetectorWorker.Schedule(m_DetectorInput);

Similarly with the landmarker using its appropriate input size and worker.

Running it with the GPUPixel backend will make Unity hang. If I run it with the GPUCompute backend, the detection appears but is distorted (it looks like its scale matches the ratio of the transformed image).

I remember one time I tried it, i was able to read an error after Unity hangs, it was probably something about GPU readback, it could be the problem or a symptom of it. Is there something I’m missing? Maybe you need a more detailed answer?

P.S: Here is the vertex function of the shader if it’s relevant.

struct appdata
{
    float4 vertex : POSITION;
    float2 uv : TEXCOORD0;
};

struct v2f
{
    float2 uv : TEXCOORD0;
    float4 vertex : SV_POSITION;
};

v2f vert (appdata v)
{
    v2f o;
    o.vertex = UnityObjectToClipPos(v.vertex);
    o.uv = v.uv;
    return o;
}

And here’s the matrix conversion function.

public static float4x4 ToFloat4x4(this float2x3 matrix) {
    return math.float4x4(
        matrix.c0.x, matrix.c1.x, matrix.c2.x, 0,
        matrix.c0.y, matrix.c1.y, matrix.c2.y, 0,
        0,           0,           0,           0,
        0,           0,           0,           0
    );
}
Sentis support for the Unity Web platform
6

Only running the first detector on GPUPixel doesn’t freeze unity (or sometimes not immediately). I could get the following warnings:

Found unreferenced, but undisposed CPUTensorData which might lead to CPU resource leak
UnityEngine.Debug:LogWarning (object)
Unity.Sentis.D:LogWarning (object) (at ./Library/PackageCache/com.unity.sentis/Runtime/Core/Internals/Debug.cs:72)
Unity.Sentis.CPUTensorData:Finalize () (at ./Library/PackageCache/com.unity.sentis/Runtime/Core/Backends/CPU/BurstTensorData.cs:114)

Found unreferenced, but undisposed ComputeTensorData which might lead to GPU resource leak
UnityEngine.Debug:LogWarning (object)
Unity.Sentis.D:LogWarning (object) (at ./Library/PackageCache/com.unity.sentis/Runtime/Core/Internals/Debug.cs:72)
Unity.Sentis.ComputeTensorData:Finalize () (at ./Library/PackageCache/com.unity.sentis/Runtime/Core/Backends/GPUCompute/ComputeTensorData.cs:81)

So I guess the freezing is because of a memory leak. Will probably make a thorough analysis with the memory profiler but wanted to provide this update for now.

P.S: I also now realized the following shader error for the image transformation when sampling the main tex but I’m not sure if it’s related

 'Hidden/ImageTransform': undeclared identifier 'LinearClampSampler'