How to render object with just a depth texture?

Unity Engine
,
1

I’ve been struggling this for days now and realized I need direct help from people who are more knowledgeable about this.

I am working with ARKit and want objects to occlude my hand if they are closer. I have at my disposable

From AR Kit SDK

  1. a depth texture of my hand
  2. a stencil texture of my hand

Solution A?

  1. Camera A renders scene to RenderTexture
  2. Camera B renders scene to a depth texture?
  3. Camera C has a quad with ShaderX
  4. ShaderX combines textures A+B+AR depth texture and use fragment shader to compare pixel values determining what gets shown.

I’m sure there are better ways to get a solution, but I don’t know my options. Also, I’m having trouble with step #2. Any help is greatly appreciated

2

Other option:

  1. Render ARKit depth texture depth into scene depth.
  2. Render scene.
  3. Fin.

Requires using a shader that writes to SV_Depth. You can use ColorMask 0 to skip rendering to the color values. My understanding is the ARKit’s depth texture is in linear meters, so you need to convert that into the non-linear depth of the current camera. Here’s the function I use to do that.

            float LinearToDepth(float linearDepth)
            {
                return (1.0 - _ZBufferParams.w * linearDepth) / (linearDepth * _ZBufferParams.z);
            }

Another option:

  1. Render scene with shaders that sample the ARKit depth texture and clip() when further away.

Have your shader pass the screen position and linear depth from the vertex to the fragment, and compare against the ARKit depth texture. Look at the built in particle shaders for an example, though that doing soft fading with the results, and needs to convert the camera depth texture into linear depth, which you shouldn’t need to do.

3

This sounds like a good route to take. I read the documentation about SV_Depth and making the frag shader output to the depth buffer for me, but how do I pass in the linearDepth parameter as in your example. I shouldn’t use tex2D because that unwrap the texture to conform along the mesh correct? Or maybe I should use a quad covering the screen so I won’t have the problem.

4

You absolutely should be using tex2D to sample the ARKit depth texture, that’s the only (sane) way to get per pixel values. You’ll want to use a command buffer to call [Blit()](https://docs.unity3d.com/ScriptReference/Rendering.CommandBuffer.Blit.html) on [CameraEvent.BeforeForwardOpaque](https://docs.unity3d.com/Manual/GraphicsCommandBuffers.html). Blit draws a full screen quad, with UVs matching the screen.

The linear depth is the value in the depth texture you get from ARKit. You’ll need to create a material in script that uses your custom shader and assign the depth texture (and maybe stencil texture to clip() against) to the material before using it in the Blit().

You’ll want your blit to look like this:

CommandBuffer myCommandBuffer = new CommandBuffer();
myCommandBuffer.Blit(null, BuiltinRenderTextureType.CurrentActive, myDepthWriteMaterial);
myCamera.AddCommandBuffer(CameraEvent.BeforeForwardOpaque, myCommandBuffer);

You just need to do that once OnEnable.

1 Like
5

Wonderful! I’m almost there. I’m using a dummy greyscale texture of a hand in the shader that you instructed me to write (shown below). The hand silhouette blocks objects that are far away from being rendered and is properly rendered when it gets closer to the camera.

The only problem that remains is that everything outside the silhouette is pitch black. Could it be that it’s because I’m writing directly to SV_Depth and meddling with the depth and that I need to add the depth of the rest of the scene?

    SubShader
    {
        Tags { "RenderType"="Opaque" }
        LOD 200

        pass
        {
            ColorMask 0
            ZWrite On
            Fog { Mode Off }
          
            CGPROGRAM

            #pragma vertex vert
            #pragma fragment frag
            #pragma target 3.0
            #include "UnityCG.cginc"
  
            sampler2D _ArDepthTex;
          
            struct v2f
            {
                float2 uv : TEXCOORD0;
                float4 vertex : SV_POSITION;
            };
  
            v2f vert(appdata_base v)
            {
                v2f o;
                o.vertex = UnityObjectToClipPos(v.vertex);
                o.uv = v.texcoord;
                return o;
            }

            float frag(v2f i) : SV_DEPTH
            {
                fixed4 col = tex2D(_ArDepthTex,  i.uv);
                float depth = col.r;
                return (1.0 - _ZBufferParams.w * depth) / (depth * _ZBufferParams.z);
            }

            ENDCG
        }

    }
    FallBack "Diffuse"
6

You said you had a stencil texture, though I don’t entirely understand in what form the stencil texture is passed in. Is it a b&w image of the hand outline? If so, sample that texture and use:
clip(stencil.r - 0.5);

7

As for why, if you’re using a dummy texture rather than the real one, that means you have a black and white texture with a range of 0.0 to 1.0, presumably the hand is black, and the background is white. Well that’s going to write to the depth as if there’s a wall 1 meter away. The real ARKit depth texture should be a floating point texture with a much larger range, so presumably the “white” areas are actually values much larger than 1.0, but honestly I have no idea since I’ve never used ARKit.

The other problem I don’t have an answer to is I don’t know how the ARKit camera view is rendered to the scene. Ideally it gets rendered into the camera first and ignores the depth buffer.

8

Yes, you’re right! I forgot about adding the stencil texture. The frag shader no longer sets the depth outside the hand so everything else renders just fine.

This was done in a test environment, the only thing to do now is to try it in the real environment fingers crossed

9

Hi there, did you succeed in doing this in the end? I’m curious & trying to do the same thing