How can I make a camera render only one layer WITH occlusion?

[patben8]

Hi everyone. I’ve been racking my head trying to figure this out. Basically what I’m looking for is to have a camera render only one layer of objects, just like changing the culling mask on a camera, however I also need the object to be occluded by objects in other layers.

Here’s an example of what I mean:
I’ve got a scene with a Sphere and a cube. The sphere is in front of the cube, partially occluding it. Like so:

And what I’m looking to achieve is a camera output that looks like this:

I managed to achieve that result by using a shader that uses a render texture of only the sphere as a mask for a second render texture of only the cube. However the issue is that if the occluding object (in this case the sphere) moves behind the object, you get the following:

Which is the same result you would get if the sphere were in front of the cube. When instead I’d want it to look like this:

Since the cube is covering the sphere, the cube is “in front” and is therefore “complete”, as opposed to having the sphere in front of the cube in which case the cube is missing a “sphere sized chunk”.

I know this is probably an obscure question but if anyone has any ideas on how I could achieve this, I would love to hear it!
Thanks in advance for any help you can provide.

PS: I’m using the URP. The ultimate goal is to pixelate the “special” masked layer while leaving all other objects rendering at full resolution.

[MaxKramarov]

Do next:

1. Draw mask geometry before opaque geometry by the special shader (depthWrite=true, stenscil=1, colorWrite=false)
2. Then draw opaque geometry that has to be masked. Draw it by your normal shaders. The mask from the previous step will be “applied” because the mask is in the depth buffer. Also, override the stencil for maskable geometry. Set it in 2.
3. You have to remove mask geometry from the depth buffer to skybox and transparent geometry display properly. For that clear depth buffer where stencil equals 2. You can clear the depth buffer by drawing a plane over the screen with a shader that has depthTest=always and sets vertex position in max possible value.

So you URP renderer should seem:

Shader for mask geometry:

Shader "Unlit/MaskShader"
{
    Properties
    {
    }
    SubShader
    {
        Tags { "RenderType"="Opaque" }
        LOD 100

        Pass
        {
            ColorMask 0
        }
    }
}

Shader for the plane that clears depth:

Shader "Unlit/RestoreDepthBuffer"
{
    Properties
    {
    }
    SubShader
    {
        Tags { "RenderType"="Opaque" }
        LOD 100

        Pass
        {
            ColorMask 0
            ZTest Always
            
            CGPROGRAM
            #pragma vertex vert
            #pragma fragment frag

            #include "UnityCG.cginc"

            struct appdata
            {
                float4 vertex : POSITION;
            };

            struct v2f
            {
                float4 vertex : SV_POSITION;
            };

            v2f vert (appdata v)
            {
                v2f o;
                o.vertex = UnityObjectToClipPos(v.vertex);
#if UNITY_REVERSED_Z
                o.vertex.z = 0;
#else
                o.vertex.z = _ProjectionParams.z;
#endif
                o.vertex.w = _ProjectionParams.z;
                return o;
            }

            void frag (v2f i)
            {
            }
            ENDCG
        }
    }
}