Why won't simple Stencil code work on HDRP Lit?

I have a simple shader (Red and Blue) which renders the green geometry where the mesh mask (green) intersects with the helmet (red) geometry. I’m trying to port this shader code to the HDRP Lit material but I have had complications.

Basic Shader (Works fine)

Stencil code put on a HDRP Lit Shader in GBuffer Pass:

Red Shader:

Shader "Stencils/Red" {
    SubShader {
        Tags { "RenderType"="Opaque" "Queue"="Geometry"}
        Pass {
            Stencil {
                Ref 254
                Comp always
                Pass replace
            }

            CGPROGRAM
            #pragma vertex vert
            #pragma fragment frag
            struct appdata {
                float4 vertex : POSITION;
            };
            struct v2f {
                float4 pos : SV_POSITION;
            };
            v2f vert(appdata v) {
                v2f o;
                o.pos = UnityObjectToClipPos(v.vertex);
                return o;
            }
            half4 frag(v2f i) : SV_Target {
                return half4(1,0,0,1);
            }
            ENDCG
        }
    }
}

Green Shader:

Shader "Stencils/Green" {
    SubShader {
        Tags { "RenderType"="Opaque" "Queue"="Geometry+1"}
        Pass {
            Stencil {
                Ref 254
                Comp equal
                Pass keep
                ZFail decrWrap
            }

            CGPROGRAM
            #pragma vertex vert
            #pragma fragment frag
            struct appdata {
                float4 vertex : POSITION;
            };
            struct v2f {
                float4 pos : SV_POSITION;
            };
            v2f vert(appdata v) {
                v2f o;
                o.pos = UnityObjectToClipPos(v.vertex);
                return o;
            }
            half4 frag(v2f i) : SV_Target {
                return half4(0,1,0,1);
            }
            ENDCG
        }
    }
}

The GBuffer Pass on my Red HDRP Lit Shader:

Pass
{

   Name "GBuffer"
   Tags { "LightMode"="GBuffer" }

   Cull [_CullMode]
   ZTest [_ZTestGBuffer]

   Stencil
   {
              Ref 254
              Comp always
              Pass replace
          }


   ColorMask [_LightLayersMaskBuffer4] 4
   ColorMask [_LightLayersMaskBuffer5] 5

   HLSLPROGRAM

The GBuffer Pass on my Green HDRP Lit Shader:

        Pass
        {

            Name "GBuffer"
            Tags { "LightMode"="GBuffer" }

            Cull [_CullMode]
            ZTest LEqual

            Stencil {
                Ref 254
                Comp equal
                Pass keep
                ZFail decrWrap
            }


            ColorMask [_LightLayersMaskBuffer4] 4
            ColorMask [_LightLayersMaskBuffer5] 5

            HLSLPROGRAM

Now the very very strange thing is when I put the Red Shaders stencil code in the HDRP Lit version, it seems to have no problem picking up the original Unlit Green shader, but will not pick up the Lit version:

But for some reason the Green HDRP Lit version seems to not be rendered at all? Why would one stencil work and not the other? I don’t understand where I’ve gone wrong.

Look at the stencil doc of HDRP.
For Lit shaders, most of the bits are reserved by the render loop, only bit 6 and 7 are kept free to use. So I think your stencil ref needs to be 64 or 128.

The other “safer” option would be to use HDRP “draw renderers” custom pass with the dedicated stencil options.

Hey thanks for the help! For some reason changing the stencil ref to 64 or 128 seems to always break the Red Shader. The strange thing is the Green shader then renders over it as red instead

I guess this is a step in the correct direction as one shape definitely is rendering as HDRP Lit but using any of the free bits (128 or 64) seems to completely break the lighting somehow. If I change it to anything back to anything else it will render normally again.

Hum, thinking of it, since your shaders are for writing into the GBuffer, you are doing deferred lighting, so you want to write to bit 1.
It needs to be taking into account when reading/writing.

So the proper stencil ref value should be 66 ? For your custom bit 6 + the bit 1 that flags deferred lit objects.

But like I said, maybe using custom passes would be less error prone :confused:

1 Like

Hey Remy I’ve been stuck on this for days on end and your solution of using ref 66 seemed to do the trick!

I had 2 questions though. Why is it one shader (Unlit Green) is clipped correctly from the side but the Lit version is not? Both have the same render Queue of Geometry+1 and the stencil code is identical?

Also in regards to using CustomPass volume, there is a few things I dont understand. So take the 2 stencils below:

            Stencil {
                Ref 254
                Comp always
                Pass replace
            }
            Stencil {
                Ref 254
                Comp equal
                Pass keep
                ZFail decrWrap
            }

Sometimes there is no ZFail condition or some aren’t used in the stencil. In the volume config I don’t seem to see an option to leave out anything like you can in the stencil code:

I understand this shader will be applied to everything on the Mask Layer but when it comes to overriding the stencil there was always an operation or two I didn’t need and I don’t see any way to leave them out. Am I doing the correct thing by overriding the stencil or should I put my stencil code in the shader and just use this to apply it to a layer?

Thanks again for all your help regardless

I’ve actually just figured out the answer to my first question, which the solution was to set ZTest from Always to LEqual and remove ZFail from Stencil.

The only issue I’m encountering at the moment is just some flickering from some camera angles/positions. Perhaps something to do with MotionVectors/SceneDepth passes being different stencil code?

Thanks again for all your help Remy

This could be a rendering other issue, double check the ordering of the passes for the two cylinders ?