DepthNormal replacement shaders and Stencil buffer

Unity Engine
,
1

Short version: I have a custom DepthNormals shader, but it’s stencil properties are incorrect rather than being copied from the material that it’s a replacement shader for.

Long version: I have an unlit shader that does masking against the stencil buffer (‘Game Opaque’). I’ve added a separate ‘Game DepthNormals’ shader, which is a copy of “Hidden/Internal-DepthNormalsTexture” but with the stencil properties and subshader stencil block added. But the stencil test being ignored in play mode so the depth+normals are generated everywhere rather than just the masked region.

The frame debugger shows that the StencilXxx values for the material aren’t being set in the DepthNormals pass (they are correct in the DepthTexture pass and the Opaque pass):


(notice the Stencil Comp is ‘Disabled’ where it should be ‘Equals’)

Unity 2019 LTS, Built-in pipeline.

Things I’ve tried:

  1. Yes, I’ve overridden the built-in DepthNormals shader via ProjectSettings->Graphics. You can see above the custom replacement shader is being used.
  2. The ‘Game Opaque’ shader also has a custom ShadowCaster pass, which is essentially the same thing (regular depth-only shader but with stencil added). This works and the ‘Stencil Comp’ is correctly set in the Frame Debugger.
  3. All the Stencil Comp etc. properties are named identically in all versions.
  4. It doesn’t matter if I declare the stencil properties as Float or Int
  5. Stencil Ref does work. I can see no differences between this one property and all the others.
  6. It doesn’t seem to matter if the stencil settings are in the SubShader or the Pass (in the DepthNormals shader)
  7. Yes, they’re in the correct subshader (opaque) in the DepthNormals shader. Fudging the colour output does cause the DepthNormals output to change colour as expected.
  8. Yes, the values are correct for the material when viewed in Play mode in the inspector.
  9. The stencil buffer contents have been validated (the mask is populated via a Command Buffer that’s present in Opaque, Depth and DepthNormals passes) in the frame buffer.
  10. Confusingly, the values in the frame debugger aren’t just the defaults set in the DepthNormals shader, but instead separate values (eg. Stencil Comp is ‘Disabled’ but the default is ‘Always’ and the source materials it’s set to ‘Equal’). This suggests that it’s explicitly disabled in the DepthNormals rendering code?
  11. This is Built-In renderer, in Forward mode, not Deferred.

At this point it feels like the replacement shader functionality is just ignoring all stencil properties (except Ref for some reason?). I can find small posts by staff that says “all properties present in the original material are present in the replacement” but no way of actually checking this is working, nor any actual documentation on how properties for replacement shaders should work.

2

I bit the bullet and made a tiny standalone test, and I’m pretty sure Unity is broken in this aspect. I suspect some common code with the deferred renderer is explicitly disabling stencil properties even for DepthNormal rendering in Forward.

Replacement shaders never get their stencil properties set - that applies to the DepthNormals pass (which has a replacement shader set at the project level) or when manually rendered via Camera.RenderWithShader. Which means that there’s no way to manually render the DepthNormal texture via replacement rendering.

The DepthTexture pass seems immune, because it doesn’t use a replacement shader, but instead an explicit pass inside the original shader (the ShadowCaster pass).

Tested and has the same issue on 2019 LTS and 2021 LTS.

3

Builtin deferred does not work properly with stencils. It’s probably not going to be fixed. SRPs might be your only choice, which of course is not really an option for an established project.

You can try submitting a bug report, but it’s like a 10% chance they’ll fix it.

4

This is with Built-in Forward. (I did mention this above, but probably not as obvious as I could have been).