Set global stencil test regardless of shader (command buffer?)

Questions & Answers
1

Hey! I need to redraw the same scene a couple of times with a different stencil mask clipping everything in the scene each time. All directly to screen on top the previous render; no render textures or post effects or any such thing involved. I would like to set up a global stencil test to apply to absolutely everything before each render.

A few options that you might have in mind that I cannot use and why:

  • Custom pipeline: This is the only solution for which I’ve actually found a way in the documentation to directly set the stencil state, but I don’t have the time, experience or desire to go with this when all I want to do is hook into the standard pipeline and do one little thing, and I think we’re stuck on too old a version of Unity anyway.
  • Shaders: I know that I can set up a stencil test in a shader, but I’m new on a project that’s already full of shaders, both regular and from ShaderForge, as well as ones that belong to various plugins such as MegaSplat, so I simply don’t have the level of control over the shaders used this project that I can just go in and edit all of them anyway. Thus I cannot use shader global variables, because that would entail modifying every shader to add a stencil test that reads it. I also cannot use replacement shaders because then I would have to reimplement every shader anyway.
  • Post effect: Some people seem to have misunderstood my question and suggested various things related to this. Not sure what’s wrong with my wording, but this has nothing to do with those, sorry. :stuck_out_tongue:

If I was working on a pure OpenGL project I would just drop glStencilFunc() and glStencilOp() in there right before rendering the scene, so that’s basically how I would like to solve this in Unity too, if at all possible, whether that be by going through the camera’s OnPreRender() and OnPostRender() using some entry point in the Unity C# API to access the corresponding functions, or if I can somehow use command buffers to hook in there, but I haven’t find any such option for those in the documentation…

Thanks!

2

To my knowledge, Unity doesn’t allow this. There are numerous workarounds, although without editing any shaders the options for using the stencil buffer are rather limited. Unfortunately, these workarounds really aren’t using the stencil buffer, but can be used in some cases to achieve a similar effect. Some of the workarounds I can think of:

Multiple Cameras

This is the quick, ugly solution, and has some severe limitations, but is the easiest to pull off. It’s the same trick you might use when making a first person game to keep hands/weapons from clipping through walls. Just render the same scene multiple times from multiple cameras with different culling masks, where parts of the scene have different layers.

Pros:

Easy to setup and extend

Requires no additional shaders

Cons

Only works on a per-object level

May require clever level design to mask the many edge cases that break the illusion

Using the Depth Buffer

While not the exact same thing as the stencil buffer, these two buffers often perform similar roles. If you just want to mask off certain areas, you can take advantage of ZTest. This requires writing a single shader. Using the SV_Depth frag output semantic, you can set a custom depth. Writing a shader that renders after everything else, writes nothing to the color buffer (you probably have to use blending rather than discard/clip), and writes 1 into the depth buffer, essentially clears depth locally wherever that object is rendered. Then, when you do the next render, don’t clear the depth buffer. Then, in the spaces where the depth buffer was overwritten, the new objects will always pass ZTest.

Pros:

Provides per-pixel masking

Can be used a bit more creatively/artistically

Cons:

Requires writing a shader

Doesn’t allow super complicated operations like stencil does. Can only be used for masking.

I apologize for the fact that both of these workarounds have serious disadvantages. The fact is, without using a RenderTexture and PostFX to implement your own version of stencil, you can’t really get all the functionality of stencil buffers. On the other hand, you can mix these two workarounds together to get effects that aren’t even easily feasible with stencil. I’ve used the depth buffer trick to make nice portal effects in the past without needing a bunch of RTs (although it would require rendering multiple times to the same RT, which happened to pretty much just be the screen).