To my knowledge, Unity doesn’t allow this. There are numerous workarounds, although without editing any shaders the options for using the stencil buffer are rather limited. Unfortunately, these workarounds really aren’t using the stencil buffer, but can be used in some cases to achieve a similar effect. Some of the workarounds I can think of:
Multiple Cameras
This is the quick, ugly solution, and has some severe limitations, but is the easiest to pull off. It’s the same trick you might use when making a first person game to keep hands/weapons from clipping through walls. Just render the same scene multiple times from multiple cameras with different culling masks, where parts of the scene have different layers.
Pros:
Easy to setup and extend
Requires no additional shaders
Cons
Only works on a per-object level
May require clever level design to mask the many edge cases that break the illusion
Using the Depth Buffer
While not the exact same thing as the stencil buffer, these two buffers often perform similar roles. If you just want to mask off certain areas, you can take advantage of ZTest. This requires writing a single shader. Using the SV_Depth
frag output semantic, you can set a custom depth. Writing a shader that renders after everything else, writes nothing to the color buffer (you probably have to use blending rather than discard/clip), and writes 1 into the depth buffer, essentially clears depth locally wherever that object is rendered. Then, when you do the next render, don’t clear the depth buffer. Then, in the spaces where the depth buffer was overwritten, the new objects will always pass ZTest.
Pros:
Provides per-pixel masking
Can be used a bit more creatively/artistically
Cons:
Requires writing a shader
Doesn’t allow super complicated operations like stencil does. Can only be used for masking.
I apologize for the fact that both of these workarounds have serious disadvantages. The fact is, without using a RenderTexture and PostFX to implement your own version of stencil, you can’t really get all the functionality of stencil buffers. On the other hand, you can mix these two workarounds together to get effects that aren’t even easily feasible with stencil. I’ve used the depth buffer trick to make nice portal effects in the past without needing a bunch of RTs (although it would require rendering multiple times to the same RT, which happened to pretty much just be the screen).