I’ve got an object in my scene that I use the stencil buffer to mask out of certain regions. Here’s the bit in my shader:
Stencil
{
Ref[_StencilRef]
Comp LEqual
}
I draw an invisible mesh (ColorMask 0) where I want the stenciled object to be visible on a render queue ahead of the rest of the scene geometry. Anywhere outside of this invisible mesh will not show my stenciled mesh. This all works fine!
One thing I’ve noticed though that isn’t working properly is that transparent objects are interacting strangely with the areas in the scene where the stenciled object is, but has been prevented from rendering to. Kind of hard to explain so here are some pics…
The orange outline is the stenciled mesh (invisible because its not inside the region where the stencil mask is drawn).
Here with it de-selected, it’s easy to see that it is intersecting in some way with these two transparent objects, even though it’s not being drawn.
Here I’ve turned the MeshRenderer on the stencil mesh off so you can see how they should be appearing.
I thought maybe it was drawing to the depth buffer even though it wasn’t drawing fragments to the screen, but it doesn’t seem to have any effect on opaque objects. So I don’t really understand what’s going on here. Any advice?
The affect the depth buffer has on the scene still depends on the draw order. Transparent queue objects draw after all opaque objects, so it won’t affect opaque objects at all if a transparent object draws to the depth, but it will affect any transparent objects that are drawn afterward.
The stenciled object is opaque geometry. The stencil is being used to mask it out of a certain area of the scene. Its sort of like a column that comes out of another piece of opaque geometry that couldn’t actually physically contain the column.
it seems weird to me that if a fragment is prevented from drawing to the screen by the stencil, it still in some way appears to do something with the depth buffer. If its discarded, it shouldn’t, no?
It’s kind of irrelevant if it’s drawing during the opaque or transparent queues. All that maters for stencils is the stencil mask has to draw before the objects it is masking. That’s going to be true regardless of if the objects are opaque or transparent.
I’m confused. When a stencil test prevents an object from rendering, it does indeed prevent any interactions with the depth buffer. However that’s not what’s happening here. Your stencil mask object is itself drawing to the depth buffer. It isn’t being prevented from rendering to depth, it’s not even being prevented from running the fragment shader. It would be a little weird if the stencil for an object prevents itself from rendering (though that is possible with overlapping geometry).
Also ColorMask 0 just says to ignore the output of the fragment shader, it still runs the fragment shader, and doesn’t prevent rendering to the depth buffer. That allows things like alpha testing via clip() or discard to function, as well as any other special output semantics like SV_Depth or SV_StencilRef to modify where and what is written to the depth and stencil. Even if the shader doesn’t have a fragment shader, which is allowed by Direct3D and OpenGL, though not directly supported by Unity, this doesn’t prevent it from writing to the depth or stencil. Indeed if a shader doesn’t have a fragment shader or does have a fragment shader but it doesn’t use alpha testing or the special output semantics, the depth and stencil are written to before the fragment shader runs.
BTW, the stencil and the depth buffer are the same thing. It’s a single 32 bit buffer where the first 24 bits are used by the depth and the last 8 are used by the stencil*.
Except on some hardware that uses a true 32 bit depth buffer and has the stencil as a separate 8 bits, so a single “24 bit depth buffer” actually uses 40 bits on the hardware.
Sorry for the confusion… that’s not actually how I have it set up. The stencil object is set up in the opposite way: the masked object can ONLY draw where the stencil object has changed the value in the stencil buffer. So its definitely not the stencil object directly causing the problem in my screenshots… its not even in the frame.
wow yes! thank you… it’s complex shader from the asset store that I modified, but yes looks like i had forgotten to add the stencil bit to the shadow pass. i put it in and now it looks right! thank you for helping me track down the problem
argh… i spoke too soon… now that i’ve added the stencil to the shadow pass, it looks like some faces are now drawing transparently or something… hard to actually tell what’s happening. may need to abandon this stencil thing
