How do I use the stencil buffer with multiple objects?

Unity Engine
1

First a note: I’ve used stencil buffers extensively in hlsl so I don’t need a rundown on how those work or what they do.

So my problem is that I have two objects, one is in front, but I need to render the other on top of it. I know unity UI uses stenciling, so I don’t trust that my stencil buffer is clear when I go to render these two objects, and I don’t even know if the object I want to make a stencil out of will render before the other.

Any additional information is welcome, but I’m mainly looking to answer these questions:

  1. How do I guarantee the stencil buffer is clear when I go to draw my first object?
  2. Do I have to alter the RenderQueue in code to make sure these two objects render in the correct order?
  3. How do I know I only have the stencil drawn from the one intended object when I go to use the stencil in the second?

Thanks!

2

bump

3

  1. You don’t need the entire stencil to be clear, you can use read and write masks to focus only on one bit. For example, if you want to check whether the first bit is set and don’t care about the rest, use ReadMask 1 Ref 1. Same with writing: WriteMask 4 Ref 4 sets only the third bit and doesn’t touch the rest. Unity only uses the last bits, so you should be safe if you don’t use those.
    Anyway, the depth & stencil buffer is cleared at the beginning of every frame, if your camera already clears depth. If you use the deferred pipeline, Unity also uses the last bit of stencil for the lighting pass, which is not cleared afterwards by default (Camera.clearStencilAfterLightingPass). You might end up with a dirty stencil buffer if you draw your objects after the ui, though…

  2. That, or by changing the Queue tag in the shader itself

2?) You don’t. If you want to differentiate between multiple objects drawing to stencil, you need to either set a different stencil value for each object (if they can overlap, different bits) or make your read pass reset the stencil and draw it right after your write pass, probably using command buffers.

4

Thanks Dolkar!

I see in the frame debugger the only time the stencil buffer is cleared is at the beginning of RenderOpaqueObjects. I’m still figuring out how unity handles all it’s graphics… but it seems that possibly each camera has it’s own stencil buffer, or perhaps clear the stencil buffer between cameras if/when depth is cleared.

I also have a suspicion that world-space canvases render to a rendertexture and screen-space canvases render directly to the backbuffer, and I also think (not sure) that they clear the stencil buffer or have their own. (Could factor into why canvases are obnoxiously GPU intensive for what they do)

Anyway, I’ve got my desired effect working pretty well, thanks again!