Reading a RenderTexture's stencil buffer in a full screen effect shader

Unity Engine
, ,
1

I feel like an idiot for not figuring this one out, but I keep going in circles. Here’s what I’m trying to do:

  • Render the scene once, with a set of objects turned off, to a RT
  • Render the scene again, with these objects turned on, to a RT, but including the stencil buffer (the special objects write to the SB)
  • For the final frame, composite the two RT results, using the second RT’s stencil buffer as the mask

I’ve tried a bunch of things, but the one I landed on that seems to get me closest is this:

    void LateUpdate() {
       RenderTexture originalTargetTexture = camera.targetTexture;
       RenderTexture originalRenderTexture = RenderTexture.active;

       //Setup for render 1
       Capture(rt1);

       //Setup for render 2
       Capture(rt2);

        camera.targetTexture = originalTargetTexture;
        RenderTexture.active = originalRenderTexture;
    }

    void Capture(RenderTexture rt) {
        Camera camera = Camera.main;
        RenderTexture.active = camera.targetTexture = rt;
        camera.Render();
    }

    void OnRenderImage(RenderTexture source, RenderTexture destination) {
        material.SetTexture("_Tex1", rt1);
        material.SetTexture("_Tex2", rt2);

        Graphics.Blit(source, destination, material);
    }

However, I have no idea how to access rt2’s stencil buffer inside material’s shader. Accessing the stencil buffer directly doesn’t work (I suspect it’s cleared between LateUpdate and the render time) and I can’t find a way to extract it from _Tex2 somehow.

What I missing?

2

Don’t worry, you’re not an idiot. The reason why it seems like this is so hard is fairly simple.

It’s not possible.

The stencil buffer can only be interacted with via the fixed function stencil operations. That’s it. It can’t be sampled from within a material via a texture.

As for it being “cleared” between passes. That’s not quite accurate. You’re right it’s not there any longer, but that’s because it’s tied to the depth buffer that was in use. And most render textures don’t have a depth buffer of any kind, and therefore no stencil buffer.

Traditionally depth buffers were either 16 bit floats, or 24 bit floats. Someone figured out they could use the other 8 bits for stencils and keep things in a 32bit alignment. Today depth buffers are 32bit floats, and the 8 bit stencil is separate, but the APIs never changed to reflect that so they’re still intrinsically tied together.

If you want to use a stencil buffer, you need to create one with your render texture, and then if you want to reuse it with a different color render texture, you need to assign that specific depth buffer as the render target.

3

First off, glad to know it’s not me :stuck_out_tongue: Thank you so much for your explanation! You lost me a bit at the end though.

The Render Textures I was using did have their depthStencilFormat set (and were created with a 24-bit depth buffer size). Can I use that texture’s buffer, or do I need a separate one?

And once I have the buffer in C#-land, how do I pass it back to the effect shader?

4

Replace camera.targetTexture = RT; with:
camera.SetTargetBuffers(RT.colorBuffer, depthRT.depthBuffer);
where the depthRT is the RT you want to use the depth & stencil buffer from. But this only works if they have exactly the same resolution.

5

I’ve tried this, but replacing targetTexture = ... with SetTargetBuffers causes my editor to bug (multiple copies of the screen buffer appearing all over), and still does not show the correct stencil mask.

I’m confused on how this should work though. I render my scene into RT1 and RT2, with RT2 holding the stencil buffer that I then want to use in my full-screen effect.

I figured I might need to set the target buffer before the final image gets rendered, but that also gave me an empty stencil.

6

Going back to this question:

Again, you don’t. That is impossible. All you can do is have a shader with a Stencil {} block that does the fixed function logic to skip rendering that shader at certain pixels.

To use a stencil with a Graphics.Blit(), you also can’t. See the documentation’s comment about this at the bottom of this page:

The documentation for SetPass() offers an example of how to do this:

Though this gist has a more complete example:

However, here’s how I’d do what you’re trying to do, which avoids a lot of the above headache with trying to pass around stencil buffers.

  • Render RT1 to a render texture as you already are.
  • Create a command buffer to blit RT1 to the screen (BuiltinRenderTextureType.CameraTarget using a stencil-masked material and add it to the camera used to render RT2. This still respects the currently active stencil buffer.
  • Render “RT2”.
  • Remove the command buffer from the camera.

This skips the whole trying to swap the depth buffer around as when RT1 is rendered onto RT2 the stencil is still active. However if you want to do anything to the “RT2” render texture then this obviously won’t work quite like you want.

The other option is I’d skip the stencil part entirely and use the alpha of RT2 as the mask. This can either be done by rendering the RT2 scene with a blank black & alpha 0.0 background, or you can use that command buffer blit trick to clear the background to black & alpha 0.0 after everything else is rendered if you need to keep the rest of the scene visible (ie: for shadows etc.).

1 Like