Stencil buffer problems using CommandBuffer.DrawMesh with deferred rendering

Unity Engine
1

Hello everyone! I’m new to the forums, so please redirect me if this is not the place to ask.

I’m playing around with CommandBuffers and deferred rendering, and to get a feel for it I wanted to try and just do a basic test. Basically, render a simple mesh with CommandBuffers in deferred rendering, and have it look the same as it would if you rendered it with a MeshRenderer. I’ve hit a road-block. This is my tiny test scene:

1092×648 8.11 KB

The mesh on the right is rendered with a MeshRenderer, the mesh on the left is rendered with CommandBuffers. The material is just a simple material with the Standard shader and a slightly yellow albedo (no textures).

So, I’ve been digging into the Frame Debugger, and I think I know what’s going on: everything goes just fine until I get to the “RenderDeferred.Lighting” stage, at which point it seems to totally ignore the mesh drawn with CommandBuffers. These are the passes in the Frame Debugger for the MeshRenderer and CommandBuffer meshes (I turned off both shadows and deferred reflections to simplify the rendering):

2212×1178 270 KB

2212×1178 264 KB

As far as I can tell, the only difference in rendering these is that the “normally rendered” mesh enables writing to the stencil buffer. It makes sense that this is the problem: the reason my mesh is ignored during the Lighting pass because it’s outside of the stencil. Testing this problem, I figured I would “trick” the stencil buffer by putting the correctly rendered mesh behind my mesh, so that at least some of it would be inside the stencil:

It’s a bit tricky to see, but it works: the parts of my CommandBuffer mesh that is directly in front of the other bird is rendered correctly, the rest is not. It seems clear that the stencil buffer is at fault.

Clearly, CommandBuffer.DrawMesh does not set whatever stencil settings is needed for deferred rendering. I’ve been digging through the docs and googling around, but I haven’t found any answers to this. Is this a bug, or is there some way I can tell the command buffer to write to the stencil buffer as part of rendering the G-buffer?

Here’s my code for setting up the CommandBuffer, btw:

buf = new CommandBuffer();
buf.name = "Draw meshes";

// This is the only material keyword the MeshRenderer
// sets that isn't set when using the command buffer,
// so set it here. 
Material.EnableKeyword("SHADOWS_SHADOWMASK");

// Simplest possible DrawMesh. The last argumenet (the
// 3) is the Deferred pass in the shader. Mesh and material 
// are provided as fields set in the editor. 
buf.DrawMesh(mesh, Matrix4x4.identity, mat, 0, 3);

cam = GetComponent<Camera>();

cam.AddCommandBuffer(CameraEvent.AfterGBuffer, buf);

I’ve tried a bunch of variants (different camera events, etc.) but I haven’t found anything that works. Help would be much appreciated!

2

Bump. Anyone got any ideas?

3

Unity’s deferred renderer stomps on a lot of stencil stuff. A few years ago there was some wiggle room there, but it seems in the last year or so they completely ignore anything stencil related in the shader files during the GBuffer passes.

If you want to play with stencils, you have to use the forward renderer. Either completely disable the deferred renderer, or use shaders with no deferred pass and render them after the gbuffer pass. If you want to affect objects that render to the gbuffer you’ll likely have to solve the problem using a completely different approach, like rendering a mask to a render texture prior to the gbuffer passes and sampling that texture in a custom shader to clip an object.

4

I used RenderDoc to figure out what Unity was doing with the stencil buffer. Putting these stencil settings in the shader worked for me, at least in CameraEvent.AfterGBuffer.

Stencil {
        Ref 192
        ReadMask 255
        WriteMask 207
        Comp Always
        Fail Keep
        ZFail Keep
        Pass Replace
}
2 Likes
5

Thanks so much for the tips! I’ll definitely try this out.

6

Thanks, this worked for me in 2021.3 for a BeforeGBuffer event. I’ve also suggested this as missing information in the CameraEvent.BeforeGBuffer documentation.