# Subracting using the stencil buffer (getting mesh intersections)

I’m trying to figure out if it’s possible to take ‘bites’ out of a scene using the stencil buffer and URP.

Something like this.

This is a little different from a lot of the general ‘hole cutting’ articles as it should be consistent in world space. in most examples I’ve found what is masked is affected by the camera’s position.

This is as far as I’ve gotten, which is not what I want. This is the wrong scenario illustrated above.

It’s possible to do 3D boolean operation rendering using stencil buffer techniques, it’s not quite trivial though. Here you have an indepth article describing how to do it: 3.4 Constructive Solid Geometry with the Stencil Buffer

I’m actually trying to do the same exact thing right now haven’t had any luck either just posted my own help question and then noticed this underneath…oops… but I have found some things with no luck I linked them in my post.

https://discussions.unity.com/t/871440

Hopefully we can help each other

So after further consideration, I don’t this is possible using the stencil buffer. Instead, I’ve created a shader that clips based on world position.

if we can just invert that sphere we might have we wanted.

I believe it is possible, although it’s a PITA to do. This technique was used to draw shadows in some old games (or newer VR games sometimes). The area “inside” of the faces of an object can be roughly rendered to the SB via something like this:

``````            Cull Off
ZTest Always
ZWrite Off
Stencil { PassFront IncrWrap PassBack DecrWrap }
ColorMask 0 // prevents drawing the actual object
``````

The stencil starts at 0. When it passes into the object, it increments (1). When it passes out, it decrements (0). If it goes in the wrong way, it’ll decrement-wrap with an 8-bit integer (0 - 1 = 255) and then increment-wrap (255 + 1 = 0). Ummm… sorry if that doesn’t make much sense. To render things that are between the two faces, add this:

``````           Stencil { Comp NotEqual Ref 0 }
``````

Neat! Now that’s good enough to render things that are inside of one object (or a set of objects), which is enough for stencil shadows since things will either be in shadow or not in shadow. It’s not good enough for handling things that are “in front of object X but behind Y”, but you have 8 bits to play with.

So let’s try this (warning: early morning code that’s totally untested):

``````// render the blue object with
Cull Off
ZTest Always
ZWrite Off
Stencil { PassFront Invert PassBack Invert WriteMask 1 }
ColorMask 0 // prevents drawing the actual object

// and the green object with
Cull Off
ZTest Always
ZWrite Off
Stencil { PassFront Invert PassBack Invert WriteMask 2 }
ColorMask 0 // prevents drawing the actual object

// and then the interesection of the two would be...
Stencil { Comp Equal Ref 3 }

// the inverse would be
Stencil { Comp NotEqual Ref 3 }
``````

Now there are some caveats to this

1. The above assumes the camera is not inside either stencil mesh. Supporting that is going to be a bit more of an issue (understatement of the century).
2. This ignores depth testing, which I think is right for your situation but it depends on your use case.
3. The meshes must be manifold. UV and normal seams are fine, but there can be no intersecting/inside geometry, no holes, no unconnected sides, etc. Cubes and spheres will work, but more complex geometry… I don’t even want to talk about some of the horrors I went through when trying stencil shadows.
4. Adding to the above, using invert instead of incrwrap/decrwrap is even scarier since now you can’t “enter twice, leave twice” – no objects that are part of the same stencil layer can ever intersect!!! That basically means one object per layer, and likely no skinned/animated meshes unless you audit all the animations to ensure no self-intersection. So yeah… anything more than intersecting a single sphere with a single cube is “at your own risk”. Once you have the stencil, the rest can be rendered h
5. It won’t work out of the box on HDRP, since HDRP uses the stencil buffer for rendering. It does have a couple user bits (32 and 64 are safe to use don’t touch the rest).
6. Every single shader you want to render inside needs that mask. If you actually are rendering a game with tons of shaders, you may want to consider rendering everything then hiding the outside areas with a post-process effect or using render targets; yeah it’ll be slower but you’ll have a lot more control and less complexity.

EDIT: If you’re set on going down this path, you can take a look here at my now-aborted attempt at stencil shadows: https://gitlab.com/burningmime/urpg-public/-/tree/master/old/shadow . It’s not the same as your use case, but it’ll maybe give you an idea of stencil buffer usage.

1 Like

I’m a bit late to this thread, but a tool I’ve been using in one of my more recent projects is this: Unity でスクリーンスペースのブーリアン演算をやってみた - 凹みTips.
The webpage is in Japanese so I would recommend the google translate extension for chrome, and it actually translates fairly well.
The source code is linked on the website but I’ll link it here as well: GitHub - hecomi/UnityScreenSpaceBoolean: Screen Space Boolean Implementation for Unity.

The only issue is that it does not actually alter the collider of the subtractee, which isn’t an issue in my case but may be for yours.

Hey Richard, I’m very curious. Do you have the complete shader that you posted in your last reply? I need to do something similar, and it’s kind of difficult for me.