Subracting using the stencil buffer (getting mesh intersections)

Unity Engine
,
1

I’m trying to figure out if it’s possible to take ‘bites’ out of a scene using the stencil buffer and URP.

Something like this.

7879717--1002232--Screenshot 2022-02-08 193900.png
7879717--1002232--Screenshot 2022-02-08 193900.png1850×840 56.7 KB

This is a little different from a lot of the general ‘hole cutting’ articles as it should be consistent in world space. in most examples I’ve found what is masked is affected by the camera’s position.

7879717--1002238--unknown.png
7879717--1002238--unknown.png1140×361 15.5 KB

This is as far as I’ve gotten, which is not what I want. This is the wrong scenario illustrated above.

7879717--1002241--Stencil.gif

Thanks in advance!

2

It’s possible to do 3D boolean operation rendering using stencil buffer techniques, it’s not quite trivial though. Here you have an indepth article describing how to do it: 3.4 Constructive Solid Geometry with the Stencil Buffer

3

I’m actually trying to do the same exact thing right now haven’t had any luck either just posted my own help question and then noticed this underneath…oops… but I have found some things with no luck I linked them in my post.

https://discussions.unity.com/t/871440

Hopefully we can help each other

4

So after further consideration, I don’t this is possible using the stencil buffer. Instead, I’ve created a shader that clips based on world position.

7892110--1004617--SphereClip.gif

7892110--1004620--upload_2022-2-13_2-38-6.jpg

7892110--1004623--upload_2022-2-13_2-42-39.png

5

if we can just invert that sphere we might have we wanted.

6

I believe it is possible, although it’s a PITA to do. This technique was used to draw shadows in some old games (or newer VR games sometimes). The area “inside” of the faces of an object can be roughly rendered to the SB via something like this:

            Cull Off
            ZTest Always
            ZWrite Off
            Stencil { PassFront IncrWrap PassBack DecrWrap }
            ColorMask 0 // prevents drawing the actual object

The stencil starts at 0. When it passes into the object, it increments (1). When it passes out, it decrements (0). If it goes in the wrong way, it’ll decrement-wrap with an 8-bit integer (0 - 1 = 255) and then increment-wrap (255 + 1 = 0). Ummm… sorry if that doesn’t make much sense. To render things that are between the two faces, add this:

           Stencil { Comp NotEqual Ref 0 }

Neat! Now that’s good enough to render things that are inside of one object (or a set of objects), which is enough for stencil shadows since things will either be in shadow or not in shadow. It’s not good enough for handling things that are “in front of object X but behind Y”, but you have 8 bits to play with.

So let’s try this (warning: early morning code that’s totally untested):

// render the blue object with
            Cull Off
            ZTest Always
            ZWrite Off
            Stencil { PassFront Invert PassBack Invert WriteMask 1 }
            ColorMask 0 // prevents drawing the actual object

// and the green object with
            Cull Off
            ZTest Always
            ZWrite Off
            Stencil { PassFront Invert PassBack Invert WriteMask 2 }
            ColorMask 0 // prevents drawing the actual object

// and then the interesection of the two would be...
           Stencil { Comp Equal Ref 3 }

// the inverse would be
          Stencil { Comp NotEqual Ref 3 }

Now there are some caveats to this

  1. The above assumes the camera is not inside either stencil mesh. Supporting that is going to be a bit more of an issue (understatement of the century).
  2. This ignores depth testing, which I think is right for your situation but it depends on your use case.
  3. The meshes must be manifold. UV and normal seams are fine, but there can be no intersecting/inside geometry, no holes, no unconnected sides, etc. Cubes and spheres will work, but more complex geometry… I don’t even want to talk about some of the horrors I went through when trying stencil shadows.
  4. Adding to the above, using invert instead of incrwrap/decrwrap is even scarier since now you can’t “enter twice, leave twice” – no objects that are part of the same stencil layer can ever intersect!!! That basically means one object per layer, and likely no skinned/animated meshes unless you audit all the animations to ensure no self-intersection. So yeah… anything more than intersecting a single sphere with a single cube is “at your own risk”. Once you have the stencil, the rest can be rendered h
  5. It won’t work out of the box on HDRP, since HDRP uses the stencil buffer for rendering. It does have a couple user bits (32 and 64 are safe to use don’t touch the rest).
  6. Every single shader you want to render inside needs that mask. If you actually are rendering a game with tons of shaders, you may want to consider rendering everything then hiding the outside areas with a post-process effect or using render targets; yeah it’ll be slower but you’ll have a lot more control and less complexity.

EDIT: If you’re set on going down this path, you can take a look here at my now-aborted attempt at stencil shadows: https://gitlab.com/burningmime/urpg-public/-/tree/master/old/shadow . It’s not the same as your use case, but it’ll maybe give you an idea of stencil buffer usage.

1 Like
7

I’m a bit late to this thread, but a tool I’ve been using in one of my more recent projects is this: Unity でスクリーンスペースのブーリアン演算をやってみた - 凹みTips.
The webpage is in Japanese so I would recommend the google translate extension for chrome, and it actually translates fairly well.
The source code is linked on the website but I’ll link it here as well: GitHub - hecomi/UnityScreenSpaceBoolean: Screen Space Boolean Implementation for Unity.

The only issue is that it does not actually alter the collider of the subtractee, which isn’t an issue in my case but may be for yours.


8

Hey Richard, I’m very curious. Do you have the complete shader that you posted in your last reply? I need to do something similar, and it’s kind of difficult for me.