Oof, attempting image based CSG in Unity using modified layered Goldfeather

Unity Engine
1

So I’ve been attempting to follow this paper here in order to perform some basic CSG subtraction in my game. Unfortunately, my shader knowledge is a little bit on the lacking side when it comes to things like the stencil buffer, depth buffer, and working with both.

What I guess I’m really looking for here is something to guide me through using the stencil buffer, especially for the operations contained in this paper. As I mentioned before, I’m a bit out of my depth, but can usually pick things up with decent training materials. Everything I’ve found relating to the stencil buffer at length has been either complete projects with little documentation, or minor things involving the render queue.

Of course, there’s the other question that looms in the back of my mind: are the contents of this paper even possible in Unity at all?

Any help would be greatly appreciated.

edit: to clarify, I do understand the concept I’m working from. Thankfully that part is easy. It’s just that I have no idea how to handle the implementation.

2

So I’ve been digging into Unity’s renderer a bit more, throwing me even further out of my depth, but it seems this might be Very Impossible with Unity? It seems to rely a lot on me getting the depth of an object even if it’s occluded to do the parity comparisons, and I’m not entirely sure that’s doable?

Hopefully there’s a shader expert who knows more than me on this topic.

3

I’m still poking at this, but the more I do, the more I realise that this is either not possible in Unity or almost prohibitively complex given how Unity works. Right now here’s my problems, as I see them at least:

  • There seems to be no real way to get per-object depth when the object is occluded by another object unless I use multiple cameras
  • There also seems to be no real way to render depth in layers to handle concave objects. This isn’t too much of a problem as I’m not using any concave objects when constructing my levels. This does lead to a fair few more brushes being used than I’d like.
  • If I could get it to work, I could probably get away with just slightly modifying standard Goldfeather to be more Unity compliant, but I’m having issues even getting that to work with my limited experiences with the stencil buffer. Everything I’m looking at has me using scripts which leaves me a frame behind, which isn’t very good for what is going to be a fast-paced shooter.

In the end, would it just be easier for me to write my own BSP CSG system a la GeoMod 1.0 and find a way to render that out to polygons in Unity?

4

So as I experimented further and further I discovered some things:

  • Image based CSG is totally possible in Unity!
  • It sucks butts though.

I managed to get something kinda working

1183×676 115 KB

As you can see it’s not really taking into account the “solid” part of Constructive Solid Geometry, but that’s not my problem. My problem is that the minute you put the camera in the hole, the hole disappears! Before anyone says anything in this super active thread I should have realised this myself, yes. Since it’s image based, of course the camera can’t intersect with it. I’m not sure why I ever thought it could.

But I’m not giving up just yet. I have some fancy ideas involving the ever patent encumbered Carmack’s Reverse, only for cutting holes in things. I wish I didn’t have to go to that end, but that’s where the situation has gotten me, unfortunately.