Stencil Decal shader. Help with proper projection?

Posted this in Scripts as that’s where I usually post stuff but then remembered there’s actually a shader section, so my apologies for the multi-post.

I recently made a mesh-based decal system and its working very well and I’m pretty happy with it. After reading this article, though, I wanted to try a different method. I played around with this and then got sidetracked by an idea to use the stencil buffer. I got it to a sorta working point:

1606157--97306--$decal_working.jpg
This is using the following shader on a cube.

Shader "Custom/SSDecal" {

    Properties {

        _MainTex ("Base (RGB)", 2D) = "white" {}

    }

    SubShader {

        Tags { "RenderType"="Opaque" "Queue"="Geometry+1"}

        ColorMask 0

        ZWrite off

        Stencil {

            Ref 1

            Comp always

            Pass replace

        }

 

        CGINCLUDE

            #include "UnityCG.cginc"

            

            sampler2D _MainTex;

            

            struct appdata {

                float4 vertex : POSITION;

            };

            struct v2f {

                float4 pos : SV_POSITION;

                float2 uv : TEXCOORD0;

            };

            v2f vert(appdata v) {

                v2f o;

                o.pos = mul(UNITY_MATRIX_MVP, v.vertex);

                float4 projPos = mul(UNITY_MATRIX_MVP, v.vertex);

                float4 posWorld = mul(_Object2World, v.vertex);

                float4 posObj = mul(_World2Object, posWorld);

                

                posObj += 0.5;

                o.uv = posObj.xz;

                

                return o;

            }

            half4 frag(v2f i) : COLOR {

                fixed4 c = tex2D(_MainTex, i.uv);

                return c;

            }

        ENDCG

 

        Pass {

            Cull Front

            ZTest Less

        }

        Pass {

            Cull Back

            ZTest Greater

        }

        

        Pass {

            Tags { "RenderType"="Opaque" "Queue"="Geometry+2"}

 

            ColorMask RGB

            Cull Front

            ZTest Always

            Stencil {

                Ref 1

                Comp notequal 

            }

 

            CGPROGRAM

            #pragma vertex vert

            #pragma fragment frag

            ENDCG

        }

    } 

}

The problem I’m running into is that because this is all mapped on the cube I end up with issues like this:
1606417--97340--$decal_moveDown.jpg

Where, because it’s mapped on the cube and not the ground, it sinks into the ground. And:
1606157--97308--$decal_inSphere.jpg

Where, when intersecting other objects, it does draw over top of them correctly, but mapped to the bottom of the cube. not over the object like this:
1606417--97342--$meshDecal.jpg

That’s the mesh decal and that mapping is what I’m looking for in this.

I tried many different methods for adjusting the uv’s but simply couldn’t get it to work properly. I thought maybe I could use the depth texture to reconstruct world space uv’s and use that but when I plugged in the camera depth texture I got the depth to the casting cube (even though it was set to ZWrite Off) like so:
1606157--97310--$depth.jpg

What I need is the depth of the objects. I thought maybe this could be because I was doing that depth test in the final pass and that I should do a pre-pass that gets the scene depth but I have a less-than-basic knowledge of doing multi-pass shaders and how and where to store said depth test.

Anyway, anyone have any ideas/critiques/warnings that this is a crazy idea and will never work never ever?

Getting the proper uv coordinates is tricky when using a stencil volume. But, with the depth values, you should be able to recalculate the world position and from that map it back to the cube. Like you are trying to do.

I’m not sure the passes now do what they should do. The z write is now disabled in the subshader, but not in the first two passes. You also switch between greater and less z test, which is the not the way to do stencil marking as far as I know.

You want to either use greater or less for both passes. And switch between incr and decr as the stencil pass function, while culling either front or back. The idea of this is to count how many times the view ray enters and exits the volume before (or after) it hits the current geometry. Ending up with 1 where still in the volume and 0 when outside.

It’s also possible to combine the back and front passes in a single pass. In that case you disable culling, set the stencil mask to 1 and the pass function to incr. The first bit will then flip every time you enter or exit the volume, leading to the same 1 when still inside behaviour.

Some more information on it can be found here:
http://en.wikipedia.org/wiki/Shadow_volume