Applying a stencil mask to an intersection

Unity Engine
,
1

Hi there,

So, I have these 2 shader files, one called Stencilled.shader and one called StencilMask.shader. When the stencil mask is applied to the material of an object, it allows the user to look through it and reveal objects with the Stencilled.shader applied to them.

I have another shader file called Intersect.shader which allows the user to apply it to an object and then intersect the object with another object to highlight the intersection.

What I’m trying to do is basically combine both the StencilMask.shader and the Intersect.shader files so that the intersection of the objects becomes a stencil mask which reveals objects with the Stencilled shader applied to them (if that makes sense).

Shader "Custom/StencilMask" {
    Properties {
        _StencilMask("Stencil mask", Int) = 0
    }
    SubShader {
        Tags {
            "RenderType" = "Opaque"
            "Queue" = "Geometry-100"
        }
        ColorMask 0
        ZWrite off
        Stencil {
            Ref[_StencilMask]
            Comp always
            Pass replace
        }
        Pass {
            CGPROGRAM
            #pragma vertex vert
            #pragma fragment frag
            struct appdata {
                float4 vertex : POSITION;
            };
            struct v2f {
                float4 pos : SV_POSITION;
            };
            v2f vert(appdata v) {
                v2f o;
                o.pos = UnityObjectToClipPos(v.vertex);
                return o;
            }
            half4 frag(v2f i) : COLOR {
                return half4(1, 1, 0, 1);
            }
            ENDCG
        }
    }
}

Shader "Custom/Stencilled" {
    Properties {
        _StencilMask("Stencil mask", Int) = 0
        _Color ("Color", Color) = (1,1,1,1)
        _MainTex ("Albedo (RGB)", 2D) = "white" {}
         _Glossiness ("Smoothness", Range(0,1)) = 0.5
         _Metallic ("Metallic", Range(0,1)) = 0.0
    }
    SubShader {
        Tags { "RenderType"="Opaque" }
        LOD 200
        Stencil {
            Ref[_StencilMask]
            Comp equal
            Pass keep
            Fail keep
        }
        CGPROGRAM
        #pragma surface surf Standard fullforwardshadows
        #pragma target 3.0
        sampler2D _MainTex;
        struct Input {
            float2 uv_MainTex;
        };
        half _Glossiness;
        half _Metallic;
        fixed4 _Color;
        void surf (Input IN, inout SurfaceOutputStandard o) {
            fixed4 c = tex2D(_MainTex, IN.uv_MainTex) * _Color;
            o.Albedo = c.rgb;
            o.Metallic = _Metallic;
            o.Smoothness = _Glossiness;
            o.Alpha = c.a;
        }
        ENDCG
    }
    FallBack "Diffuse"
}

Shader "Custom/Intersect"
{
    Properties
    {
        [HDR] _Color("Color", Color) = (1,1,1,1)
    }
    SubShader
    {
        //Rendertype: Opaque, Transparent or Overlay all give same result...
        //Queue is important here! Must be over 2500 to get front to back rendering, Overlay = 4000 (Transparent also works..)
        Tags { "RenderType"="Opaque" "Queue" = "Overlay" }
        LOD 100
        ZWrite Off
        //first pass sets the mask for the "front geometry"
        Pass
        {
            Cull Back
            ZTest Greater
            ColorMask 0
            Stencil {
                Ref 1
                Comp Always
                Pass Replace
            }
        }
        //second pass turn off culling, could be Cull Off or Cull Front (both acheive the same thing)
        //use the mask from the first pass and draw the color
        Pass
        {
            Cull Off
            ZTest Greater
            Stencil {
                Ref 1
                Comp NotEqual
            }
            CGPROGRAM
            #pragma vertex vert
            #pragma fragment frag
            #include "UnityCG.cginc"
            struct appdata
            {
                float4 vertex : POSITION;
            };
            struct v2f
            {
                float4 vertex : SV_POSITION;
            };
            float4 _Color;
            v2f vert (appdata v)
            {
                v2f o;
                o.vertex = UnityObjectToClipPos(v.vertex);
                return o;
            }
            fixed4 frag (v2f i) : SV_Target
            {
                return _Color; // just return it
            }
            ENDCG
        }
// reset the stencil buffer so other objects dont mask out this one
        Pass
        {
            //Cull Back
            ZTest Greater
            ColorMask 0
            Stencil {
                Ref 0
                Comp Always
                Pass Zero
            }
        }
    }
}

If anyone could please offer me any help, I’d be extremely grateful.

Kind Regards,
Davy.

2

You could just delete the second two passes of the intersect shader. Or change your stencil mask to use ZTest Greater, that’s really the only “magic” bit of the intersection shader. The default for all shaders is ZTest LEqual, or “less than or equal”, which means if the object is closer to the camera than the closest previously drawn opaque object, it’ll render. So ZTest Greater means it only draws if something else was drawn closer. Hence only showing on “intersections”.

Note, it’s not actually showing intersections. The two meshes might not actually be intersecting or touching at all. You can only be sure that there was an object that already rendered to the depth buffer closer to the camera than the current shader.

1 Like
3

Hi bgolus,

I really appreciate the reply. I tried both of your proposed solutions and neither of them worked.

Just to clarify what I’m trying to do, the StencilMask allows the user to see Stencilled objects when the user looks through the StencilMask. I want to be able to apply the StencilMask shader to the intersection of a cube and a larger sphere so that I can view Stencilled objects through a certain part of the sphere.

4

They were effectively the same solution. And it would work, but there’s an extra caveat that the objects you’re trying to render are likely still being depth occluded. If they can’t be seen where you want to show it before trying to use stencil masks to hide them, they can’t be seen afterwards either. Stencils won’t fix that. See my comments about ZTest Greater above, and notice all of the passes of that last shader are using it. If you want your objects to show inside another object while still rendering the “outside” object, you’ll need to do a fair bit more work to make that happen.

Adding ZTest Greater is an option, but one I suspect won’t look the way you want it to, as the object can appear sort of inside out if it has overlapping parts. The other option is ZTest Always, but that has a similar problem. An opaque 3D object that isn’t just a flat color needs to be rendered using the default ZTest LEqual in order to appear correctly sorted with itself, let alone the rest of the scene. And if you use the default it’s getting sorted against the “outside” object and thus hidden.

The trick here is likely going to be to use a depth punch through pass. You’d need extra an “invisible” pass that only renders to the z buffer before your surface shader, something like this:

Pass {
  ZTest Always
  Cull Off
  ColorMask 0
}

Add that between the Stencil {} and the CGPROGRAM in your Stencilled shader.

5

Here are some images of my example after applying your solution. So I have a cube with the stencil mask shader applied to it (coloured yellow for visibility) and then a cube with inverted normals (for hole effect) with the stencilled shader applied to it placed inside of the cube mask. Both cubes are intersecting a transparent sphere. What I’m hoping to achieve is to give the visual effect that there is a hole in the sphere but I do not require the sphere to be visible. All I need to be seen when the program is run, is the hole with a spherical shaped opening (if you get what I mean).

6977549--823025--Screenshot 2021-03-26 at 14.09.54.jpg 6977549--823028--Screenshot 2021-03-26 at 14.10.31.jpg 6977549--823031--Screenshot 2021-03-26 at 14.10.42.jpg

6

The yellow box rendering “on top” makes me think you’ve not properly set the rendering order of the materials, or the shader used on the yellow box is writing to the depth buffer which it should not be doing.

You should also try the original intersection shader with your sphere as you’ll see it doesn’t work either. This is because the “intersection” effect is reliant on having opaque geometry that has rendered to the depth buffer before the intersection shader renders. Transparent geometry both renders after most things, and more importantly does not render to the depth buffer at all. If you’re looking to intersect with a sphere, especially one that doesn’t even need to be visible, then using stencils for this is the wrong approach to begin with.

More likely you want to use a custom shader that you pass the world space position and radius of a sphere to. And then clip the object if it’s outside the range of the sphere.

Some links to examples:

7

Have you found a suitable solution to this?