Can you change the impacted depth for stencil shader?

Unity Engine
1

I’m having a problem with a stencil mask sander drawing both objects in front of and behind the object I apply it to. My scene is currently set up like so:

945×562 25.3 KB

There is simply a (blue) cube with two cylinders on top of it with a window plane in between them.
All blue/purple material are using this shader:

Shader "Custom/Past"
{
    Properties
    {
        _Color ("Main Color", Color) = (1,1,1,1)
        _MainTex ("Base {RGB}", 2D) = "white" {}
        _Glossiness ("Smoothness", Range(0,1)) = 0.5
        _Metallic ("Metallic", Range(0,1)) = 0.0
    }
    SubShader
    {
        Tags {"RenderType" = "Opaque" "Queue"="Geometry-2"}
        LOD 200

        Stencil
        {
            Ref 1
            Comp Equal
            Pass Keep
        }

        CGPROGRAM
        // Physically based Standard lighting model, and enable shadows on all light types
        #pragma surface surf Standard fullforwardshadows

        // Use shader model 3.0 target, to get nicer looking lighting
        #pragma target 3.0

        sampler2D _MainTex;

        struct Input
        {
            float2 uv_MainTex;
        };

        half _Glossiness;
        half _Metallic;
        fixed4 _Color;

        // Add instancing support for this shader. You need to check 'Enable Instancing' on materials that use the shader.
        // See https://docs.unity3d.com/Manual/GPUInstancing.html for more information about instancing.
        // #pragma instancing_options assumeuniformscaling
        UNITY_INSTANCING_BUFFER_START(Props)
            // put more per-instance properties here
        UNITY_INSTANCING_BUFFER_END(Props)

        void surf (Input IN, inout SurfaceOutputStandard o)
        {
            // Albedo comes from a texture tinted by color
            fixed4 c = tex2D (_MainTex, IN.uv_MainTex) * _Color;
            o.Albedo = c.rgb;
            // Metallic and smoothness come from slider variables
            o.Metallic = _Metallic;
            o.Smoothness = _Glossiness;
            o.Alpha = c.a;
        }
        ENDCG
    }
    //Fallback "VertexLit"
}

And the window plane in the middle is using this shader:

Shader "Custom/PastStencilMask"
{
    SubShader
    {
        Tags { "RenderType"="Opaque" "Queue"="Geometry-3" }
        ColorMask 0
        ZWrite off

        Stencil {
            Ref 1
            Comp Always
            Pass Replace
        }
        LOD 200

        Pass
        {
            Cull Back
            Ztest Less

            CGPROGRAM
            #pragma vertex vert
            #pragma fragment frag

            struct appdata
            {
                float4 vertex : POSITION;
            };
            struct v2f
            {
                float4 pos : SV_POSITION;
            };
            v2f vert(appdata v)
            {
                v2f o;
                o.pos = UnityObjectToClipPos(v.vertex);
                return o;
            }
            half4 frag(v2f i) : COLOR
            {
                return half4(1,1,1,1);
            }
            ENDCG
        }
    }
    Fallback "VertexLit"
}

I want the blue/purple objects to only be visible through the window plane.
I think the half cylinder in the foreground is being rendered because the stencil value at the window plane is always 1 regardless of depth correct? If that’s the case how would I make the stencil only impact objects behind it so the cylinder in the foreground would not render? I’ve tried messing around with a depth mask but don’t really know enough about them to make them do what I want.

2

Correct. Stencils are a purely 2D effect. The cylinder in front of the stencil geometry has no way to know what depth the stencil was rendered at unless the stencil also writes to depth. But if the stencil writes to the depth that doesn’t really help since that’ll block the objects behind it, and still won’t really help with the object in front as it’d have to use ZTest Greater which will cause other issues for anything more complex than a single cylinder.

The real answer is this isn’t something stencils, or even the depth buffer, can really handle alone. If you’re very careful about manually sorting objects you can place the objects “in front” of the stencil to render before the stencil does, but at that point you’ve already figured out it shouldn’t be visible to the stencil, so it’d be better to just disable the object in that case.

The more generalized solution is to use a second camera with its frustum setup to match where the “portal” is. That or to actually pass either an SDF or mathematical form of the stencil geometry to the shaders you want to clip and do the coverage & depth comparison against it in those objects’ shader. In the case of the above example of a basic quad geometry being used, you could do both the stencil and the in-shader clipping. All you’d need to do is pass the data for a plane (a center point and a normal) to the shader used by the cylinders and floor plane and test to see what side of the plane your objects are on. If they’re in front of the plane, discard it.

1 Like
3

Sorry, I’m still relatively new to shaders so I gotta inquire more about the process of clipping.
Would I first define the planes center position and normal as static inside the shader?
And then convert the clipping plane position to clip space to test which side of the plane any given fragment resides (using the dot product of the plane normal and a vector coming from the planes center to the fragment)?
I’m really not sure, I might be totally wrong. This whole subject may be a bit over my head.

4

I’d put a script on the quad you’re using and have it set global shader properties:

For the quad’s world space position:

And forward vector:

Then in your shader you’d have to take the world position, subtract the quad’s position, then do a dot product between that vector and the quad’s forward vector. If the value is positive, it’s in front of the quad. If it’s negative it’s behind. Then you can use the clip() function to hide pixels that are in front of the plane. Any negative value passed to clip() will cause that pixel of the object to be skipped.

// add to shader inside CGPROGRAM, but outside a function
float3 _MyPlanePos;
float3 _MyPlaneNormal;

// need to add this to Input struct
float3 worldPos;

// inside surf function
float3 planeRelativePos = IN.worldPos - _MyPlanePos.xyz;
float planeDist = dot(planeRelativePos, _MyPlaneNormal.xyz);
clip(-planeDist) // skips pixels in front of plane

Assuming you only have one “window” at a time this should work well.

3 Likes
5

Ah, so I can gather the quad properties from a traditional script and modify the shader variables there. That makes much more sense. Seeing the syntax and layout of the shader code was super helpful. Thank you very much bgolus, you’ve been super awesome in helping me understand shaders better, not just here but the many other places you’ve posted.

6

[QUOTE="

clip(-planeDist) // skips pixels in front of plane[/code]

[/QUOTE]

This is intended a good solution but it’s quite expensive on mobile android devices like Quest. Do you know of any alternative to discard, clip or alphablending (heard this was costly too)?