Outline shader does not render to depth

Unity Engine
,
1

I have an outline shader that renders a separate pass pushed back in z-space (and offset in screenpos) that get PPv2 (post processing stack v2) effects drawing on top of it (DOF and SSAO).

After some investigating in the forums I’m pretty sure the issue is that the outline shader does not render to depth and the solution is to render a shadow casting pass (see Custom Shader doesn't fill depth and normals for post processing? for example).

I’ve tried implementing a shadow pass with the same displacement code as in the outline pass but as I understand it, the shadow pass needs mesh displacement set in v.vertex (see here vert/frag shader with shadows on offset vertices ?gq=vert%2Ffrag%20shader%20with%20shadows%20on%20offset%20vertice).

This code renders to depth but not displaced. If set v.vertex before SHADOW_CASTER_FRAGMENT to some random value it does indeed displace but I’m not sure how to translate the outline displacement to v.vertex.

Thanks!

(Original shader (that I have made changes to) by the amazing x.com)

Shader "Outline"  {
    Properties {
        _Color ("Main Color", Color) = (.5,.5,.5,1)
        _MainTex ("Base (RGB)", 2D) = "white" { }
    }
   
    CGINCLUDE
    #include "UnityCG.cginc"
   
   
    struct appdata {
        float4 vertex : POSITION;
        float3 normal : NORMAL;
        float4 texcoord : TEXCOORD0; // texture coordinates
    };

    struct v2f {
        float4 pos : SV_POSITION;
        UNITY_FOG_COORDS(0)
        fixed4 color : COLOR;
        float4 screenPos : TEXCOORD0;//
    };
   
    uniform float _Outline, _XOffset, _YOffset, _OutlineZ, _Brightness;
    sampler2D _MainTex;
   
    v2f vert(appdata v) {
        v2f o;
        o.pos = UnityObjectToClipPos(v.vertex);
        o.screenPos = ComputeScreenPos(o.pos);//

        _Brightness =  2.4;
        _Outline =  0.00129;
        _OutlineZ = 0.005;
        _YOffset = -0.445;
        _XOffset = 0;

        float3 norm = normalize(mul ((float3x3)UNITY_MATRIX_IT_MV, v.vertex));
        float2 offset = TransformViewToProjection(norm.xy) + (float2(_XOffset, _YOffset) * 20);
        float4 tex = tex2Dlod(_MainTex, float4(v.texcoord.xy, 0, 0));
       
        #ifdef UNITY_Z_0_FAR_FROM_CLIPSPACE //to handle recent standard asset package on older version of unity (before 5.5)
            o.pos.xy += offset * UNITY_Z_0_FAR_FROM_CLIPSPACE(o.pos.z) * _Outline;
        #else
            o.pos.xy += offset * o.pos.z * _Outline;
        #endif
        o.pos.z -= (o.screenPos.z * _OutlineZ);

            o.color = tex * _Brightness * 1.1 * (unity_AmbientGround + unity_AmbientSky + unity_AmbientEquator);

        UNITY_TRANSFER_FOG(o, o.pos);
        return o;
    }
    ENDCG
    SubShader {
        Tags {
            "RenderType" = "Opaque"
            "Queue" = "Geometry"
        }
        UsePass "Base/FORWARD"
        Pass {
            Tags { "LightMode"="ShadowCaster" }
           
            CGPROGRAM
            #pragma vertex vert
            #pragma fragment frag
            #pragma multi_compile_shadowcaster
            #include "UnityCG.cginc"
           
            struct v2f2 {
                V2F_SHADOW_CASTER;
                float4 screenPos : TEXCOORD0;//
            };
           
            v2f2 vert(appdata_base v)
            {
                v2f2 o;
                o.pos = UnityObjectToClipPos(v.vertex);
                o.screenPos = ComputeScreenPos(o.pos);//

                _Outline =  0.00129;
                _OutlineZ = 0.005;
                _YOffset = -0.445;
                _XOffset = 0;

                float3 norm = normalize(mul ((float3x3)UNITY_MATRIX_IT_MV, v.vertex));
                float2 offset = TransformViewToProjection(norm.xy) + (float2(_XOffset, _YOffset) * 20);
                float4 tex = tex2Dlod(_MainTex, float4(v.texcoord.xy, 0, 0));
               
                #ifdef UNITY_Z_0_FAR_FROM_CLIPSPACE // to handle recent standard asset package on older version of unity (before 5.5)
                    o.pos.xy += offset * UNITY_Z_0_FAR_FROM_CLIPSPACE(o.pos.z) * _Outline;
                #else
                    o.pos.xy += offset * o.pos.z * _Outline;
                #endif

                TRANSFER_SHADOW_CASTER_NORMALOFFSET(o)
                return o;
            }
           
            float4 frag(v2f i) : SV_Target
            {
                SHADOW_CASTER_FRAGMENT(i)
            }
            ENDCG
        }

        Pass {
            Tags {
                "LightMode" = "Always" 
            }
            Name "OUTLINE"
            ZWrite On
            ColorMask RGB
            Blend One Zero

            CGPROGRAM
            #pragma vertex vert
            #pragma fragment frag
            #pragma multi_compile_fog
            fixed4 frag(v2f i) : SV_Target
            {
                UNITY_APPLY_FOG(i.fogCoord, i.color);
                return i.color;
            }
            ENDCG
        }

    }

    Fallback "Diffuse"
}
2

Yep, you’re correct that the problem is the lack of a shadow caster pass. However the solution for this isn’t going to be all that straightforward. The problem is outline shaders like this are multi-pass shaders; one pass for the outline and one pass for the direct lit object. Unity’s shadow system is setup to only render the first shadow caster pass it finds in a shader, it will not render any additional shadow caster passes as usually these are superfluous. But the only way to get the outline to render to the depth is to have it have it’s own shadow caster pass. Additionally you usually don’t want the outline to cast a shadow itself, and there’s no (easy) way to limit the shadow caster pass to only render during the camera depth pass.

The easiest solution to this is use separate mesh renderers for the outline and the object rather than having a single mesh & shader that does both. Have an outline only shader & material you assign to one mesh, and use a regular shader & material on the other. On the outline only mesh disable shadow casting on the mesh renderer component itself.

However be warned this won’t entirely fix the SSAO problem depending on which SSAO you’re using. Some SSAO methods only use the camera depth texture, at which point this should all work (though be warned the outline may still receiving AO since it’s being drawn during the opaque passes). Some SSAO methods also use the camera depth normals texture, which gets rendered in a completely different way. That uses the built in Internal-DepthNormalsTexture.shader which has several different passes in it with unique “RenderType” tags.

You’d want to give your outline only shader it’s own unique “RenderType”, and then add a pass using that tag and doing the outline offset to a copy of that Internal-DepthNormalsTexture.shader. You can then tell Unity to use that updated shader by setting an override in the project’s Graphics settings.