Transparent shadows - Changing queue from AlphaTest to Transparent

[jrhee]

Hi,

I’m working on a game with 3D characters and 2D environments, with depth sorting handled using sorting layer/ID.

I’m trying to have my characters cast shadows from a directional light onto invisible geometry, then draw those shadows on top of my environment sprites.

As shown above, I’m able to get shadows to display on an invisible plane, but my sprites are being drawn over them. I suspect I need to change the queue from “AlphaTest” to “Transparent” in the shader below, but when I do, shadows aren’t displayed at all.

Would anyone be able to point me to how I can get the below working with queue set to transparent?

Shader "Transparent/TransparentReceiveShadow"
{
    Properties
    {
        _Color("Shadow Color", Color) = (1,1,1,1)
        _Cutoff("Alpha cutoff", Range(0,1)) = 0.5
    }


    SubShader
    {
        Tags
        {
            "Queue" = "AlphaTest"
            "IgnoreProjector" = "True"
            "RenderType" = "TransparentCutout"
        }
        LOD 200
        ZWrite off
        Blend zero SrcColor


        CGPROGRAM
#pragma surface surf ShadowOnly alphatest:_Cutoff fullforwardshadows 

        fixed4 _Color;
        float _ShadowInt;

        struct Input 
        {
            float2 uv_MainTex;
        };

        inline fixed4 LightingShadowOnly(SurfaceOutput s, fixed3 lightDir, fixed atten)
        {
            fixed4 c;
            c.rgb = lerp(s.Albedo, float3(1.0,1.0,1.0), atten);
            c.a = 1.0-atten;
            return c;
        }


        void surf(Input IN, inout SurfaceOutput o) 
        {
            o.Albedo = _Color.rgb;
            o.Alpha = 1.0;
        }
        ENDCG
    }
    Fallback "Transparent/Cutout/VertexLit"
}

[Johannski]

The transparent pass can neither receive nor cast shadows, so that is not the solution for your problem. What can solve your problem is changing the render queue:
In every material you find find at the very bottom when in the renderqueue this material will be drawn

Normally you would first draw opaque geometry, then cutout materials and then transparent materials.
In your case it would make sense to first draw your 2d environment (which probably has quite a lot of transparent materials), and then your 3D materials (character and invisible objects). If you manage that the invisible objects write into the depth buffer for their full geometry you will also have no need to do the sorting yourself. But actually I#m not sure if that is possible with a single pass, I guess you would need two passes in order to draw your shadow with a cutout shader.

[jrhee]

Thanks Johannski. It’s tricky since sometimes 2D environment sprites need to be drawn over my 3D characters. I’m thinking I may need to go full 3D to meet all my sorting requirements, but will give the two pass shader a try first.

[Johannski]

Well that should be possible if the invisible 3d geometry writes into the depth buffer. For every pixel the shader for the character will do a check, if the depthvalue is greater than the one already written to the depth buffer and will only render the pixel if the pixel is in front of the one already written there.

This sadly implies that you have to be quite precise with your invisble 3d models which might lead to the conclusion that it is less work to go full 3d.

But maybe somebody else has another approach which could lead you on a more efficient way?!

[Johannski]

Hey again @jrhee
I put a bit more thought into it, and I think the solution I was offering is quite good after all. I did a small test:

As you can see, the invisible geometry can receive shadows and also culls away the red block behind it. The sprite on the other hand is not culled because it is drawn earlier in the render queue (those are the numbers in the notes).

Here is the shader for the invisible block:

Shader "Custom/ShadowDrawer"
{
    Properties
    {
        _Color ("Shadow Color", Color) = (0, 0, 0, 0.6)
    }

    CGINCLUDE

    #include "UnityCG.cginc"
    #include "AutoLight.cginc"

    struct v2f_shadow {
        float4 pos : SV_POSITION;
        LIGHTING_COORDS(0, 1)
    };

    half4 _Color;

    v2f_shadow vert_shadow(appdata_full v)
    {
        v2f_shadow o;
        o.pos = UnityObjectToClipPos(v.vertex);
        TRANSFER_VERTEX_TO_FRAGMENT(o);
        return o;
    }

    half4 frag_shadow(v2f_shadow IN) : SV_Target
    {
        half atten = LIGHT_ATTENUATION(IN);
        return half4(_Color.rgb, lerp(_Color.a, 0, atten));
    }

    ENDCG

    SubShader
    {
        Tags { "Queue"="Geometry -1" }

        // Depth fill pass
        Pass
        {
            ColorMask 0

            CGPROGRAM

            #pragma vertex vert
            #pragma fragment frag

            struct v2f {
                float4 pos : SV_POSITION;
            };

            v2f vert(appdata_full v)
            {
                v2f o;
                o.pos = UnityObjectToClipPos (v.vertex);
                return o;
            }

            half4 frag(v2f IN) : SV_Target
            {
                return (half4)0;
            }

            ENDCG
        }

        // Forward base pass
        Pass
        {
            Tags { "LightMode" = "ForwardBase" }
            Blend SrcAlpha OneMinusSrcAlpha
            CGPROGRAM
            #pragma vertex vert_shadow
            #pragma fragment frag_shadow
            #pragma multi_compile_fwdbase
            ENDCG
        }

        // Forward add pass
        Pass
        {
            Tags { "LightMode" = "ForwardAdd" }
            Blend SrcAlpha OneMinusSrcAlpha
            CGPROGRAM
            #pragma vertex vert_shadow
            #pragma fragment frag_shadow
            #pragma multi_compile_fwdbase
            ENDCG
        }
    }
    FallBack "Mobile/VertexLit"
}

It is from a project of keijiro and has, as I would have done as well, multiple passes.

You can change the render queue for every sprite that has a invisible object as I showed you, or you use this custom shader to make sure you don’t miss any sprites:

Shader "Custom/Sprites"
{
    Properties
    {
        [PerRendererData] _MainTex ("Sprite Texture", 2D) = "white" {}
        _Color ("Tint", Color) = (1,1,1,1)
        [MaterialToggle] PixelSnap ("Pixel snap", Float) = 0
        [HideInInspector] _RendererColor ("RendererColor", Color) = (1,1,1,1)
        [HideInInspector] _Flip ("Flip", Vector) = (1,1,1,1)
        [PerRendererData] _AlphaTex ("External Alpha", 2D) = "white" {}
        [PerRendererData] _EnableExternalAlpha ("Enable External Alpha", Float) = 0
    }

    SubShader
    {
        Tags
        {
            "Queue"="Geometry -50"
            "IgnoreProjector"="True"
            "RenderType"="Transparent"
            "PreviewType"="Plane"
            "CanUseSpriteAtlas"="True"
        }

        Cull Off
        Lighting Off
        ZWrite Off
        Blend One OneMinusSrcAlpha

        Pass
        {
        CGPROGRAM
            #pragma vertex SpriteVert
            #pragma fragment SpriteFrag
            #pragma target 2.0
            #pragma multi_compile_instancing
            #pragma multi_compile _ PIXELSNAP_ON
            #pragma multi_compile _ ETC1_EXTERNAL_ALPHA
            #include "UnitySprites.cginc"
        ENDCG
        }
    }
}

For all other sprites you can use the normal shader and queue.

I hope this gets you going. This way you won’t need any special sorting, the invisible objects will do all the work for you

[jrhee]

Awesome, thanks Johannski, that works great! I’m still thinking through sorting characters moving around larger assets, but this plenty to get me started. Cheers!