Alpha Cutout + Lightmap + Shadows

I’ve spent the last couple hours digging into why one of our custom shaders isn’t allowing us to bake a lightmap with it casting shadows as well as receiving the lightmap, but I’ve come at an impasse and I don’t know what to do. I’ve hobbled together what I could figure out from other threads, but I can’t seem to figure it out past this point.

Current Shader:

Default Unlit/Transparent Cutout:

Desired (achieved by reverting to my shader after performing a lightmap with Transparent Cutout turned on):

Current Code

//Unlit
//Lightmap support
//Texture
//Alpha

Shader "FFG/Lightmap/Unlit-Cutout" {
    Properties {
        _MainTex ("Base", 2D) = "white" {}
        _Cutoff ("Cutoff Alpha", Range(0,1)) = 0.5
    }

    SubShader
    {
        Tags { "Queue"="AlphaTest" "IgnoreProjector" = "True" "RenderType"="TransparentCutout" }

        Cull Off
        Blend SrcAlpha OneMinusSrcAlpha

        Pass {
            Name "ForwardBase"
            Tags { "LightMode" = "ForwardBase" }
            CGPROGRAM
                #pragma vertex vert
                #pragma fragment frag
                #pragma multi_compile_fwdbase
                #pragma multi_compile LIGHTMAP_ON LIGHTMAP_OFF
                #include "UnityCG.cginc"

                struct v2f
                {
                    fixed4 pos : SV_POSITION;
                    fixed2 lmap : TEXCOORD1;
                    fixed2 pack0 : TEXCOORD0;
                };

                uniform sampler2D _MainTex;
                uniform fixed4 _MainTex_ST;
                uniform float _Cutoff;

                #ifdef LIGHTMAP_ON
                    sampler2D unity_Lightmap;
                    fixed4 unity_LightmapST;
                #endif

                v2f vert(appdata_full v)
                {
                    v2f o;
                    o.pos = mul(UNITY_MATRIX_MVP, v.vertex);
                    o.pack0.xy = TRANSFORM_TEX(v.texcoord, _MainTex);

                    #ifdef LIGHTMAP_ON
                        o.lmap.xy = v.texcoord1.xy * unity_LightmapST.xy + unity_LightmapST.zw;
                    #endif
                    return o;
                }

                fixed4 frag(v2f i) : COLOR
                {
                    fixed4 c = tex2D(_MainTex, i.pack0);
                    if(c.a <= _Cutoff) {
                        discard;
                    }

                    #ifdef LIGHTMAP_ON
                        c.rgb *= DecodeLightmap(tex2D(unity_Lightmap, i.lmap));
                    #endif
                    return c;
                }
            ENDCG
        }
     
        Pass {
                Name "ShadowCollector"
                Tags { "LightMode" = "ShadowCollector" }
             
                Fog {Mode Off}
                ZWrite On ZTest LEqual

                CGPROGRAM
                #pragma vertex vert
                #pragma fragment frag
                #pragma multi_compile_shadowcollector

                #define SHADOW_COLLECTOR_PASS
                #include "UnityCG.cginc"
                #include "AutoLight.cginc"

                struct v2f {
                    V2F_SHADOW_COLLECTOR;
                    float2  uv : TEXCOORD5;
                };

                uniform float4 _MainTex_ST;

                v2f vert (appdata_base v)
                {
                    v2f o;
                    TRANSFER_SHADOW_COLLECTOR(o)
                    o.uv = TRANSFORM_TEX(v.texcoord, _MainTex);
                    return o;
                }

                uniform sampler2D _MainTex;
                uniform fixed _Cutoff;

                fixed4 frag (v2f i) : SV_Target
                {
                    fixed4 texcol = tex2D( _MainTex, i.uv );
                    clip(texcol.a - _Cutoff );
                 
                    SHADOW_COLLECTOR_FRAGMENT(i)
                }
                ENDCG
        }

        // Pass to render object as a shadow caster
        Pass {
            Name "ShadowCaster"
            Tags { "LightMode" = "ShadowCaster" }
            Offset 1, 1
         
            Fog {Mode Off}
            ZWrite On ZTest LEqual Cull Off

            CGPROGRAM
            #pragma vertex vert
            #pragma fragment frag
            #pragma multi_compile_shadowcaster
            #include "UnityCG.cginc"
            #include "AutoLight.cginc"

            struct v2f {
                V2F_SHADOW_CASTER;
                float2  uv : TEXCOORD1;
            };

            uniform float4 _MainTex_ST;

            v2f vert( appdata_base v )
            {
                v2f o;
                TRANSFER_SHADOW_CASTER(o)
                o.uv = TRANSFORM_TEX(v.texcoord, _MainTex);
                return o;
            }

            uniform sampler2D _MainTex;
            uniform fixed _Cutoff;

            float4 frag( v2f i ) : SV_Target
            {
                fixed4 texcol = tex2D( _MainTex, i.uv );
                clip(texcol.a - _Cutoff );
             
                SHADOW_CASTER_FRAGMENT(i)
            }
            ENDCG
        }
    }
}

Any assistance would be much appreciated!

I should mention that I grabbed the shader source for Unlit/Transparent Cutout and put it into a shader with a different name and it stops casting shadows properly. So is Unity hard coded to have different rendering settings with the default shaders based on their name?

What built in shader did you say you tried that wouldn’t work when you renamed it? I tried renaming Transparent/cutout diffuse and it worked. If you were trying Unlit/Transparent Cutout well that one shouldn’t of worked since it doesn’t support shadows.

and here is something that does what you want i think.
http://wiki.unity3d.com/index.php/Transparent_Diffuse_Two_Pass_Lit_Shadows
but i also found this

I dont know which one answers your question better.

Thanks for the reply!

I tested out Unlit/Transparent Cutout and it does indeed cast shadows even though there isn’t a fallback that performs the shadow cast/catch lightmodes. Go ahead and test it yourself it works for some reason (which I assume to be hard coding of the name of the shader) it works. The one that I used was the source for “Transparent/Cutout/VertexLit” and put it into my own shader name and it stops working as well.

Transparent/cutout is a surface shader that does all the “magic” behind the scenes. I don’t want the overhead of a surface shader, and I don’t want the overhead of a fragment Color multiplication. So I’m trying to create a shader that is my own that can cast/catch shadows correctly. Doesn’t seem like it would be that tough, but it is turning into a major headache.

The answer to this age-old question is so frustratingly simple… yet was quite a pain to find…

Your shader simply needs ‘Transparent’ in the name (the name within the shader, not the filename). So in my case, changing:

Shader “CustomShaders/Foliage”

to:

Shader “CustomShaders/Foliage (Transparent)”

solves the problem…

You may also want to include: Tags { “Queue”=“AlphaTest” “RenderType”=“TransparentCutout”} - but these are ignored when it comes to light baking, it’s the name that matters…

@bluescrn ,

Could you please provide the code for this shader which you mentioned. I am looking for a shader which combines the below:

Unlit/TransparentCutout + Shadows + Lightmap

My objective is to bake shadows into a texture which has Unlit/TransparentCutout shader.

So far, I am able to achieve Unlit/TransparentCutout + Shadows, but when I bake the scene, shadows disappear from the lightmap texture.

Wow this is an old post. I actually asked a Unity dev at Unite 2014 that year and was told the same thing, that there was a magic word search for “Transparent” that was used in the lightmapper to bake shadows. :frowning:

Is this still actual? Can’t get it to work neither with Progressive or Enlighten lightmappers. Shader does have MetaPass. Tried with clip(0.5 - alpha) and returning float4(albedo.rgb,alpha). Emission is working (i.e. shader is ok), but shadows does not affected by alpha.

Well, okay. If texture has alpha channel shadows are working ok. Is MetaPass data just ignored? What the hell Unity?

Found a solution. If u wanna use an alpha channell for another purposes, then transparency u need to add for ligtmapper shader property
[NoScaleOffset] _TransparencyLM (“Transmission LightMap”, 2D) = “white” {}
Where will be some texture where black = opaque and white = transparent.

1 Like