Shadow blending

Unity Engine
1

Hey there!

I’m trying to have my (dynamic) objects cast real time shadow on baked lighting in my scene.

I’ve found out that shader Mobile/Diffuse, Directional Light with Mixed mode and lightmapper with Subtractive mode do exactly what I need it.

However… It would be great if I could achieve that in a custom vertex/fragment shader.

So far I’ve done this and it almost works well:

Shader "Main/Enviroment/Enviroment LightMap" {
  Properties {
    _MainTex ("Tex", 2D) = "white" {}
  }
    SubShader {
        Pass {


            Tags { "LightMode" = "ForwardBase" }

            CGPROGRAM

            #pragma vertex vert
            #pragma fragment frag
            #include "UnityCG.cginc"


            #pragma multi_compile_fwdbase
            #pragma noforwardadd

            #include "AutoLight.cginc"

            uniform sampler2D _MainTex;

            struct vertexInput
            {
              half4 vertex : POSITION;
              half4 coords : TEXCOORD0;
              half4 lightmap : TEXCOORD1;
            };

            struct vertexOutput
            {
                half4 pos : SV_POSITION;
                half4 uv0 : TEXCOORD0;
                half4 lightmap : TEXCOORD1;
                LIGHTING_COORDS(2,3)
            };


            vertexOutput vert(vertexInput v) {
                vertexOutput o;


                o.pos = UnityObjectToClipPos (v.vertex);

                o.uv0 = v.coords;


                #ifdef LIGHTMAP_ON
                  o.lightmap.xy = v.lightmap.xy * unity_LightmapST + unity_LightmapST.zw;
                #endif

                TRANSFER_VERTEX_TO_FRAGMENT(o);

                return o;
            }

            fixed4 frag(vertexOutput i) : COLOR {

                half4 col = tex2D(_MainTex, i.uv0);

                fixed atten = LIGHT_ATTENUATION(i);

                #ifdef LIGHTMAP_ON
                        col.rgb *= DecodeLightmap(UNITY_SAMPLE_TEX2D(unity_Lightmap, i.lightmap));
                #endif


                return col * atten;

            }

            ENDCG
        }
        Pass
        {
          Name "ShadowCaster"
          Tags { "LightMode" = "ShadowCaster" }

          ZWrite On ZTest LEqual Cull Off

          CGPROGRAM
          #pragma vertex vert
          #pragma fragment frag
          #pragma target 2.0
          #pragma multi_compile_shadowcaster
          #include "UnityCG.cginc"

          struct v2f {
            V2F_SHADOW_CASTER;
          };

          v2f vert( appdata_base v )
          {
            v2f o;
            UNITY_SETUP_INSTANCE_ID(v);
            TRANSFER_SHADOW_CASTER_NORMALOFFSET(o)
            return o;
          }

          float4 frag( v2f i ) : SV_Target
          {
            SHADOW_CASTER_FRAGMENT(i)
          }
          ENDCG
        }
    }
}

In my shader everything seems to be working fine except for shadow blending - what am I missing? What am I doing wrong? What other magical MACROS do I need to use?

In default mobile/diffuse shader the “Realtime Shadow Color” in Lighting/Mixed Lighting controls how much dynamic shadows blends with lightmaps. How can I achieve that?

EDIT:

I’ve added some pictures


1 Like
2

Sorry for bumping this topic but this seems achievable… Any help would be welcome.

3

You’re multiplying the shadows on top of the lightmaps so the result you’re getting makes sense.

Try something like this: Change:

col.rgb *= DecodeLightmap(UNITY_SAMPLE_TEX2D(unity_Lightmap, i.lightmap));

to

col.rgb *= min(DecodeLightmap(UNITY_SAMPLE_TEX2D(unity_Lightmap, i.lightmap)),atten);

and remove the * atten at the return.

4

Thank you for your reply!

min() returns the minimum values of A and B - A is my shadow which is completly black (0) and B is the lightmap (other values in range of 0-1). That way this function always returns 0 where my shadow is - this has the exactly same effect as previously.

In Mobile/Diffuse the shadow color and transparency depends on the “Realtime Shadow Color” in Lighting-> Mixed Lighting (Subtractive). If I could somehow ‘connect’ this color picker value to my shader I could probably make the min() function work just by setting shadow other values than black - that would result in blending between those values.

5

You’re right. The way I did it a while back was I had a script that set my own shadow color with Shader.SetGlobalColor and then I’dd add that to atten. But there probably is a way to get the Realtime Shadow Color as well. I just don’t know it from the top of my head.

6

I’ve looked inside compiled shader variant for Mobile/Diffuse but there are so many magical Unity macros that ‘do stuff’ that I am pulling my hair out to figure this thing out :slight_smile: Did you ever tried to make that work?

7

Ok, so I am a lot closer now.

I’ve figured out that there is a “unity_ShadowColor” variable and it is responsible for “Realtime Shadow Color” color picker in Lighting → Mixed Lighting (Subtractive).

Shader "Lit/Diffuse With Shadows"
{
    Properties
    {
        [NoScaleOffset] _MainTex ("Texture", 2D) = "white" {}
          _shadowColor ("shadow", color) = (1,1,1,1)
    }
    SubShader
    {
        Pass
        {
            Tags {"LightMode"="ForwardBase"}
            CGPROGRAM
            #pragma vertex vert
            #pragma fragment frag
            #include "UnityCG.cginc"
            #include "Lighting.cginc"

            // compile shader into multiple variants, with and without shadows
            // (we don't care about any lightmaps yet, so skip these variants)
            #pragma multi_compile_fwdbase
            // shadow helper functions and macros
            #include "AutoLight.cginc"

            struct vertexInput {
                half4 vertex : POSITION;
                half3 normal : NORMAL;
                half4 texcoord : TEXCOORD0;
                half4 lightmap : TEXCOORD2;
                UNITY_VERTEX_INPUT_INSTANCE_ID
            };

            float3 _shadowColor;

            struct vertexOutput
            {
                half4 pos : SV_POSITION;
                half2 uv0 : TEXCOORD0;
                UNITY_SHADOW_COORDS(1) // put shadows data into TEXCOORD1
                half3 diff : COLOR0;
                half3 ambient : COLOR1;
                half4 lightmap : TEXCOORD2;
                half3 worldPos : TEXCOORD3;
            };
            vertexOutput vert (vertexInput v)
            {
                vertexOutput o;



                half3 worldNormal = UnityObjectToWorldNormal(v.normal);

                half3 worldPos = mul(unity_ObjectToWorld, v.vertex).rgb;
                /*half nl = max(0, dot(worldNormal, _WorldSpaceLightPos0.xyz));*/
                /*o.diff = nl * _LightColor0.rgb;*/
                o.ambient = ShadeSH9(half4(worldNormal,1));
                // compute shadows data

                o.pos = UnityObjectToClipPos (v.vertex);

                o.uv0 = v.texcoord;

                o.worldPos = worldPos;

                #ifdef LIGHTMAP_ON
                  o.lightmap.xy = v.lightmap.xy * unity_LightmapST + unity_LightmapST.zw;
                #endif

                UNITY_TRANSFER_SHADOW(o, o.lightmap.xy)

                return o;
            }

            sampler2D _MainTex;

            half4 frag (vertexOutput i) : SV_Target
            {
                UNITY_SETUP_INSTANCE_ID(i)
                half4 col = tex2D(_MainTex, i.uv0);

                half3 lightMap = DecodeLightmap(UNITY_SAMPLE_TEX2D(unity_Lightmap, i.lightmap));

                half3 shadow = (1 - UNITY_SHADOW_ATTENUATION(i, i.worldPos));

                half3 substractedLight = lightMap - shadow;

                col.rgb *= max(substractedLight, unity_ShadowColor);

                return half4(col.rgb,1);
            }
            ENDCG
        }

        Pass
        {
          Name "ShadowCaster"
          Tags { "LightMode" = "ShadowCaster" }

          ZWrite On ZTest LEqual Cull Off

          CGPROGRAM
          #pragma vertex vert
          #pragma fragment frag
          #pragma target 2.0
          #pragma multi_compile_shadowcaster
          #include "UnityCG.cginc"

          struct vertexInput {
              half4 vertex : POSITION;
              half3 normal : NORMAL;
              half4 texcoord : TEXCOORD0;
              UNITY_VERTEX_INPUT_INSTANCE_ID
          };

          struct vertexOutput{
            V2F_SHADOW_CASTER;
          };

          vertexOutput vert( vertexInput v )
          {
            vertexOutput o;
            UNITY_SETUP_INSTANCE_ID(v);
            TRANSFER_SHADOW_CASTER_NORMALOFFSET(o)
            return o;
          }

          half4 frag(vertexOutput i) : SV_Target
          {
            SHADOW_CASTER_FRAGMENT(i)
          }
          ENDCG
        }
      }
    }

This works almost well. I can use the “Realtime Shadow Color” to change the ‘blackness’ of the shadows in order to blend them. However using this current formula I am adjusting lightmaps too - this isn’t ideal cause in order for shadows to blend I need to push this color rather far resulting in lightmap losing its darkness and changing overall look. Perfect scenario would be - “Realtime Shadow Color” adjusts only the realtime shadow colors and leave lightmaps out of it.

1 Like
8

hey gaxx, thank you so much for this shader. this was exactly what i was looking for.
i had issues to get an unlit lightmp shader to support shadows correctly but thanks to you they work exactly like expected! thumbs up!

Edit:
i checked it a little more deeply and it seems to have some issues with the blending between dynamic received shadows and the lightmap baked shadows

EDIT 2:
i fixed the issues that i had with your shader with these changes:

float atten = 1-LIGHT_ATTENUATION(i);
half3 substractedLight = lightMap - atten * i.diff; // - shadow;
col.rgb *= min(lightMap, max(substractedLight, unity_ShadowColor));

nonetheless, big thanks!

9

Hey!

I’ll check your solution as soon as I’ll dig up that thing - it’s been a while since then.

Thanks for reply :slight_smile:

EDIT:

Ok, I did check how this changes would affect and yes, this blends same as Mobile/Diffuse! Great that you solved it!

There are problems with this approach in general. In order to blend correctly between lightmap and dynamic shadows, you need to know the correct color for unity_ShadowColor. Too much black and you have overlapping shadows, too much white and your shadow is not visable. Another problem is when you have multiple levels with different lightning scenarios. This value must be controlled for proper blend. This is unpractical approach unfortunately and propably leave to much free space for errors.