Struggling To Get Shadows For Cutout Shader To Meet My Criteria

Unity Engine
1

Hi,

I’ve made a cutout shader that combines vertex colour along with a supplied Range to cut out the mesh. I’m having a problem getting the shadows to behave how I want it to though. I’m quite new with shaders so would appreciate some help.

Here is what I currently have. As you change the slider on the material the building mesh cuts away in sections allowing you to see inside.

The shader currently falls short because it doesn’t meet these two criteria I have…

  • The building should cast a shadow on all other objects. The shadow cast should not take into account the cutout area, it should be the shadow of the full building (As you can see above the shadow cast on the ground and boxes gets cut at the moment),
  • The building should receive shadows from all other objects. The shadow should not appear on the cutout area of the building and the shadow should not pass through the cutout area to the inside. (As you can see the sphere casting a shadow onto the building passes through the cutout area at the moment).

I’m not sure in my shader how to control the shadows/lighting in this way. If I were doing baked lighting I would achieve the above requirements by default, but I would like to do it with realtime lighting if possible.

Would appreciate some pointers in the right direction if someone could take a look?

Bellow is my current shader and I have attached the test project shown above.

Thanks,
Jon

Shader "Custom/VertexCutout"
{
    Properties
    {
        _Color ("Color", Color) = (1,1,1,0.5)
        _MainTex ("Texture", 2D) = "white" {}
        _DissolvePercentage("Dissolve", Range(0,1)) = 0.0
    }
  
    SubShader
    {
        CGPROGRAM
      
        #pragma surface surf StandardSpecular addshadow vertex:vert
        #pragma target 3.0
      
        struct Input
        {
            float4 vertColor;
            float2 uv_MainTex;
        };
      
        fixed4 _Color;
        sampler2D _MainTex;
        half _DissolvePercentage;

        void vert(inout appdata_full v, out Input o)
        {
            UNITY_INITIALIZE_OUTPUT(Input, o);
            o.vertColor = v.color;
        }

        void surf(Input IN, inout SurfaceOutputStandardSpecular o)
        {
            // Draw the texture
            o.Albedo = tex2D (_MainTex, IN.uv_MainTex) * _Color;
          
            // Cut out the vertexs
            fixed3 c = IN.vertColor.rgb;
            clip(c - _DissolvePercentage);
        }

        ENDCG
    }
    Fallback "Legacy Shaders/Transparent/Cutout/VertexLit"
}

4714715–445760–UnityShaderTestProject.zip (123 KB)

2

Controlling shadow casting in a shader is done with a separate ShadowCaster pass. In the case of a surface shader (like yours), Unity has three ways of doing a ShadowCaster pass. The default is to use the ShadowCaster pass of your fallback shader (in this case, the VertexLit shader). If you wish to modify shadow casting based on your surface shader, you can add the tag ‘addshadow’ (like you have done), which will override the fallback. Finally, you can write your own pass and use it after your main shader content. In your case, to achieve the results you want, all you need to do is to remove the ‘addshadow’ tag from the surface shader declaration (line 14). This will ignore the alpha cutout when generating the shadow-caster pass, meaning the lights in the scene will see the building as though it is there in whole.

3

Hi, thanks for your response, I will read up more about ShadowCaster with surface shader.

You suggested just removing the ‘addshadow’ tag to achieve the effect I’m after. When I do that though it all gets a bit messy. The shadows cast by the building start rendering on the cutout area and the shadows cast onto the ground don’t show through the cutout area of the building. Shadows stop showing inside the building too. You can see this bellow.

I have found that if I also set the ‘Queue’ tag to ‘Transparent’ as well as removing ‘addshadow’ it starts to clean it all up a bit but it no longer receives shadows from other objects or itself. You can see this bellow.

I feel like every tag I add or remove gives me some of what i want. I’m not sure if there is a combination that will give me everything I’m after or if I’m going to have to look more down the road of writing custom passes.

Any more help on this is greatly appreciated.

4

I’ll be honest, I forgot that depth information is written in with the shadow-caster pass; hence the issues you see here. You could try adding a custom pass after (something along the lines of this);

    ...
    ENDCG

    Pass
    {
        Name "ShadowCaster"
        Tags { "LightMode" = "ShadowCaster" }

        CGPROGRAM
        #pragma vertex vert
        #pragma fragment fragD
        #pragma target 2.0
        #pragma multi_compile_shadowcaster
        #pragma multi_compile_instancing // allow instanced shadow pass for most of the shaders

        float4 fragD (v2f i) : SV_Target
        {
            return 0;
        }
        ENDCG
    }

}
Fallback "Legacy Shaders/Transparent/Cutout/VertexLit"
...

Which may or may not work. The reason making the object transparent works is because it doesn’t write to the depth buffer; however this also means Unity can’t use screen-space shadows, so it won’t receive any shadows.

A workaround would be to have your house not cast shadows, but instead have a copy of the house that only casts shadows.

5

The problem here is the shadowcaster pass is used for both shadow casting and shadow receiving (technically the camera depth texture, which is what the shadows cast on). You want those two use cases to behave different depending on if it’s being used as a shadow caster or shadow receiver, but Unity does not offer a consistent way to test for one or the other. A custom shadow caster pass can be used instead of the addshadow parameter, but it doesn’t avoid that fundamental problem.

I started a thread several years ago on the topic of discerning between the two, and the hack in the first post is still today the only reliable way to do it.

You can put that bit of code into your surf function, and only do the clip() in the case of a perspective camera, which will hopefully only be during the shadow receiving and main scene rendering.

2 Likes