Cut-Out Shader casts wrong shadows

[launzone]

Hey,
I am working on a custom cut-out shader. (i am using custom toon shaders for all elements in my game that receive “noisy” shadows) I got the part working that the albedo gets cut off based on the alpha of a texture. The shader doesn’t cast and receive shadows in the right way yet though.
It seems like it casts and receives the shadow using the full geometry and not with the cut out texture.

i don’t really have an idea how to go about this, i tried looking at the standard and other cut out shaders but they are black boxes for me. any help would be appreciated!

this are the tags and pragma and include things:

Tags {
                "LightMode" = "ForwardBase"
                "PassFlags" = "OnlyDirectional"
                "Queue" = "AlphaTest"
                "IgnoreProjector" = "True"
                "RenderType" = "TransparentCutout"
            }

            Cull off

            CGPROGRAM
            #pragma vertex vert addshadow fullforwardshadows
            #pragma fragment frag
            #pragma multi_compile_fwdbase
          
            #include "UnityCG.cginc"
            #include "Lighting.cginc"
            #include "AutoLight.cginc"

this are the parts where i get the shadows:

v2f vert (appdata v)
            {
                v2f o;
                o.pos = UnityObjectToClipPos(v.vertex);
                o.worldPos = mul(unity_ObjectToWorld, v.vertex);
                o.uv = TRANSFORM_TEX(v.uv, _MainTex);
                o.worldNormal = UnityObjectToWorldNormal(v.normal);
                o.viewDir = WorldSpaceViewDir(v.vertex);
                o.color = v.color;
                TRANSFER_SHADOW(o)
                return o;
            }

here is where i do the cut out part:

float4 albedo = tex2D(_MainTex, i.uv);
                clip(tex2D(_MainTex, i.uv).a * albedo.a - _Cutoff);

[Invertex]

You need to make a ShadowCaster pass that just outputs depth using this same shader math you have here.
You can see an example in the “Implementing Shadow Casting” section near the bottom of this documentation:

Unity - Manual: Custom shader fundamentals

[bgolus]

@launzone

#pragma vertex vert addshadow fullforwardshadows

I just wanted to point out this line. addshadow and fullforwardshadows are optional parameters for the #pragma surface used by Surface Shaders. They have no affect on vertex fragment shaders. Surface Shaders are vertex fragment shader generators, and can generate multiple passes including the "LightMode"="ForwardBase" pass you have above, as well as several others. If you use the addshadow it generates a "LightMode"="ShaderCaster" pass.

If you’re writing the vertex fragment shader manually, you’ll need to also write the shadow caster pass manually, like @Invertex mentioned above.

[launzone]

okay thank you both i will give this a try and report back when i figure it out!

[launzone]

@Invertex

You need to make a ShadowCaster pass that just outputs depth using this same shader math you have here.
You can see an example in the “Implementing Shadow Casting” section near the bottom of this documentation:
https://docs.unity3d.com/Manual/SL-VertexFragmentShaderExamples.html

I understand by following this documentation to implement a “standard” shadow casting pass but I don’t see how to change it in any way? It looks exactly the same. The doc mentions : "… object has custom vertex shader based deformations, or has alpha cutout / semitransparent parts… " which is actually both true of my shader but I just don’t see what to do about it. Maybe I just don’t understand the basics of making a shadow casting pass, are there some basic tutorials on this? All I see are just black-boxes like “SHADOW_CASTER_FRAGMENT(i)” and I have no idea how I can affect the shadow that is being cast.
Also, can you tell me what you mean when you say the ShadowCaster pass outputs depth? I don’t see the fragment shader outputting color or anything like I would be used to in the example in the documentation.
Sorry for all the questions! I thought I grasped shaders a little bit better by now but I keep getting confused by all these unity functions and included stuff that is predefined.

[launzone]

i guess i can see how it will be easy to do the same transformations to the vertex in the vert of the shadow caster pass but how do i use the texture of cutout?

[bgolus]

See this built in shader:

TwoTailsGames/Unity-Built-in-Shaders/blob/master/DefaultResourcesExtra/AlphaTest-VertexLit.shader#L129

The short version is any code you’re doing in your forward base pass that would affect the shape / shadow of the object need to also be done in the shadow caster pass too.

On a side note, your example shader code above is:

float4 albedo = tex2D(_MainTex, i.uv);
clip(tex2D(_MainTex, i.uv).a * albedo.a - _Cutoff);

Why are you multiplying the alpha value of _MainTex by itself?

[launzone]

okay, i think i figured it out, this post was very helpful.

in my shadowcaster pass i discarded based on texture alpha:

float4 frag(v2f i) : SV_Target
            {
                float a = tex2D(_MainTex, i.uv).a;
                if (a < _Cutoff) {
                    discard;
                }
                SHADOW_CASTER_FRAGMENT(i)
            }

thank to bgolus and invertex for pointing me in the right direction!

[launzone]

@bgolus

See this built in shader:
https://github.com/TwoTailsGames/Unity-Built-in-Shaders/blob/master/DefaultResourcesExtra/AlphaTest-VertexLit.shader#L129

The short version is any code you’re doing in your forward base pass that would affect the shape / shadow of the object need to also be done in the shadow caster pass too.

On a side note, your example shader code above is:

float4 albedo = tex2D(_MainTex, i.uv);
clip(tex2D(_MainTex, i.uv).a * albedo.a - _Cutoff);

Why are you multiplying the alpha value of _MainTex by itself?

yes upon closer inspection that didn’t make much sense