Edge Culling with Custom Surface Shader?

I want to create a custom standard surface shader that uses different texture and color inputs to the built-in standard specular shader.

I’ve done this already and I’m happy enough with the results but there’s one remaining problem that I cannot find the solution to.

When I select a model using the built-in standard shader the highlighted edges of its mesh match up with the visible edge of the texture which is the behavior I want (I’m making some models for Tabletop Simulator and the edge highlighting functions the same in that game as it does in the Unity Editor).

However, when I select a model using my custom surface shader, the edges extend to the edge of the mesh and ignore the alpha of the texture. I’ve attached some images to demonstrate this issue.

Standard Shader Edge

Custom Shader Edge

I’ve tried several methods to generate the shader and this issue still persists. There must be some built-in function that Unity uses to render the edges like it does but I don’t know how to access it via the shader. I would appreciate any help with this issue.

EDIT: Here is the code for the shader:

Shader "Custom/Custom Surface Shader"
{
    Properties
    {
        [MainColor][HideInInpspector]_Color ("Color", Color) = (1,1,1,1)
        [MainTexture][NoScaleOffset]_Diffuse ("Diffuse", 2D) = "white" {}
        [NoScaleOffset]_Gloss ("Gloss", 2D) = "white" {}
        [NoScaleOffset]_Mask ("Mask", 2D) = "black" {}
        [Normal][NoScaleOffset]_Normal ("Normal", 2D) = "bump" {}
        [NoScaleOffset]_Specular ("Specular", 2D) = "gray" {}
        [HDR]_EmissionColor ("EmissionColor", Color) = (0,0,0,0)
        _MaskColorRed ("Mask Color Red", Color) = (1,1,1,1)
        _MaskColorGreen ("Mask Color Green", Color) = (1,1,1,1)
        _MaskColorBlue ("Mask Color Blue", Color) = (1,1,1,1)

        _Cutoff ("Alpha cutoff", Range(0,1)) = 0.5
    }
    SubShader
    {
        Tags
        {
            "RenderType"="TransparentCutout" "Queue"="Transparent-1" "IgnoreProjector"="True"
        }
        LOD 300

        CGPROGRAM
        #pragma surface surf StandardSpecular alphatest:_Cutoff fullforwardshadows addshadow

        #pragma target 3.0

        fixed4 _Color;

        fixed3 _ColorRed;
        fixed3 _ColorGreen;
        fixed3 _ColorBlue;
        fixed3 _EmissionColor;

        sampler2D _Diffuse;
        sampler2D _Gloss;
        sampler2D _Mask;
        sampler2D _Normal;
        sampler2D _Specular;

        struct Input
        {
            float2 uv_Diffuse;
        };
        
        UNITY_INSTANCING_BUFFER_START(Props)
        UNITY_INSTANCING_BUFFER_END(Props)

        void surf(Input IN, inout SurfaceOutputStandardSpecular o)
        {
            fixed4 tex = tex2D(_Diffuse, IN.uv_Diffuse);
            fixed4 mask = tex2D(_Mask, IN.uv_Diffuse);
            fixed3 c = tex.rgb * _Color.rgb;
            fixed3 red = LerpWhiteTo(_ColorRed.rgb, mask.r);
            fixed3 green = LerpWhiteTo(_ColorGreen.rgb, mask.g);
            fixed3 blue = LerpWhiteTo(_ColorBlue.rgb, mask.b);
            o.Albedo = c.rgb * red * green * blue;

            o.Specular = tex2D(_Specular, IN.uv_Diffuse).rgb;
            o.Smoothness = tex2D(_Gloss, IN.uv_Diffuse).r;
            o.Normal = UnpackNormal(tex2D(_Normal, IN.uv_Diffuse));
            o.Emission = mask.a * _EmissionColor;
            o.Alpha = tex.a * _Color.a;
        }
        ENDCG
    }
    FallBack "Diffuse"
}

Your shader should have a line starting with #pragma surface somewhere. Add addshadow to the end of that line and see if that fixes it. This makes Unity generate a shadow caster pass, which I think is what’s used for the outline as well.

Thanks for the help but I’ve tried that to no effect.

I found out the solution myself. The surface shader code specifically wants a property called “_MainTex” to be the source of the “main texture”, which was the diffuse texture in my case. Swapping out the property name fixed all the issues I was having. Hopefully if someone else is having a similar issue they’ll find this solution and it works.