What could cause a stencil shader to fail on a macOS build but not on Windows?

Unity Engine
,
1

Hi,

I’ve noticed a shader problem in my game with some macOS builds. One person in particular has a Macbook with a AMD Radeon Pro 5300M. Unfortunately I don’t have a Mac to test this on, and I was never able to figure out how to run it through a VM.

Basically, I use a shader with a Stencil pass that acts as a mask. Wherever a sprite is drawn with this shader, it should “cut” a hole into the Masked sprites below it. In Mac builds, at least some of the time, the masking doesn’t seem to work — the lower sprite is fully visible.

Here’s the shader:

Shader "Custom/UI/Stencil (Unlit)"
{
    Properties
    {
        [PerRendererData] _MainTex ("Sprite Texture", 2D) = "white" {}
        _Color ("Tint", Color) = (1,1,1,1)
        [MaterialToggle] PixelSnap ("Pixel snap", Float) = 0
        [HideInInspector] _RendererColor ("RendererColor", Color) = (1,1,1,1)
        [HideInInspector] _Flip ("Flip", Vector) = (1,1,1,1)
        [PerRendererData] _AlphaTex ("External Alpha", 2D) = "white" {}
        [PerRendererData] _EnableExternalAlpha ("Enable External Alpha", Float) = 0

        [MaterialToggle] _HideTexture ("Hide Texture", Float) = 0

        [Toggle(UNITY_UI_ALPHACLIP)] _UseUIAlphaClip ("Use Alpha Clip", Float) = 0
        _AlphaClip("Alpha Clip", Range(0, 1)) = 0.01

        [IntRange] _StencilRef ("Stencil Ref", Range(0,10)) = 1
        [IntRange] _StencilComp ("Stencil Comparison", Range(0,8)) = 3
        [IntRange] _StencilPass ("Stencil action to perform on succeed", Range(0,7)) = 2
        [IntRange] _StencilFail ("Stencil action to perform on fail", Range(0,7)) = 0
        [IntRange] _StencilZFail ("Stencil action to perform on zfail", Range(0,7)) = 0
    }

    SubShader
    {
        Tags
        {
            "Queue"="Transparent"
            "IgnoreProjector"="True"
            "RenderType"="Transparent"
            "PreviewType"="Plane"
            "CanUseSpriteAtlas"="True"
        }

        Cull Off
        ZWrite Off
        Blend One OneMinusSrcAlpha
        Lighting Off       
        
        Stencil
        {           
            Ref [_StencilRef]
            Comp [_StencilComp]
            Pass [_StencilPass]
            Fail [_StencilFail]
            ZFail [_StencilZFail]
        }

        Pass
        {
            CGPROGRAM
            #pragma vertex vert
            #pragma fragment frag
           
            #include "UnityCG.cginc"

            #pragma multi_compile_local _ UNITY_UI_ALPHACLIP

            struct VertInput
            {
                float4 vertex : POSITION;
                float4 tangent : TANGENT;
                float3 normal : NORMAL;
                float4 texcoord0 : TEXCOORD0;
                float4 texcoord1 : TEXCOORD1;
                fixed4 color : COLOR;
            };

            struct FragInput
            {
                float4 pos : SV_POSITION;
                float4 color : COLOR0;
                float2 texcoord0 : TEXCOORD0;
            };

            sampler2D _MainTex;
            float4 _MainTex_ST;
            fixed4 _Color;
            float _HideTexture;

            float _AlphaClip;

            FragInput vert(VertInput input)
            {
                FragInput output;
                UNITY_INITIALIZE_OUTPUT(FragInput, output);

                output.pos = UnityObjectToClipPos(input.vertex);
                output.texcoord0 = TRANSFORM_TEX(input.texcoord0, _MainTex);
                output.color = input.color * _Color;

                return output;
            }
           
            fixed4 frag (FragInput input) : SV_Target
            {
                // sample the texture
                fixed4 col = tex2D(_MainTex, input.texcoord0) * input.color;
                col.rgb *= col.a;
               
                clip(col.a < 0.01f ? -1 : 1);

                //alpha clip
                #ifdef UNITY_UI_ALPHACLIP
                clip (col.a - _AlphaClip);
                #endif

                if (_HideTexture == 1)
                    return fixed4(0,0,0,0);
                else
                    return col;
            }
            ENDCG
        }
    }

Fallback "Transparent/VertexLit"
}

I really don’t know much about graphics programming, so I’m hoping that there’s just some hardware-specific line that I need to add.

Alternatively, if this shader relies on some hardware feature that’s missing from the above mentioned GPU, I can try to research a workaround. But right now, I don’t know what problem I’m trying to work around, and it’s hard to troubleshoot without being able to dive in there with the frame debugger.

2

Update: I never found out what the problem was, but I solved it by using this shader as a base to replace my own shader: Stencil · supyrb/ConfigurableShaders Wiki · GitHub

FWIW, the issue had something to do with Apple’s Metal graphics API, because it didn’t occur when I switched to OpenGL in the Player settings.