UI Custom Shader/Material is ignoring masking

Hey guys,

I have a custom shader which colorizes sprites and UI images. It works perfectly! I currently have a scroll rect that will display unlocked objects which I am dynamically colorizing so they have the color swap shader.

I discovered that the unity mask does not play nicely with images that use custom materials / shaders. I took the default ui shader and have been dissecting it trying to figure out what bits I need to transfer over to my custom shader so it masks properly. I found a bunch of resources online regarding the stencil related stuff, and transferred all of that over. But there is one line in particular from the unity default ui shader that when I comment it out, the masking doesn’t work there either. So it seems like I need this line, I’m just struggling to make the shader not error out. I’m not very good with shaders.

Here is the unity default ui shader

Shader "UI/Default"
{
   Properties
   {
       [PerRendererData] _MainTex("Sprite Texture", 2D) = "white" {}
       _Color("Tint", Color) = (1,1,1,1)

       _StencilComp("Stencil Comparison", Float) = 8
       _Stencil("Stencil ID", Float) = 0
       _StencilOp("Stencil Operation", Float) = 0
       _StencilWriteMask("Stencil Write Mask", Float) = 255
       _StencilReadMask("Stencil Read Mask", Float) = 255

       _ColorMask("Color Mask", Float) = 15

       [Toggle(UNITY_UI_ALPHACLIP)] _UseUIAlphaClip("Use Alpha Clip", Float) = 0
   }

       SubShader
       {
           Tags
           {
               "Queue" = "Transparent"
               "IgnoreProjector" = "True"
               "RenderType" = "Transparent"
               "PreviewType" = "Plane"
               "CanUseSpriteAtlas" = "True"
           }

           Stencil
           {
               Ref[_Stencil]
               Comp[_StencilComp]
               Pass[_StencilOp]
               ReadMask[_StencilReadMask]
               WriteMask[_StencilWriteMask]
           }

           Cull Off
           Lighting Off
           ZWrite Off
           ZTest[unity_GUIZTestMode]
           Blend SrcAlpha OneMinusSrcAlpha
           ColorMask[_ColorMask]

           Pass
           {
               Name "Default"
           CGPROGRAM
               #pragma vertex vert
               #pragma fragment frag
               #pragma target 2.0

               #include "UnityCG.cginc"
               #include "UnityUI.cginc"

               #pragma multi_compile_local _ UNITY_UI_CLIP_RECT
               #pragma multi_compile_local _ UNITY_UI_ALPHACLIP

               struct appdata_t
               {
                   float4 vertex   : POSITION;
                   float4 color    : COLOR;
                   float2 texcoord : TEXCOORD0;
                   UNITY_VERTEX_INPUT_INSTANCE_ID
               };

               struct v2f
               {
                   float4 vertex   : SV_POSITION;
                   fixed4 color : COLOR;
                   float2 texcoord  : TEXCOORD0;
                   float4 worldPosition : TEXCOORD1;
                   UNITY_VERTEX_OUTPUT_STEREO
               };

               sampler2D _MainTex;
               fixed4 _Color;
               fixed4 _TextureSampleAdd;
               float4 _ClipRect;
               float4 _MainTex_ST;

               v2f vert(appdata_t v)
               {
                   v2f OUT;
                   UNITY_SETUP_INSTANCE_ID(v);
                   UNITY_INITIALIZE_VERTEX_OUTPUT_STEREO(OUT);
                   OUT.worldPosition = v.vertex;
                   OUT.vertex = UnityObjectToClipPos(OUT.worldPosition);

                   OUT.texcoord = TRANSFORM_TEX(v.texcoord, _MainTex);

                   OUT.color = v.color * _Color;
                   return OUT;
               }

               fixed4 frag(v2f IN) : SV_Target
               {
                   half4 color = (tex2D(_MainTex, IN.texcoord) + _TextureSampleAdd) * IN.color;

                   #ifdef UNITY_UI_CLIP_RECT
                   color.a *= UnityGet2DClipping(IN.worldPosition.xy, _ClipRect);
                   #endif

                   #ifdef UNITY_UI_ALPHACLIP
                   clip(color.a - 0.001);
                   #endif

                   return color;
               }
           ENDCG
           }
       }
}

Here is the palette swap shader that I’m using:

Shader "Sprites/ColorSwapV3"
{
   Properties
   {
       [PerRendererData] _MainTex("Sprite Texture", 2D) = "white" {}
       _SwapTex("Color Data", 2D) = "transparent" {}
       _Color("Tint", Color) = (1,1,1,1)
       [MaterialToggle] PixelSnap("Pixel snap", Float) = 0
       [HideInInspector] _RendererColor("RendererColor", Color) = (1,1,1,1)
       [HideInInspector] _Flip("Flip", Vector) = (1,1,1,1)
       [PerRendererData] _AlphaTex("External Alpha", 2D) = "white" {}
       [PerRendererData] _EnableExternalAlpha("Enable External Alpha", Float) = 0

       _StencilComp("Stencil Comparison", Float) = 8
       _Stencil("Stencil ID", Float) = 0
       _StencilOp("Stencil Operation", Float) = 0
       _StencilWriteMask("Stencil Write Mask", Float) = 255
       _StencilReadMask("Stencil Read Mask", Float) = 255

       _ColorMask("Color Mask", Float) = 15

       [Toggle(UNITY_UI_ALPHACLIP)] _UseUIAlphaClip("Use Alpha Clip", Float) = 0
   }

       SubShader
       {
           Tags
           {
               "Queue" = "Transparent"
               "IgnoreProjector" = "True"
               "RenderType" = "Transparent"
               "PreviewType" = "Plane"
               "CanUseSpriteAtlas" = "True"
           }

           Stencil
           {
               Ref[_Stencil]
               Comp[_StencilComp]
               Pass[_StencilOp]
               ReadMask[_StencilReadMask]
               WriteMask[_StencilWriteMask]
           }

           Cull Off
           Lighting Off
           ZWrite Off
           ZTest[unity_GUIZTestMode]
           Blend SrcAlpha OneMinusSrcAlpha
           ColorMask[_ColorMask]

           Pass
           {
           CGPROGRAM
               #pragma vertex vert
               #pragma fragment frag
               #pragma target 2.0
               #pragma multi_compile_instancing
               #pragma multi_compile _ PIXELSNAP_ON
               #pragma multi_compile _ ETC1_EXTERNAL_ALPHA
               #include "UnitySprites.cginc"

               v2f vert(appdata_t IN)
               {
                   v2f OUT;

                   UNITY_SETUP_INSTANCE_ID(IN);
                   UNITY_INITIALIZE_VERTEX_OUTPUT_STEREO(OUT);

               #ifdef UNITY_INSTANCING_ENABLED
                   IN.vertex.xy *= _Flip.xy;
               #endif

                   OUT.vertex = UnityObjectToClipPos(IN.vertex);
                   OUT.texcoord = IN.texcoord;
                   OUT.color = IN.color * _Color * _RendererColor;

                   #ifdef PIXELSNAP_ON
                   OUT.vertex = UnityPixelSnap(OUT.vertex);
                   #endif

                   return OUT;
               }

               sampler2D _SwapTex;
               float4 _ClipRect;

               fixed4 frag(v2f IN) : SV_Target
               {
                   fixed4 c = SampleSpriteTexture(IN.texcoord);
                   fixed4 swapCol = tex2D(_SwapTex, float2(c.x, 0));
                   fixed4 final = lerp(c, swapCol, swapCol.a);
                   final.a = c.a * IN.color.a;
                   final.rgb *= final.a * IN.color.rgb;

               #ifdef UNITY_UI_CLIP_RECT
                   c.a *= UnityGet2DClipping(IN.worldPosition.xy, _ClipRect);
               #endif

               #ifdef UNITY_UI_ALPHACLIP
                   clip(c.a - 0.001);
               #endif

                   return final;
               }
           ENDCG
           }
       }
}

The line that makes masking break with unitys default ui shader is:

 #ifdef UNITY_UI_CLIP_RECT
                   c.a *= UnityGet2DClipping(IN.worldPosition.xy, _ClipRect);
               #endif

I’m not entirely sure if I’m using it right, but it is my only lead at this point! Any suggestions or tips would be greatly appreciated! I am just trying to make the color swap shader work with ui masking :slight_smile:

Thanks guys!

1 Like

bump!
@bgolus sorry for the ping, but your name comes up in all my other shader searches lol

When I do that it says invalid subscript ‘worldPosition’ - but then when I add a struct for v2f to add a world pos it says i have more than one definition for v2f, i notice that goes away if i remove inlude unitysprites.cginc but then it blows up on my SampleSpriteTexture call lol this whole process has been 1 step forward, 5 backwards :frowning:

If you’re using this for a UI shader, you should build off of the UI shader instead of the Sprite shader. That’ll fix a lot of the headache you’re experiencing trying to mix the two.

2 Likes

So, it is primarily a sprite shader. The reason I am currently using it in the UI is for part of my game where I need to utilize UI related things, it is kind of hacky, but so i’ll apply a sprite renderer as well as respective sprite that I need, run the coloring code then on that same object I just pass in the new colored material to its ui image, so the ui image is using this material and recolors successfully - the problem is that shader/material ignores my mask.

I also tried generating a sprite during run time using the custom colored texture, that almost was working but it wasnt translating property to the ui image. But basically, I’m trying to make this sprite shader work for ui elements where masking is needed. I think all the pieces are there, I’m just overlooking something silly. Do you have any tips for extracting the colorized texture from the shader? I feel like I was close with that approach, I was able to the see the color that it should be and it was masking however, it wasnt drawing the shape, just displaying the custom color lol

I’m wondering if it is possible to just add a third texture to the shader called combineTex and then just make combineTex = mainTex + SwapTex? Then when I create a sprite during run time I should be able to just grab the combineTex, and pass that sprite to the ui image sprite and leave the material untouched so i will have the masking.

If you need a shader to work with various Canvas UI stuff, you want to use the UI-Default.shader, or a shader built off of that. The only reason to use the Sprite shader is if you’re looking to support hardware that doesn’t support ETC2 and thus needs to store the sprite alpha as a separate texture … hardware which Unity does not support anymore and is a feature that’s been stripped from the URP 2D renderer, or pixel snapping … which the Canvas renderer handles itself without needing the material setting so is also unneeded.

All the other features in the Sprite shaders are related to features that the Sprite Renderer component uses, which if you’re using this as part of the UI, you shouldn’t be using as they also won’t support the UI features you want.

2 Likes

No. Because that’s not something you can really do.

You could do something like Blit() to a render texture using a material that colorizes a texture, and then use that render texture in the UI Image component. But there really isn’t a reason to do that since you can just add the code you’re using in the sprite shader to the UI-Default.shader and it’ll work just as well.

Thank you so much for the information and guidance! Would you be able to help me migrate the custom coloring over to the ui default shader?

To this question, give it a try yourself again, and if you’re having problems understanding parts feel free to come back and ask questions. There’s not that much extra code in the color swap shader. Try comparing to the default sprite shader to see what was added, and try adding those lines to the default UI shader.

So I was able to have it retooled for use in the UI! But now I am facing another issue. After talking with the developer of the shader, basically the shader doesn’t handle compression at all, which up until now has not been an issue for me. I am getting the proper color swapping on PC / Mac / iOS and MOST android… but on my Galaxy S10 it just refuses to color at all. I’m not sure how I can get around this, this is a pixel art game so the texture atlas that I am using is set to Point (no filter) and no compression, I’m not sure if that is the issue or not. I am able to get the recoloring to work on my S10 with different images which is odd. It seems like the S10 doesn’t like coloring pixel art.

Was wondering if there was a way to modify the shader further to ensure it works on the s10 or possibly something I can do in unity? I tried adding the two shaders to the shaders to include in the build and that didn’t seem to do anything.