Aliasing When Using UI Mask

I’ve just used the Mask from the new UI for the first time, using a rounded rect as the mask image. The image being masked has really bad aliasing on its corners, where the mask image is rounded with smooth antialiasing.

Is this supposed to happen? Is there a way to get smooth curves with a UI mask?

I made a little shader that allows you to use a mask to hide part of a sprite smoothly.

  1. Create a new shader and past the code below.
  2. Create a new material and choose Ui > Mask as the shader.
  3. Assign the new material to your sprite.
  4. Assign your mask to the Mask Texture slot in your material.
  5. Voilà!

It is a bit harder on the system than the built-in solution, but it works.

Shader "UI/Mask"
{
	Properties
	{
		[PerRendererData] _MainTex ("Sprite Texture", 2D) = "white" {}
		_MaskTex ("Mask Texture", 2D) = "white" {}
		_Color ("Tint", Color) = (1,1,1,1)
		
		_StencilComp ("Stencil Comparison", Float) = 8
		_Stencil ("Stencil ID", Float) = 0
		_StencilOp ("Stencil Operation", Float) = 0
		_StencilWriteMask ("Stencil Write Mask", Float) = 255
		_StencilReadMask ("Stencil Read Mask", Float) = 255

		_ColorMask ("Color Mask", Float) = 15
	}

	SubShader
	{
		Tags
		{ 
			"Queue"="Transparent" 
			"IgnoreProjector"="True" 
			"RenderType"="Transparent" 
			"PreviewType"="Plane"
			"CanUseSpriteAtlas"="True"
		}
		
		Stencil
		{
			Ref [_Stencil]
			Comp [_StencilComp]
			Pass [_StencilOp] 
			ReadMask [_StencilReadMask]
			WriteMask [_StencilWriteMask]
		}

		Cull Off
		Lighting Off
		ZWrite Off
		ZTest [unity_GUIZTestMode]
		Blend SrcAlpha OneMinusSrcAlpha
		ColorMask [_ColorMask]

		Pass
		{
		CGPROGRAM
			#pragma vertex vert
			#pragma fragment frag
			#include "UnityCG.cginc"
			
			struct appdata_t
			{
				float4 vertex   : POSITION;
				float4 color    : COLOR;
				float2 texcoord : TEXCOORD0;
			};

			struct v2f
			{
				float4 vertex   : SV_POSITION;
				fixed4 color    : COLOR;
				half2 texcoord  : TEXCOORD0;
			};
			
			fixed4 _Color;
			
			
			v2f vert(appdata_t IN)
			{
				v2f OUT;
				OUT.vertex = mul(UNITY_MATRIX_MVP, IN.vertex);
				OUT.texcoord = IN.texcoord;
				#ifdef UNITY_HALF_TEXEL_OFFSET
				OUT.vertex.xy += (_ScreenParams.zw-1.0)*float2(-1,1);
				#endif
				OUT.color = IN.color * _Color;
				return OUT;
			}

			sampler2D _MainTex;
			sampler2D _MaskTex;
			
			fixed4 frag(v2f IN) : SV_Target
			{
				half4 color = tex2D(_MainTex, IN.texcoord) * IN.color;
				half4 mask = tex2D(_MaskTex, IN.texcoord);
				
				color.a *= mask.r;
				clip (color.a - 0.01);
				return color;
			}
		ENDCG
		}
	}
}

If you uncheck “Generate Mip Maps” on the sprite of mask it also helps a little :slight_smile:

Hey @alexandre-fiset

Thanks for the awesome shader!! It doesn’t seem to support alpha transparency though. See the attached picture as viewed with the UI/Mask shader compared to how it should look.
Would you be willing to adjust the shader to support transparency?

Appreciate it,
Kurt


doesnt work unless im setting it up wrong