Sprite Renderer Mask on Specific Sorting Layer

2D
1

Hi guys,

I’m making a 2D game that zombies come out of the ground. The visuals are similar to Plants Vs. Zombies. There are 4 lanes that spawn zombies. I’m having trouble to find a good solution for that. There are 3 sprites that I’m working it: the background, the tomb and the zombie. I could simply get piece of the ground image and put above the zombie, but that would not be a great solution, because I want to change the position of the tombs and change the background image. I would have to slice the background for each situation. The obvious solution was to mask the zombie part that is beneath the ground. I found some mask shader on the Internet, and for all that worked on Unity 5, they masked everything that was above the mask. I could not configure the shader to select the sprites that I wanted to mask or to specify any parameter that limited the layer that the mask would affect. Here are the images of the game to illustrate what I’m saying:

This is two zombies getting out of the tomb. The Tomb on the center of the image is on the Sorting Layer “Tomb1”, the zombie on the Sorting Layer “Zombie1”, the tomb beneath in on the Sorting Layer “Tomb2”, and the zombie “Zombie2”.

When I apply the mask at the Sorting Layer “Zombie1”, it masks everything above, including the “Tomb2” and “Zombie2”. My question is if it is possible to limit the mask to have effect only in the Sorting Layer that the mask is included. The shader that I’m using is this:

 Shader "Custom/MaskTest" {
     Properties
     {
         _MainTex ("Base (RGB) Alpha (A)", 2D) = "white" {}
         _Cutoff ("Base Alpha cutoff", Range (0,.9)) = .5
     }
 
     SubShader { 
         Tags {"Queue" = "Transparent+1"}
          Offset 0, -1
         ColorMask 0
         ZWrite On
         Pass
         {
             AlphaTest Greater [_Cutoff]     
             SetTexture [_MainTex] {
                 combine texture * primary, texture
             }
         }
     }
}

I found that there is an Alpha Version of Unity with Mask Component to SpriteRenderer, but I don’t have the Pro License, and this game will be release before december (the month that this feature will be released, according to Unity Road Map). I appreciate for reading this far.

2

We have SpriteMasking coming with Unity 5.4 (Platform roadmaps | Unity)… ie is March.

Here is what you can do if you want to try things on your own.
For Masks, create two versions of a shader… one to mark the mask’s presence, other to remove the mask after its done.

  1. Create a shader that increments the stencil buffer whenever the mask is rendered. This shader discards a pixel if the mask pixel has alpha above a pre-selected cutoff (something like 0.2).
  2. Create a shader (or variation of the first) that decrements the stencil buffer whenever the mask is rendered. Also discards pixel if alpha is above the pre-selected cutoff.

For maskables (zombies), modify the Defaults-Shader available in Unity to use the stencil buffer and draw only if the stencil value > 0 (or whatever ref value you start with for the masks)
Encapsulate zombies in the same layer with Masks (start with ON, end with OFF for the layer)

1 Like
3

Thank you very much!! It worked!!! The keyword that I was missing was “Stencil”. I didn’t know the existence of this buffer and its use for 2D games. I found this forum thread: http://forum.unity3d.com/threads/is-it-possible-to-clip-a-sprite-with-another-sprite.254144/ and with a little googling and your tutorial I could make it work.
I’ll just check the shaders and share with you all.

4

Hi! Here are the shaders that solve this “problem”. First, the shader for the SpriteRenderer of the objects that will masked.

Shader "Sprites/Stencil Draw In Mask"
{
    Properties
    {
        [PerRendererData] _MainTex("Sprite Texture", 2D) = "white" {}
        _Color("Tint", Color) = (1,1,1,1)
        [MaterialToggle] PixelSnap("Pixel snap", Float) = 0
    }

    SubShader
    {
        Tags
        {
            "Queue" = "Transparent+1"
            "IgnoreProjector" = "True"
            "RenderType" = "Transparent"
            "PreviewType" = "Plane"
            "CanUseSpriteAtlas" = "True"
        }

        Cull Off
        Lighting Off
        ZWrite Off
        Blend One OneMinusSrcAlpha

        Pass
        {
            Stencil
            {
                Ref 0
                Comp Equal
            }

        CGPROGRAM
            #pragma vertex vert
            #pragma fragment frag
            #pragma multi_compile _ PIXELSNAP_ON
            #include "UnityCG.cginc"

            struct appdata_t
            {
                float4 vertex   : POSITION;
                float4 color    : COLOR;
                float2 texcoord : TEXCOORD0;
            };

            struct v2f
            {
                float4 vertex   : SV_POSITION;
                fixed4 color : COLOR;
                half2 texcoord  : TEXCOORD0;
            };

            fixed4 _Color;

            v2f vert(appdata_t IN)
            {
                v2f OUT;
                OUT.vertex = mul(UNITY_MATRIX_MVP, IN.vertex);
                OUT.texcoord = IN.texcoord;
                OUT.color = IN.color * _Color;
                #ifdef PIXELSNAP_ON
                OUT.vertex = UnityPixelSnap(OUT.vertex);
                #endif

                return OUT;
            }

            sampler2D _MainTex;
            sampler2D _AlphaTex;
            float _AlphaSplitEnabled;

            fixed4 SampleSpriteTexture(float2 uv)
            {
                fixed4 color = tex2D(_MainTex, uv);
                if (_AlphaSplitEnabled)
                    color.a = tex2D(_AlphaTex, uv).r;

                return color;
            }

            fixed4 frag(v2f IN) : SV_Target
            {
                fixed4 c = SampleSpriteTexture(IN.texcoord) * IN.color;
                c.rgb *= c.a;
                return c;
            }
        ENDCG
        }
    }
}

And the shader for the mask:

Shader "Sprites/Stencil Mask"
{
    Properties
    {
        [PerRendererData] _MainTex("Sprite Texture", 2D) = "white" {}
        _Color("Tint", Color) = (1,1,1,1)
        [MaterialToggle] PixelSnap("Pixel snap", Float) = 0
        [MaterialToggle] ShowTexture("Show Texture", Float) = 0
    }

        SubShader
        {
        Tags
        {
            "Queue" = "Transparent"
            "IgnoreProjector" = "True"
            "RenderType" = "Transparent"
            "PreviewType" = "Plane"
            "CanUseSpriteAtlas" = "True"
        }

        Cull Off
        Lighting Off
        ZWrite Off
        Blend One OneMinusSrcAlpha

        Pass
        {
            Stencil
            {
                Ref 1
                Comp always
                Pass replace
            }
        CGPROGRAM
            #pragma vertex vert
            #pragma fragment frag
            #pragma multi_compile _ PIXELSNAP_ON
            #pragma multi_compile _ SHOWTEXTURE_ON
            #include "UnityCG.cginc"

            struct appdata_t
            {
                float4 vertex   : POSITION;
                float4 color    : COLOR;
                float2 texcoord : TEXCOORD0;
            };

            struct v2f
            {
                float4 vertex   : SV_POSITION;
                fixed4 color : COLOR;
                half2 texcoord  : TEXCOORD0;
            };

            fixed4 _Color;

            v2f vert(appdata_t IN)
            {
                v2f OUT;
                OUT.vertex = mul(UNITY_MATRIX_MVP, IN.vertex);
                OUT.texcoord = IN.texcoord;
                #ifdef SHOWTEXTURE_ON
                OUT.color = IN.color * _Color;
                #else
                OUT.color = IN.color * _Color;
                OUT.color.rgb = 0;
                #endif
                #ifdef PIXELSNAP_ON
                OUT.vertex = UnityPixelSnap(OUT.vertex);
                #endif

                return OUT;
            }

            sampler2D _MainTex;

            fixed4 frag(v2f IN) : SV_Target
            {
                fixed4 c = tex2D(_MainTex, IN.texcoord) * IN.color;
                if (c.a < 0.2) discard;
                c.rgb *= c.a;
                c.a = 0;
                return c;
            }
        ENDCG
        }
    }
}

I created a copy of this mask shader changing the shader name to “Sprites/Stencil Mask Restore” and Stencil Ref value to 0. Then, I created the materials for the shaders mentioned. Finally, set up the scene as @sandboxed explained. First the mask sprites, then the sprites to be masked and finally the restore mask sprite. I put the last one as a child of the mask sprite, to easily fit one over the other, and set the appropriate material.
Here is the result, thanks to @sandboxed

5 Likes
5

Cool!!!
With SpriteMasking coming to Unity in 5.4, you will not need to bother about setting up the order of mask and maskables. Also masks will be able to affect multiple layers.

6

Just wanted to thank you for posting the shaders code, it works great and saved me a lot of trouble :slight_smile:

7

The game is available at Google Play: https://play.google.com/store/apps/details?id=com.funbites.thestarvingdead
Hope you like it.

8

Thanks for sharing the shader.

Could anyone point me in the direction of getting this to work with alpha blending?

9

I think you just have to copy and paste the shader that you want. Just add this part to the shader and follow those steps.

  • Stencil
  • {
  • Ref 0
  • Comp Equal
  • }
10

Nope. Hopefully 5.5. Masking multiple layers is a must, please Unity let’s not wait until 6.

1 Like
11

joaobsneto - Thanks, Thanks, Thanks !

12

Thank you for posting the shaders, joaobsneto, and for everyone who helped! :slight_smile:

13

Hi, I used this shader and it is working but it doesn’t have soft edges. Is it possible to use the alpha channel to get soft edged blending rather than the all or nothing threshold?

14

I don’t think so… but I’m not a shader expert. I think that stencil doesn’t support alpha blending. Am I right @sandboxed ?

15

5.4 is out and it seems like there is no masking for SpriteRenderer still.

16

yes no masking still.

17

In the Unity Roadmap it says…

@sandboxed what are “Workflow concerns”?
When do you think Sprite Masking will be available?
Thank you

18

Another solution I’m using, with unity 2017.1’s SpriteMask feature:

using UnityEngine;

namespace DCGame.UnityFeaturePatch
{
  [RequireComponent(typeof(SpriteMask))]
  public class SpriteMaskRangeFixer : MonoBehaviour
  {
    private static int index = 0;
    public SpriteRenderer[] targets;
    public SpriteMask mask;

    private void Start()
    {
      mask = GetComponent<SpriteMask>();

      foreach (var target in targets)
      {
        target.sortingOrder = index + 2;
        target.sortingLayerName = mask.sortingLayerName;
      }
      mask.isCustomRangeActive = true;
      mask.backSortingOrder = index + 1;
      mask.frontSortingOrder = index + 2;

      index += 2;

      if (index >= 1000)
      {
        index = 0;
      }
    }
  }
}