Shader stencil and Sprite Masks not working together

Unity Engine
1

Hi everyone,
I’m working on a game where I have ‘shadows’ made of semi transparent sprites. The game has a day/night cycle and as such the sprites rotate and scale constantly based on the time of day. Their alpha is also calculated against the time of day and changes constantly too. This is all working really well except if shadows overlap, as you’d expect, both semi transparent sprites are visible. I do not want them to blend like that.

The problem:

The desired result:

(I simply turned the sprite renderers off to demonstrate)

The general idea, which does not work on my shadows, seems to be:
-all shadows need to use the same material and shader.
-said shader is a copy of sprites/default but uses stencil to do something like:

Stencil {
Ref 2
Comp NotEqual
Pass Replace
}

I, however, can make this shader work fine on other objects, just not my shadows.

Eventually, I came to the realisation that it doesn’t work because the shadows also use sprite masks. I have two different types of shadows, one for the general time of day and one for reacting to close light sources, and I use masks to make them visible and invisible respectively (i.e. in close proximity to a light source, you cannot see the ‘time of day’ shadow but can instead see the shadow rotated inversely from the light source, such as a camp fire). It unfortunately means I can’t not use sprite masks and mask interaction.

I assume, therefore, that the sprite mask is making use of the stencil buffer, and I’m essentially trying to do two things at once. If I change the mask interaction on the shadows, I essentially turn the shadows into sprite masks. Not what I want.

TL;DR
Is there any way I can use the Stencil buffer and sprite mask interaction of a sprite renderer on the same sprite?

OTHERWISE:
I have seen it suggested I could make a separate camera that only renders the shadows, put them into a render texture, run a shader or some sort on that (I guess test if alpha > x and then if so lower to x?) and re-render that, so they are all one blob? This might even run faster, but I have no idea how to do it and can’t find any documentation that explains any of it.

If anyone can shine some light on this (because, you know, shadows…) I’d be more than grateful :slight_smile:

2

In case anyone comes across this in the future and has the same problem I had, I found a solution!

It involves four slight variants on the sprites/default shader, which you can download from the usual unity downloads page.

I stopped using sprite masks, and instead wrote my own shaders that do what sprite masks do, but also allow me to have this added stencil functionality. Each shader remains a single pass shader and only needs a few changes.

You need to remove all your sprite masks and set mask interaction for all your sprites to be none. Sprite masks you’ll be making are sprites set with a different material, they just use the standard sprite renderer. I suggest for 2D games putting them on a layer underneath everything so they’re not visible.

These stencils need to be inside a shader pass before CGPROGRAM.

Sprite Mask:

Stencil{
Ref 1
Comp Always
Pass Replace
}

This says ‘always write the stencil buffer to 1’ for the shape that your sprite mask is.

Visible Inside Mask:

Stencil{
Ref 1
Comp Equal
}

This says ‘is the buffer 1? If yes, draw’. The mask will set the buffer to 1, so this will only be visible inside your mask.

Visible Outside Mask with Sprite Blending:

Stencil{
Ref 0
Comp Equal
Pass DecrWrap
}

This says ‘is the buffer 0? (which it will be outside the mask), if so, decrease to 255’. The reason for this is that if another shadow is in the same place, it’ll see a buffer of 255, so the comparison check will fail and it’ll not draw.

Visible Outside Mask without Blending:

Stencil{
Ref 1
Comp NotEqual
}

This says ‘is the buffer 1? If no, draw’. This doesn’t care about the merging and simply checks if it’s out of a mask. This is important as I had a lot of issues before using this fourth shader where the blending sprites were acting as a mask for sprites visible outside of mask on a different sorting layer to the blending ones. No idea what that’s about, but I fixed it so I’m happy.

Finally, another important code is the following:

First, replace the #pragma fragment SpriteFrag to be:

#pragma fragment frag

Then underneath the #pragma lines, before ENDCG, add a new fragment section which you’ve just referenced:

fixed4 frag(v2f IN) : SV_Target
{
fixed4 c = tex2D(_MainTex, IN.texcoord) * IN.color;
if (c.a<0.1) clip (-1);
c.rgb *= c.a;
return c;
}

This cuts off the sprites properly to their transparency, so sprites of any shape will work. Otherwise, you’ll have weird blank areas surrounding your sprites.

And you’re done! Enjoy, hopefully someone somewhere finds this helpful. Last week I knew nothing about shaders, today I managed to solve this :slight_smile:

9 Likes
3

Hi, thanks for your instructions, but i just cant get it run, can u upload your shader file as an example. Thanks

2 Likes
4

bump.
Please put the whole shader

2 Likes
5

For anyone wondering how to get this effect with, for example, a standard lit or unlit URP sprite shader. Create a new shader graph of the type you require (lit/unlit). Hook this graph up with a SampleTexture2D (make sure this is named MainTex). Plug the color and alpha into the outputs on shader graph. This is now a completely normal unaffected sprite shader. In editor, click ‘view generated code’. Copy the generated code into a new .shader file. Now you can add the aforementioned stencil in and voila you have a graphless URP sprite shader with stencil manips. I use this exact technique to make Tilemaps which act as sprite masks!
TL;DR
-You don’t need OP’s whole shader, you can add the stencil feature to your own shaders for your own specific needs!

1 Like
6

I had some struggles getting it to work, turns out the logic here is correct but it’s critical to make sure the mask shader renders before your other shaders so it can write the stencil. “Queue”=“Geometry-100” seems to do it. Full mask shader here:

(BTW this also works with SpriteShapeRenderer in case anyone wants to use those as a mask)

Shader"Custom/SpriteCustomMask"
{
    Properties
    {
        [PerRendererData] _MainTex ("Sprite Texture", 2D) = "white" {}
        _Color ("Tint", Color) = (1,1,1,1)
        [MaterialToggle] PixelSnap ("Pixel snap", Float) = 0
        [HideInInspector] _RendererColor ("RendererColor", Color) = (1,1,1,1)
        [HideInInspector] _Flip ("Flip", Vector) = (1,1,1,1)
        [PerRendererData] _AlphaTex ("External Alpha", 2D) = "white" {}
        [PerRendererData] _EnableExternalAlpha ("Enable External Alpha", Float) = 0
    }

    SubShader
    {
        Tags
        {
            "Queue"="Geometry-100"
            "IgnoreProjector"="True"
            "RenderType"="Opaque"
            "PreviewType"="Plane"
            "CanUseSpriteAtlas"="True"
        }

Cull Off

Lighting Off

ZWrite Off

Blend One OneMinusSrcAlpha

Stencil
{
    Ref 1

    Comp always
    Pass replace
}

ColorMask 0

        Pass
        {
  CGPROGRAM
				#pragma vertex vert
				#pragma fragment frag
				#pragma multi_compile _ PIXELSNAP_ON
#include "UnityCG.cginc"


sampler2D _MainTex;
struct appdata_t
{
    float4 vertex : POSITION;
    float4 color : COLOR; //this is from the spriterenderer, it turns out
    float2 texcoord : TEXCOORD0;
};

struct v2f
{
    float4 vertex : SV_POSITION;
    fixed4 color : COLOR;
    float2 texcoord : TEXCOORD0;
};


v2f vert(appdata_t IN)
{
    v2f OUT;
    OUT.vertex = UnityObjectToClipPos(IN.vertex);
    OUT.texcoord = IN.texcoord;
    OUT.color = IN.color;
    
#ifdef PIXELSNAP_ON
					OUT.vertex = UnityPixelSnap(OUT.vertex);
#endif

    return OUT;
}




fixed4 frag(v2f IN) : SV_Target
{
    fixed4 c = fixed4(1, 1, 1, 0);//tex2D(_MainTex, IN.texcoord);
    return c;
    
}

            ENDCG
        }
    }
}