Is it possible to AlphaTest prior to writing to the Stencil Buffer?

Questions & Answers
1

I want to use the stencil buffer to store mask information for a 2D game. The idea is that first I draw a sprite only to the stencil buffer (no RGB) e.g. value = 1 and then any sprites/pixels drawn on top are tested against this and masked if the value is 1.

This is relatively simple but the problem I’m having is only writing to the stencil buffer in pixels where the src alpha is above a certain value rather than across the entire quad. I want the mask to follow the shape of the sprite i.e. by pixel rather than geometry.

I’ve tried including “AlphaTest Greater 0.5” but it doesn’t seem to be working.

Here’s my masking shader:

Shader "Custom/StencilMask" {
	Properties {
		_MainTex ("Base (RGB)", 2D) = "white" {}
	}
	 SubShader {
        Tags {"Queue"="Transparent" "IgnoreProjector"="True" "RenderType"="Transparent"}
		ZWrite Off Lighting Off Cull Off Fog { Mode Off } Blend Zero One
		LOD 110
        Pass {
        	// Only write to stencil where alpha is greater than 50%
			AlphaTest Greater 0.5
        
            Stencil {
            	// Set stencil value to one if we pass
                Ref 1
                // Don't compare, we are only interesting in writing
                Comp always
               	// This tells it to set the value, this should always pass because nothing is writing to the depth buffer.
                Pass replace
            }
        }
    }
}

Then the sprite to be masked has the following inside it’s pass:

	Stencil {
        // Compare zero with the stencil value.
        Ref 0
        // Passes if it's equal. Masked areas have value 1
        Comp equal
    }

This works as intended however the stencil buffer seems to be written into regardless of the alpha value of the pixel.

To restate the title question: Is it possible to AlphaTest prior to writing to the Stencil Buffer?

I have already found a work around which is to write to the depth buffer instead to mask pixels drawn below. However I would prefer to use the stencil buffer if possible as it gives me more fine tuned control over which sprites mask which other sprites.

Thanks in advance,

Tim

2

This is possible if you put an explicit discard statement in your shader.

For example:

fixed4 color(...): COLOR {
    fixed4 col = ...;
    if (col.a < 0.5) discard;
    return col;
}

I’ve tested this and it definitely works. I don’t know how efficient it is though - I know it’s often a bad idea to use discards. Still, it might be OK in your use-case/on your platform.

Hope it helps.