Stencil shader ZTest for a specific layer only?

Unity Engine
1

Hello everyone :slight_smile:
i want to make a stencil shader to cutout a sphere from a cylinder. The part of the cylinder behind(coverd by) the sphere should be invisible. Only the front arc should be visible. So far the setup is no problem. However i also have other objects in the scene that are opaque and should not be covered by the stencil. Also the cutout part needs to be rendered on top all object.
So what i tried:
Cull Shader (Rendered on layer 2001 therefore ZWrite on won’t block the other objects):

Shader "Custom/Cull"
{
    SubShader
    {
        ColorMask 0
        Stencil{
            Ref 1
            Pass replace
        }
        ZWrite on
        ZTest always
        Pass { }
    }
}

Top shader (Rendered on layer 2002):

Shader "Custom/Top"
{
    Properties {
        _Color("Color", Color) = (1,1,1,1)
    }
    SubShader
    {
        Color[_Color]
        Stencil{
            Ref 1
            //Only render if pass the stencil buffer test i.e. equal or greater
            Comp gequal
        }
        Pass  { }
    }
}

I attached a screenshot of the result. Unfortunately if the ring is covered by a scene object it won’t be rendered on top which makes sence, because the Top shader only renderes everything inside the stencil mask and outside only if the ztest succed. So i need some kind of ztest that only tests outside the mask. Else the covered part of the cylinder will be visible, too. I thought about using a second rendercamera or multiple materials but that sound a bit cumbersome. Any idea on how to get this result? Thank you for reading this :slight_smile:

4177141--369340--screenshot.PNG

2

So, funny thing … your stencil isn’t doing anything here. The effect you have so far is 100% from the ZWrite of the first pass. In fact your stencil settings are such that they have no effect at all on the second shader. You’re writing to depth and stencil ref 1 with the first, and the second is testing if ref 1 is greater or equal to the value in the stencil. The default value is 0, and the previous pass is setting 1, so unless you have other things in the scene writing to the stencil, it’ll always pass the comparison. So you could strip out all of that and it’ll still work just as well as it already is.

So, solutions. There are two ways I can think of off the top of my head that don’t use render textures.

Option 1: Z punch through
This requires 3 shaders at different queues to work. The first one is the cylinder itself, with a shader with ColorMask 0, ZWrite On, and ZTest Always, just like your sphere pass, but which uses a maxed out z depth.

o.pos = UnityObjectToClipPos(v.vertex);
o.pos.z = 0.000001; // assumes you’re using Direct3D

The other two passes are what you already have, with out the stencil stuff, and bumped up in their queues so they render after this first one. Done. That first shader effectively clears the depth buffer so that nothing in the scene will occlude it.

Option 2: Stencil
Render your sphere as you currently are, but change your cylinder to be a two pass shader. The first pass is identical to what you have, but with ZTest Always, and Stencil Ref 0 Comp Equal (move this all inside the Pass {}). The second pass uses ZTest LEqual (the default), and Stencil Ref 1 Comp Equal. This will mean the first pass of your second shader will render everywhere the sphere is not. The second pass will only render where the sphere was. Together it’ll appear as one ring, but rendered as two separate passes.

This only really works because your object is a solid color. If it was textured, the lack of ZWrite for the “outside” area may cause some issues with how the ring gets rendered.

I’d also recommend using queues above 3001. If you really want it to render on top, otherwise alpha tested (2450) and transparent (3000) queues will render over this.