how to disable the Stencil block via shader properties?

Unity Engine
1

I’m trying to write a shader for making holes (masking). Here’s the bare-bone minimal shader I came up with:

Shader "Test/stencil read"
{
    Properties
    {
        _StencilMask ("Stencil Mask", Int) = 1
    }

    SubShader
    {
        Tags { "RenderType"="Opaque" }
   
        Pass
        {
            Stencil
            {
                Ref [_StencilMask]
                ReadMask [_StencilMask]
                //WriteMask [_StencilMask]
                Comp NotEqual
                Pass Keep
            }
        }
    }
}

So by default this will only draw pixels if there’s nothing in the mask for the given layer.

Now the thing is, I’d like it to be an uber-shader so that for any material we can just set a property, and it becomes masked by objects of a given layer. So I can make the comparison function a property as well:

    Properties
    {
        [Enum(CompareFunction)] _StencilComp ("Stencil", Int) = 0
...
            Stencil
            {
                Comp [_StencilComp]
...

so that we either check against the mask with NotEqual, or we bypass it with Disabled (see Unity - Scripting API: Rendering.CompareFunction.Disabled).
Problem is, contrarily to what the docs say, either way the stencil is used, I can see plainly in GPA:

IDIRECT3DDEVICE9_SETRENDERSTATE( D3DRS_STENCILENABLE, True )
...
IDIRECT3DDEVICE9_SETRENDERSTATE( D3DRS_STENCILFUNC, D3DCMP_ALWAYS )

So my question is, is there a way to disable the Stencil block entirely via properties/keywords? Just want to make sure there are no hidden cost when I don’t want masking.
As far as I know, there’s no way to #define state blocks using keywords…?
I also tried stuff like

            Stencil
            {
                Mode Off
...
            }

like for the Fog… no luck, parse error :frowning: !

2325776–156970–test stencil read.shader (407 Bytes)
2325776–156971–test stencil write.shader (405 Bytes)

2

Just set everything to the default values. So mainly:

Comp Always
Pass Keep

This is the most you can do and should not have any hidden costs. If Comp is set to Always, there is no stencil read. If Pass is set to Keep, there is no stencil write.

1 Like
3

Okay, so what you’re saying is Unity is setting states anyway, but if the values are right the hardware will just do nothing?

4

There is an actual on and off, but not in Unity it seems. You can almost assume that the driver will disable it for you anyway, but still…

5

Yeah that’s a bit frustrating. Thanks for the answer!

6

I believe you want to set [_StencilComp] to CompareFunction.Disabled to disable the Stencil Op completely.

7

See my initial post, either way the stencil state is visible in GPA:

… though I don’t know if it has any cost; fingers crossed.

8

founded that you can actually do it.

    Properties {
        _Stencil ("Stencil ID", int) = 4
        _StencilComp ("StencilComp", Int) = 3 //3 = equal
    }
    SubShader {
        Stencil {
            Ref [_Stencil]
            Comp [_StencilComp]
            Pass keep
            Fail keep
        }

and then at runtime use ‘always’ (8 is ‘always’ in the enum)

 material.SetInt ("_StencilComp", 8);

and ‘disable’ is 0

1 Like
9

Was a consensus found on the proper way to disable stencil operations via script?

10

I’d like to also know this. We use this on a lot of objects and I would love to know I can do this without performance impact.

Edit: I went with jvo3dc’s suggestion. At least functionally it appears to work correctly. For performance, I just can’t be a 100% sure.

11

Properties:

//stencil ref
_StencilRef ("StencilRef", int) = 1 
     
//0:smile:isable 1:Never 2:Less 3:Equal 4:LessEqual 5:Greater 6:NotEqual 7:GreaterEqual 8:Always
[Enum(UnityEngine.Rendering.CompareFunction)] _StencilComp ("StencilComp", Int) = 8
     
//0:Keep 1:Zero 2:Replace 3:IncrSat 4:smile:ecrSat 5:Invert 6:IncrWrap 7:smile:ecrWrap
[Enum(UnityEngine.Rendering.StencilOp)] _StencilOp ("StencilOp", int) = 2

SubShader:

Stencil{
     Ref [_StencilRef]
     Comp [_StencilComp]
     Pass [_StencilOp]
     ...
}
12

Reviving this thread, wondering if anybody knows for a fact whether there’s a performance impact from having StencilComp=Disabled vs not having the code at all in the shader?

I just did a small profiling test myself and there seems to be up to a .5ms difference (CPU renderering time) in the scene I was testing but I’m not 100% sure if that’s it or not.

1 Like
13

Once you define a “Stencil” info in the file, even if it contains only the default values:

        Stencil
        {
            Ref 0
            CompBack Always
            PassBack Keep

            CompFront Always
            PassFront Keep
        }

it is slower :frowning:

1 Like
14

Works like a charm. Many thanks!
Further references on the Stencil Comp and Stencil Opetaions, for those who want to control different things than simply enabling / disabling: https://discussions.unity.com/t/602226

15

Unity 2022.3 (DX11), test using the UI/Default material in UGUI: Set the Stencil Comp to 0 (disabled). In the FrameDebugger, it can be seen that the Stencil Comp is Disabled. However, when capturing frames through RenderDoc, it is found that the StencilEnable in the state set by OMSetDepthStencilState is True.

16

I tested another simple Unlit shader that supports Stencil for rendering a cube.
Set the stencil comp to 0 and I saw StencilEnable true inside RenderDoc as well.