Using Shader.SetGlobal fails for setting Stencil States

Unity Engine
1

Found a frustrating bug in Unity recently where I discovered that setting stencil states via Shader.SetGlobalInt() would fail to have any effect. It was strange as some others reported it had worked for them in the past, but then broken.

Looking into the issue again today I happily stumbled upon the cause and got a workaround in the process.

So first thing to remember when using Shader.SetGlobal to change values in a shader, is that the parameter name cannot be a property in the shaders property block. If the parameter is also a property, the property will override or cause the SetGlobalcommand to fail.

More importantly due to Unity caching Shader Properties in the asset ( you can see this in the inspector if you switch to debug mode when viewing a material) it means that if you start with a property in the property block, then remove it to make it a global parameter, setGlobal will still fail as Unity thinks the property still exists!

This caught me out for a quite a while, though the solution is simple, either create a new material or to use an editor script to remove unused properties.

Setting Stencil Ref via SetGlobal is doomed to failure.

So the problem is setting Stencil Ref state via SetGlobal tends to fail in most cases. In fact from testing I discovered that it will always fail unless you happen to update any other shader state at the same time! e.g

SubShader
    { 
        Tags { "RenderType"="Opaque" }
        LOD 200

        Stencil
        {
            Ref [_GlobalDiffuseStencilRef]
            Comp equal
            Pass keep
            Fail keep
        }
...
}

will fail to update the Stencil Ref value via Shader.SetGlobalInt(ā€œ_GlobalDiffuseStencilRefā€, value)

whilst the following and adding Shader.SetGlobalInt(ā€œ_GlobalZWriteā€, value) will work fine.

SubShader
    { 
        Tags { "RenderType"="Opaque" }
        LOD 200
 
        // BugFix:
        // Requires (any?) state setting to ensure that Stencil state updates via Shader.SetGlobal.
        ZWrite [_GlobalZWrite]

        Stencil
        {
            Ref [_GlobalDiffuseStencilRef]
            Comp equal
            Pass keep
            Fail keep
        }

This suggests to me something wrong in Unityā€™s code that for some reason fails to update stencil states unless another unrelated state has/is also updated.

At least this is true for all the tests Iā€™ve made so far and since its a frustrating bug I really hope the workaround here will work across the board.

From testing this bug is present in 4.7.1, 5.4.2, and 2017.1.0b6.

Unsure if the unrelated state has to be ā€˜ZWriteā€™, would be very surprised if it is, so I suspect any other state like cull, colorMask, ZTest etc will work, But I donā€™t think setting anyo other stencil state would work, I think the entire stencil block gets ignored until you change another state.

It may be you donā€™t even need to set the other state, just have it present in the shader, though Iā€™m not sure how that would work.

I tested changing the other state using a property from the shader material block, instead of using Shader.SetGlobal on it, and that also failed, so the key seems to be setting a parameter that is unrelated to stencil states first.

Edit: After further testing Iā€™ve determined that

Using ColorMask as the ā€˜otherā€™ unrelated state does not work. So not all other states will work to force an update of Stencil Ref State. However even better Iā€™ve discovered it doesnā€™t have to be an unrelated state, in fact using any other stencil state will work. e.g.

Stencil
        {
            Ref[_GlobalDiffuseStencilRef]
            Comp[_GlobalDiffuseStencilCompare] // equal
            Pass keep
            Fail keep
        }

This will now work fine, no need for ZWrite global setting. I had assumed that since stencil ref state wasnā€™t being updated that all the stencil states would fail, but that doesnā€™t seem to be the case. It increasingly appears that SetGlobal only fails with stencil ref state and not any other stencil states and that the ref value can be forced to update as long as you also update another stencil state.

Indeed I tested setting Ref in combination of one of Comp, Pass and Fail and in all cases setting at least one of those worked to ensure that the stencil ref was also updated. Yet stencil Ref on its own will always fail. I didnā€™t test other stencil states ( Zfail, ReadMask, WriteMask ).

This is pretty good news as it became apparent having to set another state such as ZWrite was going to be a problem as it interfered with other requirements of various shaders. Knowing now that you only have to set one other stencil state ( or at least one of Comp, Pass, Fail ) is much easier to work with, since if you are setting stencil ref at the global level chances are very high that the other stencil states will all be the same as well.

1 Like
2

Your workaround is indeed still applicable to Unity 2019.2.

1 Like