Stencil Op/Comparison Values

Web Resources
1

The ShaderLab stencil documentation lists what the available stencil operations and comparison functions are, but does not mention that these correspond to various integer values (the UI shaders use Float variables that appear to get set by the masks/etc at runtime to take advantage of this). It would be nice if these were listed in the docs along with the values. The Stencil Operations list seems to match up, but the Comparison Functions does not. I couldn’t find a list of the latter anywhere online, so I messed about for a bit and came up with one (hopefully it doesn’t turn out this is device-specific or something, though I don’t imagine the UI shaders working too well if that were the case):

Comparison Functions

  • 0 - Always (?)
  • 1 - Never
  • 2 - Less
  • 3 - Equal
  • 4 - LEqual
  • 5 - Greater
  • 6 - NotEqual
  • 7 - GEqual
  • 8 - Always (? This is the default for the UI shaders so I suspect this one is technically the ‘correct’ Always, but any value beyond it will also count as Always)

Stencil Operations (these seem to line up with the order they’re shown in the docs):

  • 0 - Keep (?)
  • 1 - Zero
  • 2 - Replace
  • 3 - IncrSat
  • 4 - DecrSat
  • 5 - Invert
  • 6 - IncrWrap
  • 7 - DecrWrap
10 Likes
2

Thank you so much! This should really be somewhere easy to find in the docs… maybe even the shaderlab page.

3

Good spot - thanks for reporting it and doing the research. I’ve added it to our list.

4

Yeah they basically correspond to Unity - Scripting API: StencilOp and Unity - Scripting API: CompareFunction enum values. I’ll see about updating the docs.

3 Likes
5

I’m sorry to bump such an old thread, but I wanted to point out 2 years later that, as of Unity 2018, the documented order of the comparison function enumeration still does not match up with these values in the ShaderLab manual… it gave me a headache for this past half hour. Would you please add it to the documentation to save others the trouble of relying on user support for reference that should be part of the product?

10 Likes
6

Where the update?:frowning:

7

still not updated

2 Likes
8

On Switch and possibly other platforms, a comparison function value of 0 appears to stop the material from rendering, while 8 functions as expected. This stumped me for a while. These values should really be in the docs, along with an explanation on what the difference between 8 and 0 is. Additionally it would be nice for them to just be a dropdown in editor instead of an int

1 Like
9

somewhat related, the Enums do not function as labeled & can be reproduced using URP in 2019.3.3f1 & validating the depth stencil state with RenderDoc (though it is not a URP issue)

In URP RenderObjects.cs line 28, the depthCompareFunction is defined as CompareFunction.LessEqual

When taking a RenderDoc capture, checking in the API Inspector it can be seen that ID3D11DeviceContext::OMSetDepthStencilState event has the depth state being set as GreaterEqual

Other elements in the Enum have similar unexpected behavior, I logged a bug on it recently & hopefully it will be fixed.

1 Like
10

Comparison Functions

  • 0 - Disabled
  • 1 - Never
  • 2 - Less
  • 3 - Equal
  • 4 - LEqual
  • 5 - Greater
  • 6 - NotEqual
  • 7 - GEqual
  • 8 - Always
3 Likes
11

6017312--648719--upload_2020-6-24_18-59-47.png
6017312--648719--upload_2020-6-24_18-59-47.png1009×642 43.8 KB

12

it is sad that after after 5 years the docs still don’t show the numeric values

2 Likes
13

You can find these values in the ShaderLab documentation: Unity - Manual: ShaderLab command: Stencil

2 Likes