I have made a stencil shader mask and object. The stencil object behind the stencil mask can only be seen when looking through the stencil mask. (code for the stencil object and mask below)
Does anyone know how I can make a stencil mask for my current stencil mask?
Like I have said more often on the forum here. Don’t put your reference to 1 and your masks to 255. If you want to only use one bit, set your masks to 1 too.
It’s quite possible to mask the mask by using a second bit in the stencil buffer.
Step 1: (Write to the first bit)
Ref 1
Comp always
Pass replace
WriteMask 1
Step 2: (If the first bit is set, write to the second bit)
Ref 3
Comp equals
Pass replace
ReadMask 1
WriteMask 2
Step 3: (If the second bit is set, render)
Ref 2
Comp equals
ReadMask 2
Another way would be to just write to bits 1 and 2 in the first two steps and directly compare with both in the third step.
I am really new to shader, there for sorry for the stupid questions that now follows:
“How should I implement this shader? And does this work for masks in masks in masks?”
As far as I understand, I need do the following three steps:
Step 1 - Create a shader file for a stencil object
Step 2 - Implement my code for a stencil object, but change the (read/write) masks to 1?
Step 3 - Create a shader file for a stencil mask
Step 4 - Implement your code for a shader mask (that can also see read other shader masks) within the “Stencil{…}” of my code?
Or do I need to create a new shader file specificly for a shader mask mask?
Also, where would I use my “Queue” = “Geometry-100”?
The way those shaders are set up, you don’t need any extra shader. It can be controlled from the settings in the material. My first 2 steps can be done with the first shader and the last step with the second shader.
The render order is important, but you can control that through the material too. (The queue in the shader is just the default value for the material.)
I like your way of explaining, but I am not sure where I made the mistake in the code, which applies (as far as I understood) your feedback.
Step 1 and 2 are added as subshaders, to the “mask” shader.
Step 3 is for the “object/non-mask” shader.
But it seems something isn’t going as I planned, now all are invisible.
A short overview of the changes are demonstrated in the code below:
Stencil mask
Shader "Custom/Stencil/UI/Default-Mask"
{
// Subshader to define the base shader for the object
SubShader
{
Tags
{
"Queue" = "Geometry-100"
"IgnoreProjector" = "True"
"RenderType" = "Transparent"
"PreviewType" = "Plane"
"CanUseSpriteAtlas" = "True"
}
//Step 1: (Write to the first bit)
Stencil
{
Ref 1
Comp always
Pass replace
ReadMask 1
WriteMask 1
}
} // End subshader stencil for masking mask
// Subshader to define the base shader for the object
SubShader
{
Tags
{
"Queue" = "Geometry-100"
"IgnoreProjector" = "True"
"RenderType" = "Transparent"
"PreviewType" = "Plane"
"CanUseSpriteAtlas" = "True"
}
//Step 2: (If the first bit is set, write to the second bit)
Stencil
{
Ref 3
Comp equal
Pass replace
ReadMask 1
WriteMask 2
}
} // End subshader stencil for masking mask
}
Stencil object
Shader "Custom/Stencil/UI/Default-Object"
{
SubShader
{
Tags {
"Queue" = "Transparent"
"IgnoreProjector" = "True"
"RenderType" = "Transparent"
"PreviewType" = "Plane"
"CanUseSpriteAtlas" = "True"
}
//Step 3: (If the second bit is set, render)
Stencil {
Ref 2
Comp equal
ReadMask 2
}
} // End subshader
}
I really wan’t to make this work.
It may be better of sending the whole code, so anyone can have a closer look.
There for, I have put a test project and have added it to this reply.
The current code works for Meshes and UI, it is a little more advance, but works as told above.
Also here is a image that helps explaining the situation:
Don’t know if you’re still working on this, but you’re setting the wrong ref values. The ref value is the one used for the comparison, so now you’re setting it to 1, then checking if it’s equal to 3, then checking if it’s equal to 2. Of course they are all invisible, the comparison is always false. Try using Ref 1 for all of them, for example.
Also, it seems you can do two stencils in the same pass by doing two separate Stencil { }.
Big thanks to jvo3dc for the explanation, it was very helpful.
Yes, that’s very much the case. I think it works because the bitwise operations just happen to work out (those indices were 100 and 101, respectively). However, that means it may interfere with other stencil masks, so even if it works I scrapped that approach, and ended up using a single pass with Pass IncrSat. That way I can feed the increased value to the second masked object.
Maybe other people needing to mask a mask can use the same approach.
So, if the first bit is set, then set the second bit. So this mask is writing in the second bit, but it itself is masked by the first bit.
Ref is 3 in this case, but could be anything with the last two bits sets. So 7, 15 or 255 would do exactly the same. It’s the masks that really control it here.
Hey, so I’m trying to do something similar with nested masks, but in my case I have multiple potential masks inside my outer mask that each could be showing a different underlying image/scene. Trying to understand what’s being suggested above and whether that could apply to this situation? Or perhaps this is the same exact situation and I’m not realizing it?
Just for completeness: You don’t need to hardcode the stencil in the shader. You can pass them as properties as shown here: Stencil · supyrb/ConfigurableShaders Wiki · GitHub
This makes it way easier to iterate and experiment with stencil shaders
Hey noethis, don’t know if you’re still interested, but I ended up figuring out a setup with the new Sprite Mask component. If you play with the custom range you can selectively mask only certain layers or sortingOrder intervals in the same layer. You could use an outer stencil that applies to everything, and inner stencils that only mask a given sortingOrder. I found it’s way easier to maintain than using multiple shaders and materials.
I don’t know how Unity handles it behind the scenes. You don’t need any custom shader code nor to modify the stencil buffer yourself, you just set it in the editor component and it works with the default sprite shader.
Oh, but I’m looking into using it with tilemap, which does not work with the editor component as it’s meant for the sprite renderer.
So I can’t use the editor component.
