Trying to get a mask effect with Stencil Buffer

Unity Engine
1

Hi all!

I’m just getting my head around the whole stencil buffer thing because it looks like it could be really handy!
I’m really close to getting this effect working but now I’m not really sure if it’s possible. I was hoping to mask the first object (purple) only if it is behind the mask (quad). I want everything else in the scene to remain unaffected.

You can see the quad is fighting it out with background objects here and I’m not sure how to get around that.
Any ideas?

Thanks.
Pete

2

Hi,

Can we see your shaders with the stencil please?
The grey cube is a standard shader?

3

Yeah the grey one is a standard shader with no stencil settings but damn! I broke the original scene and can’t even get the shaders to do that again :frowning:
This whole stencil thing looks really interesting but it’s like a crazy puzzle to work out. It would be ace if there was a nice tute on it or something because you could do some really interesting stuff with it.

4

I think the thing you have in the scene view is more like a bug. You will have the same effect with this shader :

Shader "Custom/Test" {
    Properties {
        _Color ("Color", Color) = (1,1,1,1)
    }
    SubShader {
        Tags { "RenderType"="Opaque" "Queue"="Geometry"}
        LOD 200

        ZWrite On
        ColorMask 0

        Pass{}
    }
    FallBack "Diffuse"
}

In scene view :

The way stencil works, your front top corner should be masked.

But maybe I’m wrong, I’ve seen topics where " bgolus " answered about stencil, he could answer this I think.

1 Like
5

Hey thanks man!

1 Like
6

3343629--261186--icm57-8o4fs.gif
Did you mean something alike?

That’s done with two shaders:

Red material shader:

...........................................
        _StencilMask("Stencil Mask", Range(0, 255)) = 1
    }
    SubShader {
        Tags { "RenderType"="Opaque" }
        LOD 200
       
        Stencil{
            Ref [_StencilMask]
            Comp NotEqual
            Pass Zero

        }
...........................................

Quad material shader code:

    Shader "Custom/Test" {
        Properties {
            _Color ("Color", Color) = (1,1,1,1)
            _StencilMask("Stencil Mask", Range(0, 255)) = 1
        }
        SubShader {
            Tags { "RenderType"="Opaque" "Queue"="Geometry-1"}
            LOD 200

            //ColorMask 0
             Stencil{
            Ref [_StencilMask]
            Comp Never
            Fail Replace

        }
            Pass{}
        }
       
    }
2 Likes
7

It’s all about the rendering order. If the stencil plane has ZWrite On, and it renders first, then the objects that render afterward and behind it won’t get rendered, even without using stencils.

In @Balikk 's example you’re seeing what happens when the objects get sorted differently as the camera moves around. That first gif is almost entirely just the effects of the depth buffer. The second gif is more what I would expect from the stencil taking effect.

@tomekkie2 's example is the stencil working, just like @Balikk 's second gif.

If by “behind” you actually want that intersection between the plane and the purple box like in your original gif and @Balikk 's first gif, then you just need to render the scene first, then the plane, then the purple box, which you can do by setting the queue and no stencil at all. However if you need it to work with, say, transparent effects that are in the scene then you might have some problems as there’s only that one depth buffer and there’s now a big solid quad floating invisibly in the scene that’ll block effects behind it just as well as that purple box. Stencil has the benefit of only affecting the things you want it to, but you can’t really do that intersection effect with those as easily and you get @tomekkie2 's look. My suggestion for if you want both is to not using depth buffers or stencil, but use something like the cross section shaders or potentially a render texture in which you render out the mask in and then the shader on the purple box reads from and that drives the alpha for a cutout / alpha test.

2 Likes
8

Hey thanks @bgolus
Sorry that was a pretty bad gif I originally posted, it does show my amount of knowledge when it comes to shaders though :slight_smile:

Here’s what I have so far,

This is working pretty cool really but I was thinking it would be nice if could have one of the teeth potentially poking out of the mouth when it closes.

I think in order to get that I’d have to mask the outside area and use a Zdepth mask to allow the tooth to come in front…

Not too sure, but it’s moving along, thanks for all the help, I had no idea about cross section shaders, they look super handy.

Pete

6 Likes