[RESOLVED] Trying to make holes in walls with stencil buffer shaders

Hey,

I’m developing a project where users can build a room and be able to put a hole on a wall.

I’m using stencil buffer shaders to do it :

  • one used for the plane making a hole in the wall :
Shader "Custom/Stencil/Mask OneZLess"
{
    SubShader
    {
        Tags { "RenderType"="Opaque" "Queue"="Geometry-1" }
        ColorMask 0
        ZWrite off
       
        Stencil
        {
            Ref 1
            Comp always
            Pass replace
        }
       
        Pass
        {
            Cull Back
            ZTest Less
       
            CGPROGRAM
            #pragma vertex vert
            #pragma fragment frag
           
            struct appdata
            {
                float4 vertex : POSITION;
            };
            struct v2f
            {
                float4 pos : SV_POSITION;
            };
            v2f vert(appdata v)
            {
                v2f o;
                o.pos = mul(UNITY_MATRIX_MVP, v.vertex);
                return o;
            }
            half4 frag(v2f i) : COLOR
            {
                return half4(1,1,0,1);
            }
           
            ENDCG
        }
    }
}
  • one for the wall that needs to be cut by the plane :
Shader "Custom/Stencil/Diffuse NotEqualOne"
{

Properties
{
    _Color ("Main Color", Color) = (1,1,1,1)
    _MainTex ("Base (RGB)", 2D) = "white" {}
}

SubShader
{
    Tags { "RenderType"="Opaque" "Queue"="Geometry" }
    LOD 200

    Stencil
    {
        Ref 1
        Comp notequal
        Pass keep
    }

    CGPROGRAM
    #pragma surface surf Lambert

    sampler2D _MainTex;
    fixed4 _Color;

    struct Input
    {
        float2 uv_MainTex;
    };

    void surf (Input IN, inout SurfaceOutput o)
    {
        fixed4 c = tex2D(_MainTex, IN.uv_MainTex) * _Color;
        o.Albedo = c.rgb;
        o.Alpha = c.a;
    }
   
    ENDCG
}

Fallback "VertexLit"
}

The walls are like this :
2778620--200994--room.png

  • The red wall uses the wall shader with Ref = 1
  • The 2 blue planes use the mask shader with Ref = 1
  • The red green uses the wall shader with Ref = 2
  • The 2 yellow planes use the mask shader with Ref = 2

In game it give me this :


Here, no problem, I can see through both holes.

2778620--201016--Capture d’écran 2016-09-06 à 14.21.21.png
Same thing if I look from the other side, no problem.

The problem is when I’m looking through one hole and there’s another behind it. In this case the second hole doesn’t render properly, as you can see :
2778620--201020--Capture d’écran 2016-09-06 à 14.23.17.png

Has anyone ever tried to do something like this ?

It looks like you’re rendering in this order: mask1, mask2, wall1, wall2
try rendering in this order: mask1, wall1, mask2, wall2

You can do this without making additional shaders; in the upper-right of your inspector, there’s a dropdown where you can set the inspector to “debug mode”. Do this, and you will be able to manually set the “Render Queue” of selected materials to force materials to draw in a certain order. Any material with a render queue of “-1” will use the “Queue” assigned in it’s shader.

Side note
It looks like there’s some shadow acne on your white wall. Try changing the shadow near/far planes in the quality settings or change the “bias” and “normal bias” settings on your light that casts the shadow to fix it

1 Like

Thanks for your answer, you put me on the right track !

The last thing I had to do was setting the render queues of my objects at runtime.

So now we have :

  • Red wall : stencil = 1, render queue = 4000
  • Blue planes : stencil = 1, render queue = 3999
  • Green wall : stencil = 2, render queue = 3998
  • Yellow planes : stencil = 2, render queue 3997

I also had to write a custom standard shader using stencil buffer for my walls, so I could put a normal map on their material.

With this I can now see through multiple holes. I can even properly see through a window behind a hole.

Here’s what it looks like :
Pictures

3 Likes

good job, it looks great! Those door frames really hide the stencil imperfections

Sorry for this late comment, but your wall recive shadows?

Shadows still work, but you can’t stencil away the portion that casts or receives the shadow. That means the empty door frame in this instance will cast a shadow as if it was a solid wall, and any shadow cast on the door frame will appear as a shadow floating in mid-air.

Does anyone have a standard shader version of the 2nd shader?

1 Like

Could you please share your shader ?? I am finding a solution same with this.

Thanks!