Trying to create stencil lights but I am hopelessly lost

Unity Engine
,
1

Hi,
I have looked all over the internets for this, but found nothing that could help me get anywhere. Reading the documentation made me realise all there is left for me to do is read up on everything and just try until I have something that works. There are no assets on the asset store, or code available that I could find anywhere either. All I could find that was relevant to this was this post, but it had no code to look at: Wind Waker Style Firefly Lights — polycount

So the question is, how can I make this work with stencils in shaders in Unity 3D? Attached is the current setup and hilited in blue is the overlapping issues I get (among others). I started this based on the Unity Stencil documentation here: Unity - Manual: ShaderLab command: Stencil

The shader code I have is here:

Shader "Test/Light" {
    Properties {
        _Color ("Main Color", Color) = (1,1,1,0)
    }
    SubShader {
        Tags { "RenderType"="Opaque" "Queue"="Geometry+2"}

        ColorMask RGB
        Cull Front
        ZTest Always
        Blend One One
        Stencil {
            Ref 1
            Comp notequal
        }

Pass
        {
            CGPROGRAM
            #pragma vertex vert
            #pragma fragment frag
            // make fog work
            #pragma multi_compile_fog
         
            #include "UnityCG.cginc"

            fixed4 _Color;

            struct appdata
            {
                float4 vertex : POSITION;
            };

            struct v2f
            {
                float4 vertex : SV_POSITION;
            };
         
            v2f vert (appdata v)
            {
                v2f o;
                o.vertex = UnityObjectToClipPos(float4(v.vertex.xyz, 1.0f)); //UnityObjectToClipPos(v.vertex);
                return o;
            }
         
            fixed4 frag () : SV_Target
            {
                return _Color;
            }
            ENDCG
        }
    }
}

And here:

Shader "Test/LightHelp" {
    SubShader {
        Tags { "RenderType"="Opaque" "Queue"="Geometry+1"}
        ColorMask 0
        ZWrite off
        Stencil {
            Ref 1
            Comp always
            Pass replace
        }

        CGINCLUDE
            struct appdata {
                float4 vertex : POSITION;
            };
            struct v2f {
                float4 pos : SV_POSITION;
            };
            v2f vert(appdata v) {
                v2f o;
                o.pos = UnityObjectToClipPos(v.vertex);
                return o;
            }
            half4 frag(v2f i) : SV_Target {
                return half4(1,1,0,1);
            }
        ENDCG

        Pass {
            Cull Front
            ZTest Less
     
            CGPROGRAM
            #pragma vertex vert
            #pragma fragment frag
            ENDCG
        }
        Pass {
            Cull Back
            ZTest Greater
     
            CGPROGRAM
            #pragma vertex vert
            #pragma fragment frag
            ENDCG
        }
    }
}

All I want is for the surface that intersects the mesh to be coloured in a bright colour without the issues you can see in the attached image “show2.PNG” below, but I seem to be unable to get somewhere with this.

Please help. (There is also an asset store idea for you here as mentioned earlier, I’d be willing to pay for it and am sure others would too.)

3831520--323296--show1.PNG
3831520--323296--show1.PNG768×660 80.9 KB

3831520--323299--show2.PNG
3831520--323299--show2.PNG585×671 116 KB

3831520--323302--show3.PNG
3831520--323302--show3.PNG794×419 25.8 KB

1 Like
2

Hey there!

I saw your tweet and, as I said over there, the issue is tied to multiple aspects of how your shaders were set up.
Since the “Light” shader only draws fragments wherever neither of the stencil passes have before, but doesn’t clear the stencil buffer afterwards, what’s happening is the Second Stencil Pass from the light cone further away is writing 1 to the stencil Buffer before the Light pass of the light cone closest to the camera is drawn.
The quick fix to this is to include Pass, Fail and ZFail all set to Zero in the “Light” shader’s Stencil block, this way all the stencil buffer values changed by each light cone should reset right before the next one begins to draw.

I’d also recommend making these kinds of shaders’ RenderType and Queue to “Transparent” and “Transparent-10” respectively and have them not write to the Zbuffer (Zwrite Off) so that they don’t get in the way of other Geometry-typed shaders and also don’t mess with the depth information so that other visuals such as transparent particles behave as expected and always draw on top instead of inconsistently flickering due to sorting issues.

I’ve expanded a little bit on the idea, making a quick single-material, 2-pass Stencil Light shader that allows additive light volume overlapping so that it essentially behaves like normal lights minus falloff (that requires some more in-depth Zbuffer handling, pun intended).

Check these out:
IllinformedUnawareBalloonfish
MagnificentCoolArabianhorse

Shader "Custom/StencilLight"
{
    Properties
    {
        _MainTex("Texture", 2D) = "white" {}
        _Color("Color",Color) = (1,1,1,0.5)

        _StencilRef("Stencil Reference Value", Float) = 20
    }
    SubShader
    {
        Tags { "RenderType" = "Transparent" "Queue" = "Transparent-10"}
        LOD 100
        Pass
        {
            //This pass only draws under opaque objects
            Ztest Greater
            //Does not affect the zbuffer, we will still need surface depth information
            //not only for this shader but for the other transparent stuff that might get drawn after
            Zwrite off
            //Draw the insides of the primitive
            Cull Front
            //Do not output any color information
            Colormask 0
            Name "Stencil Greater"
            //As long as the Ztest passes, 20 shall be written to the stencil buffer
            Stencil
            {
                comp always
                ref [_StencilRef]
                pass replace
            }
        }
        /*
        This pass is now redundant
        Pass
        {
            Zwrite off
            Ztest lequal
            Cull Back
            Colormask 0
            Name "Stencil LEqual"
            Stencil
            {
                comp equal
                ref 20
                pass keep
                fail zero
                zfail zero
            }
        }*/

        Pass
        {
            Name "Color"
            //Once more we don't want to touch the zbuffer
            Zwrite off
            //Standard Ztest and backface culling, left these here just for clarity
            Ztest Lequal
            Cull Back
            //Additive, but taking alpha output into account for blending.
            Blend SrcAlpha One

            //Will only draw if intersecting with the stencil value from the previous pass
            //Even if the Ztest fails, it will still clear the stencil value
            //so that any following lights also render correctly without reading previous lights' stencil values.
            Stencil
            {
                comp equal
                ref [_StencilRef]
                pass zero
                fail zero
                zfail zero
            }

            CGPROGRAM
            #pragma vertex vert
            #pragma fragment frag

            #include "UnityCG.cginc"

            struct appdata
            {
                float4 vertex : POSITION;
                float2 uv : TEXCOORD0;
            };

            struct v2f
            {
                float2 uv : TEXCOORD0;
                float4 vertex : SV_POSITION;
            };

            sampler2D _MainTex;
            float4 _MainTex_ST;
            fixed4 _Color;
            v2f vert(appdata v)
            {
                v2f o;
                o.vertex = UnityObjectToClipPos(v.vertex);
                o.uv = TRANSFORM_TEX(v.uv, _MainTex);
                return o;
            }

            fixed4 frag(v2f i) : SV_Target
            {
                fixed4 col = tex2D(_MainTex, i.uv) *_Color;
                return col;
            }
            ENDCG
        }
    }
}

Be sure to notice multi-pass shader drawcalls are not dynamically batched, and at least for this approach, that is good, because after each stencil-only pass, a “Color” pass is needed in order to clear the Stencil Buffer before the next light volume is drawn so that no weird stencil overlaps happen. This amounts to 2 drawcalls for each light volume, which can get a bit expensive so use with care.

1291×643 87 KB

I hope this helps you on your journey to better, less glitchy streetlights. Have fun!

1 Like
3

Thank you so much for this! I will try it out once I get home from work today. :3

I am fullly aware of the drawcalls needed, but I thought maybe I could take a CombineMesh on the streetlight volumes per “block” or per street, and that could help get the drawcalls down. I am not that concerned about that, more about getting it to work for the intended effect, and your explanation is great for this. I think maybe I can get the falloff working because I have an idea how that might work (similar to what I did for the windows, except it takes distance from object center into consideration instead of cameraposition in relation to object world position).

1 Like
4

Oh by the way, what would it take to make the intersection only valid for the first object surface facing towards the center of the light? In its current configuration the intersectional light also works for faces that face away from it, and that also are behind other faces. I think that doing a normal-check might help solve it, but I am not sure exactly how.

1 Like
5

To answer to the first message, while combining all meshes would reduce drawcalls, it’d mean a single pass would write the stencil for many light volumes at the same time, that would cause the Color pass to maybe sometimes draw one light volume over another’s stencil mask, creating artifacts and plain wrong intersections.

Making the lights behave more like normal lights would require you to sample both the depth and normals texture each frame in order to obtain the surface information, since the part that is rendered in the Color pass is nothing like the surface it seems to be projected onto. That’d be much closer to the usual Deferred Lighting approach since that’s the main purpose of it’s G-buffer.

On top of that, making the lights stop at the first object they hit would probably require depth textures, or a more complex stencil system paired with vertex extrusion techniques applied to at least the dynamic objects involved.

If you want the lights to only intersect with static, simpler geometry, maybe you could get away with raycasting each extruded vertex against, say, the ground mesh, then add a little offset to make the light volume mesh actually intersect the surface, in Editor time.

6

Well, this maybe is a little more complex than I thought, then… ;_;

But also, looking at the blending that happens here it looks like the blending the lights currently have is completely different: https://agilethief.artstation.com/projects/OBBXy?album_id=70137

What is the secret to this blending? This is exactly what I would like to have. Is it to render everything in the shade and then “cut out” from that shaded area?

7

Hello and sorry for the delayed reply.

The effect shown in that example is achieved by using Additive blending operations during the Color Pass(which is the same thing my shader does), apparently on top of standard lighting, which is also rendered additively. Notice the smooth lighting on the character, which doesn’t seem to be affected by the torch, presumably because the character mesh is also writing to the stencil buffer in order to avoid being lit by the stencil lights. Either that or the character’s draw calls are queued after all opaque environment and stencil lights are rendered.

In a Deferred Lighting scenario, the surface’s Glossiness, Albedo, Normal and AO information could also be read to further modify how the lit fragments behave.

I went ahead and set up a simple scene using the same shader I posted a few days back.
SolidShoddyArkshell
In order to achieve such a noticeable effect you should avoid reaching extreme lighting values and create contrast by supplying lower ambient lighting values. Having the scenario meshes be lit by the standard light is optional and entirely an aesthetic choice.

In this scene I completely turned off sunlight, thus making the default procedural sky darken and “naturally” lowering the real time ambient light in the scene and giving it a rather blueish tint that makes the warmth of the fire even more noticeable. On top of that, I could’ve used different meshes instead of Unity’s default UV Sphere to achieve the polygonal look in agilethief’s example.

8

I tried turning off the sunlight, using #414141 for the light colour, switching to deferred lighting, and using 0.5 ambient skylight, and this is what it ended up looking like against objects using the Standard shader.

It does not look like in your example.

Am I not understanding, are extra lights needed (as in the campfire) to supplement this, or does this not work in WebGL?

3864358--327598--tyst1.PNG
3864358--327598--tyst1.PNG868×609 40.6 KB

9 
Shader "Custom/StencilLight"
{
    Properties
    {
        _Color("Color",Color) = (1,1,1,0.5)
        _Intensity("Intensity", Float) = 1

        _StencilRef("Stencil Reference Value", Float) = 20
    }

    SubShader
    {
        Tags
        {
            "RenderType" = "Transparent"
            "Queue" = "Transparent"
        }

        LOD 100

        Pass
        {
            Name "Stencil Greater"
            Ztest Greater
            Zwrite off
            Cull Off
            Colormask 0
            Lighting Off

            Stencil
            {
                comp always
                ref[_StencilRef]
                pass replace
            }
        }

        Pass
        {
            Name "Color"
            Zwrite off
            Ztest Lequal
            Cull Back
            Lighting Off
            Blend SrcAlpha One

            Stencil
            {
                comp equal
                ref[_StencilRef]
                pass zero
                fail zero
                zfail zero
            }

            CGPROGRAM
            #pragma vertex vert
            #pragma fragment frag

            fixed _Intensity;

            struct appdata
            {
                float4 vertex : POSITION;
            };

            struct v2f
            {
                float4 vertex : SV_POSITION;
            };

            fixed4 _Color;
            v2f vert(appdata v)
            {
                v2f o;
                o.vertex = UnityObjectToClipPos(v.vertex);
                return o;
            }

            fixed4 frag(v2f i) : SV_Target
            {
                return _Color * _Intensity;
            }

            ENDCG
        }

        Pass
        {
            Name "Color 2"
            ZTest off
            ZWrite on
            Cull Front
            Lighting Off
            Blend SrcAlpha One
            BlendOp Add

            Stencil
            {
                Ref[_StencilRef]
                Comp equal
                Pass zero
            }

            CGPROGRAM
            #pragma vertex vert
            #pragma fragment frag

            fixed _Intensity;

            struct appdata
            {
                float4 vertex : POSITION;
            };

            struct v2f
            {
                float4 vertex : SV_POSITION;
            };

            fixed4 _Color;
            v2f vert(appdata v)
            {
                v2f o;
                o.vertex = UnityObjectToClipPos(v.vertex);
                return o;
            }

            fixed4 frag(v2f i) : SV_Target
            {
                return _Color * _Intensity;
            }

            ENDCG
        }
    }
}

Here is adding a third pass so that I can go inside the 3D object without having it disappear on me, however I would like to get it down to just two passes if possible.

1 Like
10

Hey how do I turn this into a URP shader? Thanks

4 Likes