Particle alpha blending shader (2D)

Unity Engine
,
1

Hello, I have a problem with my shader on a Particle System.

I am trying to create a smoke effect in my game using a Particle System.
However the particles were semi-transparent creating a messy unwanted effect when overlapping.

I wanted the particles to make one homogeneous smoke mass without additive blending.
I found two sources dealing with this issue and both solved this problem.
https://mispy.me/unity-alpha-blending-overlap/
https://gamedev.stackexchange.com/questions/158128/how-to-write-a-transparent-shader-for-a-sprite-that-ignores-transparent-sprites

I learned about Stencil buffers and both shaders in those sources made them look correct.

Now the issue arises when the particles are not on the same sorting layer and even then, there is some odd tomfoolery afoot.

This is what happens when the two Particle Systems overlap.

The distant smoke pillar on the sorting layer behind the first one culls the one in front, but only the “light” part of the distant smoke does it.

Now when i put the smaller distant smoke Two layers behind the bigger one.

That’s when the entirety of the smoke becomes a mask of some sort for the other pillar.

This is the two smoke pillars on the same sorting layer with the smaller smoke pillar in front of the bigger one.

Despite being behind the bigger one, its rendered on top.

I only learned about Stencil buffers and Custom Shader Creation yesterday so don’t really understand the entirety of those shaders.

This is the shader that I use:

Shader "Unlit/AlphaBlend"
{
    Properties
    {
        _MainTex("Sprite Texture", 2D) = "white" {}
        _Color("Tint", Color) = (1,1,1,1)           

        [Enum(UnityEngine.Rendering.CompareFunction)] _StencilComp("Stencil Comparison", Int) = 3
        _Stencil("Stencil ID", Float) = 0
        [Enum(UnityEngine.Rendering.StencilOp)] _StencilOp("Stencil Operation", Int) = 3        _StencilWriteMask("Stencil Write Mask", Float) = 255
        _StencilReadMask("Stencil Read Mask", Float) = 255

        _ColorMask("Color Mask", Float) = 15

        [Toggle(UNITY_UI_ALPHACLIP)] _UseUIAlphaClip("Use Alpha Clip", Float) = 0
    }

        SubShader
        {
            Tags
            {
                "Queue" = "Transparent"
                "IgnoreProjector" = "True"
                "RenderType" = "Transparent"
                "PreviewType" = "Plane"
                "CanUseSpriteAtlas" = "True"
            }

            Stencil
            {
                Ref[_Stencil]
                Comp[_StencilComp]
                Pass[_StencilOp]
                ReadMask[_StencilReadMask]
                WriteMask[_StencilWriteMask]
            }

            Cull Off
            Lighting Off
            ZWrite Off
            ZTest[unity_GUIZTestMode]
            Blend SrcAlpha OneMinusSrcAlpha
            ColorMask[_ColorMask]

            Pass
            {
                Name "Default"
            CGPROGRAM
                #pragma vertex vert
                #pragma fragment frag
                #pragma target 2.0

                #include "UnityCG.cginc"
                #include "UnityUI.cginc"

                #pragma multi_compile __ UNITY_UI_CLIP_RECT
                #pragma multi_compile __ UNITY_UI_ALPHACLIP

                struct appdata_t
                {
                    float4 vertex   : POSITION;
                    float4 color    : COLOR;
                    float2 texcoord : TEXCOORD0;
                    UNITY_VERTEX_INPUT_INSTANCE_ID
                };

                struct v2f
                {
                    float4 vertex   : SV_POSITION;
                    fixed4 color : COLOR;
                    float2 texcoord  : TEXCOORD0;
                    float4 worldPosition : TEXCOORD1;
                    UNITY_VERTEX_OUTPUT_STEREO
                };

                fixed4 _Color;
                fixed4 _TextureSampleAdd;
                float4 _ClipRect;

                v2f vert(appdata_t v)
                {
                    v2f OUT;
                    UNITY_SETUP_INSTANCE_ID(v);
                    UNITY_INITIALIZE_VERTEX_OUTPUT_STEREO(OUT);
                    OUT.worldPosition = v.vertex;
                    OUT.vertex = UnityObjectToClipPos(OUT.worldPosition);

                    OUT.texcoord = v.texcoord;

                    OUT.color = v.color * _Color;
                    return OUT;
                }

                sampler2D _MainTex;

                fixed4 frag(v2f IN) : SV_Target
                {
                    half4 color = (tex2D(_MainTex, IN.texcoord) + _TextureSampleAdd) * IN.color;

                    #ifdef UNITY_UI_CLIP_RECT
                    color.a *= UnityGet2DClipping(IN.worldPosition.xy, _ClipRect);
                    #endif

                    #ifdef UNITY_UI_ALPHACLIP
                    clip(color.a - 0.001);
                    #endif

                    return color;
                }
            ENDCG
            }
        }
}

If anyone could help me with this, I would greatly appreciate it.
Thank you.

2

Bump, anyone please?

3

This is a super tough setup to deal with. The problem is traditionally alpha blended surfaces need to be rendered in back to front order, as transparent shaders will always render “on top” of things that have been rendered previously. But for stencil rendering you actually want the reverse order where objects in the back render after those in the foreground, assuming you want the foreground smoke to mask the background smoke.

The other problem I see is the case in the third image where the foreground smoke is being “hidden” by the background smoke in the areas the background smoke isn’t visible. That’s being caused by the fact the background smoke is rendering to all of those pixels, and thus setting the stencil values for them, and then the wall and house sprites are rendering over the background smoke. That hides the smoke, but the stencil values remain untouched and thus still block the foreground smoke.

I can think of a few ways to make things work better. The first option would change the requirements a bit. Specifically, if you’re okay with background smoke and foreground smoke overlapping, this can fixed by changing the settings you’re using on the material, and using separate materials for the foreground and background smoke, or using a two pass shader.

For the option that just requires two materials, make separate foreground and background copies of the material. Then change the settings from “Equal”, “0”, “Increment”, to “Not Equal”, “1”, “Replace” for the background, and the same but “2” for the foreground. After that the background smoke will be visible through the foreground smoke, but each layer’s smoke will not overlap with other smoke particles in that layer.

The option that uses one material, but a two pass shader, you would add a second pass that is a copy of the first one, but with ColorMask 0 so it doesn’t render anything to the color, and hardcoded stencil settings of Stencil { Ref 0 Comp Always Pass Replace } which will clear the stencil buffer after each particle system runs. This changes the behavior so that it only prevents particles from within a single emitter from overlapping, and even emitters in the same layer will overlap, so it’s probably not what you want.

The last option I can think of requires a very creative setup that involves a second shader, three materials, and possibly some c# code. The basic idea is this: Render the foreground smoke twice, first before the background smoke, and using a shader that uses that ColorMask 0 setting, but is otherwise exactly what you already have. Then render the background smoke as you already are, then render the foreground smoke with a material that uses a Stencil ID of 1 instead of 0. The trick is you’d need two copies of the foreground smoke emitter with the same random seed assigned so they match up. The easiest way to do that might be to have a custom c# script you assign to the foreground smoke that duplicates it, matches up the seed, puts it in a layer before the background smoke, and assigns the “invisible stencil” material to it. That way you don’t have to do that all by hand constantly.

1 Like
4

Thank you for taking the time to look into this. I really appreciate it.

I actually have thought of having a second identical Particle System “filling the gap” that the background one created as well as rendering the particles without the alpha in an empty scene and just making it into a Sprite Sheet adding the transparency to the sheet and using it as an Animation. Not dynamic but it would behave as expected which was with smokes overlapping as you mentioned.
I didn’t think the distant smoke should be invisible behind the first one which is something I probably should have said in the original post for clarity sake.

I posted here before going on this endeavor since I felt there has to be a more elegant way of doing this.

If anyone finds this later : same seed particle systems https://support.unity.com/hc/en-us/articles/208982486-I-need-two-particle-systems-to-behave-identically-to-each-other

I actually am using two different materials but only for Pixel art purity sake, the distant smoke is a tiny bit smaller than the first one so it uses a different sheet with smaller particles, because I didn’t want to scale it down or change the particle size since resizing Pixel art like that is something you should not do. That purity is broken right after that by rotating sprites so I have no idea why I’m so selective about that.

But yes, changing the material settings fixes the issue.

The second smoke is visible through the first one but it no longer does the unwanted masking effect.

Thank you again for helping me and other who might have the same issue in the future.

1 Like