Can a Shader Not Render Pixels if A Camera's background is Transparent?

[kuchaku]

Hi. I’m not familiar with shader language and am making a 2D game where I’d like a texture to not render over transparent (empty) areas of of the first camera.

Below I’m drawing a procedural texture of black squares to simulate lighting. Would it be possible for a shader to ‘overlay’ this texture onto pixels that aren’t transparent or ‘empty’ in one camera, so I could place a second camera including clean backgrounds.

I don’t want torches lighting up the sky and mountains. The closet comparison I have is how on photoshop you can have the overlay feature that doesn’t draw anything if the pixel is transparent. It would be really useful for decals too. Thanks for reading.

[Zicandar]

You can do this using blend mode or stencil buffer.
With blend mode you use DstAlpha OneMinusDstAlpha, Zero One
With stencil you simply make the “normal” shaders write a bit to the stencil buffer, then when doing the lights/decals only apply them when it’s set.

[kuchaku]

@Zicandar

You can do this using blend mode or stencil buffer.
With blend mode you use DstAlpha OneMinusDstAlpha, Zero One
With stencil you simply make the “normal” shaders write a bit to the stencil buffer, then when doing the lights/decals only apply them when it’s set.

I will research both approaches. Thank you. The texture is procedurally generated so I can’t seem to apply it as a light cookie.

[kuchaku]

Well, your blendmode code took a step possibly in the right direction:

But it is silhouetted as it lost the alpha information of the black texture being overlayed. I haven’t gotten too deep into the stencil buffer as it would require custom shader for everything.

The problem I’m facing is it appears procedural textures imported as ‘advanced’ can’t be used as light cookies and since my texture is generated dynamically based on modifiable level design with SetPixel operations this is painting me into a corner.

If I learn the custom shader language, could I preserve the alpha of the texture being overlayed with custom operations?

[Zicandar]

@kuchaku

If I learn the custom shader language, could I preserve the alpha of the texture being overlayed with custom operations?

This wouldn’t even be about the shader language, all you need to understand is a single line!

DstAlpha OneMinusDstAlpha, Zero One

http://docs.unity3d.com/Manual/SL-Blend.html
The second pair of words I wrote is for what happens to the alpha channel.
Zero One means ignore the alpha coming out of this pixel shader, and only care about the alpha of whatever exists in the buffer.

[kuchaku]

I think I actually do understand it now. The alpha was how I faked the shadow. I used SetPixel and created a transparent shadow texture that is calculated according to ever changing level layouts on the fly.

So when it tried to blend this transparent texture, it ignored the transparency of this texture and layered it black. I tried as many blend mode combinations and none were coming up proper.

The problem is advanced procedural textures with read/write capabilities can’t be used as light cookies as far as I know so if I’m going to use a procedural texture I need some method of faking. Stencils, your second method are looking promising.

[kuchaku]

I found a solution that works but is very slow. I stencil in any foreground elements:

Shader "Sprites/SpritesUnlitStencilSet"
{
    Properties
    {
        [PerRendererData] _MainTex ("Sprite Texture", 2D) = "white" {}
        _Color ("Tint", Color) = (1,1,1,1)
        [MaterialToggle] PixelSnap ("Pixel snap", Float) = 0
    }

    SubShader
    {
        Tags
        {
            "Queue"="Transparent"
            "IgnoreProjector"="True"
            "RenderType"="Transparent"
            "PreviewType"="Plane"
            "CanUseSpriteAtlas"="True"
        }

        Cull Off
        Lighting Off
        ZWrite Off
        Blend One OneMinusSrcAlpha

        Pass
        {
         Stencil {
            Ref 254
            Comp always
            Pass replace
        }
     
     
        CGPROGRAM
            #pragma vertex vert
            #pragma fragment frag
            #pragma multi_compile _ PIXELSNAP_ON
            #include "UnityCG.cginc"
         
            struct appdata_t
            {
                float4 vertex   : POSITION;
                float4 color    : COLOR;
                float2 texcoord : TEXCOORD0;
            };

            struct v2f
            {
                float4 vertex   : SV_POSITION;
                fixed4 color    : COLOR;
                half2 texcoord  : TEXCOORD0;
            };
         
            fixed4 _Color;

            v2f vert(appdata_t IN)
            {
                v2f OUT;
                OUT.vertex = mul(UNITY_MATRIX_MVP, IN.vertex);
                OUT.texcoord = IN.texcoord;
                OUT.color = IN.color * _Color;
                #ifdef PIXELSNAP_ON
                OUT.vertex = UnityPixelSnap (OUT.vertex);
                #endif

                return OUT;
            }

            sampler2D _MainTex;

            fixed4 frag(v2f IN) : SV_Target
            {
                 fixed4 c = tex2D(_MainTex, IN.texcoord) * IN.color;
                 if (c.a<0.1) discard;            //Most IMPORTANT working Code Very slow
                 c.rgb *= c.a;
                 return c;
            }
        ENDCG
        }
    }
}

And discard any transparent pixels in the process (very slow). Then I check the stencil buffer with another shader:

Shader "Sprites/SpritesUnlitStencilCheckFor1"
{
    Properties
    {
        [PerRendererData] _MainTex ("Sprite Texture", 2D) = "white" {}
        _Color ("Tint", Color) = (1,1,1,1)
        [MaterialToggle] PixelSnap ("Pixel snap", Float) = 0
    }

    SubShader
    {
        Tags
        {
            "Queue"="Overlay"
            "IgnoreProjector"="True"
            "RenderType"="Transparent"
            "PreviewType"="Plane"
            "CanUseSpriteAtlas"="True"
        }

        Cull Off
        Lighting Off
        ZWrite Off
        Blend One OneMinusSrcAlpha

        Pass
        {
   Stencil  {
            Ref 254
            Comp LEqual
            Fail keep
            }
     
     
        CGPROGRAM
            #pragma vertex vert
            #pragma fragment frag
            #pragma multi_compile _ PIXELSNAP_ON
            #include "UnityCG.cginc"
         
            struct appdata_t
            {
                float4 vertex   : POSITION;
                float4 color    : COLOR;
                float2 texcoord : TEXCOORD0;
            };

            struct v2f
            {
                float4 vertex   : SV_POSITION;
                fixed4 color    : COLOR;
                half2 texcoord  : TEXCOORD0;
            };
         
            fixed4 _Color;

            v2f vert(appdata_t IN)
            {
                v2f OUT;
                OUT.vertex = mul(UNITY_MATRIX_MVP, IN.vertex);
                OUT.texcoord = IN.texcoord;
                OUT.color = IN.color * _Color;
                #ifdef PIXELSNAP_ON
                OUT.vertex = UnityPixelSnap (OUT.vertex);
                #endif

                return OUT;
            }

            sampler2D _MainTex;

            fixed4 frag(v2f IN) : SV_Target
            {
                     
         
                fixed4 c = tex2D(_MainTex, IN.texcoord) * IN.color;
                c.rgb *= c.a;         
             
                return c;
            }
        ENDCG
        }
    }
}

If I don’t discard the pixels it looks terrible as it stencils in the transparent areas this is what it looks like:

If I do (and every object in the entire game would need to do this with this method) it looks fine, but it will add up with more sprites:

If anyone is aware of a method of multiplaying the alpha of your the primarily drawn texture with the alpha of backbuffer or using a render texture’s alpha? Finding another way to mask out the transparent area behind the camera that is more efficient I’d appreciate it.

If there isn’t a more efficient method, at least I found a way that works and maybe someone else will find some use in it if approaching a similar issue.

[Zicandar]

I’m not completly sure of exactly what blend you want, but you CAN multiply the result of your pixel shader with the alpha in the back buffer!
Also using a Blend srcAlpha oneMinusSrcAlpha, Zero One you can draw sprites that don’t affect the backbuffers alpha!
If you could share the shaders you are using it’d really help!

[kuchaku]

@Zicandar

I’m not completly sure of exactly what blend you want, but you CAN multiply the result of your pixel shader with the alpha in the back buffer!
Also using a Blend srcAlpha oneMinusSrcAlpha, Zero One you can draw sprites that don’t affect the backbuffers alpha!
If you could share the shaders you are using it’d really help!

The shaders I’m currently using were included in an above post. You sent a PM I missed awhile back. I’ll hit you up on that.

The shader would need sto do two things:

1. Render a sprite completely normally with transparency and color would be a bonus
2. Multiply the result of that render with background alpha (or choose not to draw) when the background alpha is low.

My duct taped solution of stenciling out foreground elements and discarding pixels 'works’s in a ham fisted way. I haven’t put much work into this specifically since, but the long term solution appears to be a grab pass (which grabs the backbuffer) or a render texture of the foreground camera, to use as a comparative reference and then multiply with that. It’s basically a matter of learning shader language.