Is Alpha to coverage possible in Unity?

[brisck1]

I’m wondering if anyone knows if Unity 5 supports the Alpha to coverage technique as described here: Humus - 3D

My goal is to achieve soft edged, anti-aliased alpha masks for foliage. I realize this is a forward rendering feature and requires MSAA, but it would be super nice to have in Unity. As of now, alpha masked textures just look aliased and horrible.

I’ve seen some talk of enabling AlphaToMask in on the forums from a few years back but I’ve never seen any examples of it actually working nor have I managed to get it working myself.

Any help of advice would be greatly appreciated!

Thanks.

[bgolus]

Yes, it works, but it takes more than just adding AlphaToMask On to a surface shader. First it only works with anti-aliasing turned on so it doesn’t work with deferred. Second it works well in a vertex & fragment shader but you’ll need to modify individual the passes of a surface shader so you’ll need to click on the “generate” button and modify the vertex & fragment shader a surface shader creates to get it to work. Only the forward base and add passes should use it. Everything else needs to be using clip().

One more warning; it doesn’t play well with realtime directional shadows. Because of the way Unity currently does the main directional light’s shadows as a full screen pass using the depth buffer you can get odd shadow fringing depending on how you setup your shadowcaster pass. Additional shadows and light maps work fine though.

[brisck1]

Thanks for the reply, interesting, I will give that a go and report back

[brisck1]

Ok I gave this a go by creating a simple unlit alpha masked shader (I used the built-in Unlit-AlphaTest.shader as a base) then adding in AlphaToMask On as suggested and recompiled the shader.

At first, it appears to have worked, although not quite as expected:

it looks softer from up close, and I can see various levels of transparency however it is still aliased around the edge of the mask - not quite what I was expecting in comparison to the images from the link in my original post.

Am I perhaps doing something wrong in the shader code:

// Unlit alpha-cutout shader.
// - no lighting
// - no lightmap support
// - no per-material color

Shader "AlphaToCoverage/Alpha to Coverage Test" {
Properties {
    _MainTex ("Base (RGB) Trans (A)", 2D) = "white" {}
    _Cutoff ("Alpha cutoff", Range(0,1)) = 0.5
}
SubShader {
    Tags {"Queue"="AlphaTest" "IgnoreProjector"="True" "RenderType"="TransparentCutout"}
    LOD 100

    Lighting Off
    AlphaToMask On
  
    Pass {
        CGPROGRAM
            #pragma vertex vert
            #pragma fragment frag
            #pragma multi_compile_fog
          
            #include "UnityCG.cginc"

            struct appdata_t {
                float4 vertex : POSITION;
                float2 texcoord : TEXCOORD0;
            };

            struct v2f {
                float4 vertex : SV_POSITION;
                half2 texcoord : TEXCOORD0;
                UNITY_FOG_COORDS(1)
            };

            sampler2D _MainTex;
            float4 _MainTex_ST;
            fixed _Cutoff;

            v2f vert (appdata_t v)
            {
                v2f o;
                o.vertex = mul(UNITY_MATRIX_MVP, v.vertex);
                o.texcoord = TRANSFORM_TEX(v.texcoord, _MainTex);
                UNITY_TRANSFER_FOG(o,o.vertex);
                return o;
            }
          
            fixed4 frag (v2f i) : SV_Target
            {
                fixed4 col = tex2D(_MainTex, i.texcoord);
                clip(col.a - _Cutoff);
                UNITY_APPLY_FOG(i.fogCoord, col);
                return col;
            }
        ENDCG
    }
}

}

To clarify; I have Forward rendering enabled and MSAA set to 8x.

Thanks again!

[bgolus]

Remove the clip(), or set the cutoff to something low like 0.1

[brisck1]

Just tried setting the cutoff to 0.01 and it now looks a lot softer, thanks!

Such a bummer that it doesn’t work with dynamic directional shadows though Hopefully i’ll be able to work around it somehow.

So I have one last question left: I want to apply this to a shader built using Shader Forge (which doesn’t currently support AlphaToMask) so I’m attempting to insert the AlphaToMask On line into the SF shader code in the same way as I did before, however it does not seem to work for some reason, could you take a look at this shader code and tell me what’s going wrong?:

// Shader created with Shader Forge v1.26
// Shader Forge (c) Neat Corporation / Joachim Holmer - http://www.acegikmo.com/shaderforge/
// Note: Manually altering this data may prevent you from opening it in Shader Forge
/*SF_DATA;ver:1.26;sub:START;pass:START;ps:flbk:,iptp:0,cusa:False,bamd:0,lico:1,lgpr:1,limd:0,spmd:1,trmd:0,grmd:0,uamb:True,mssp:True,bkdf:False,hqlp:False,rprd:False,enco:False,rmgx:True,rpth:0,vtps:0,hqsc:True,nrmq:1,nrsp:0,vomd:0,spxs:False,tesm:0,olmd:1,culm:0,bsrc:0,bdst:1,dpts:2,wrdp:True,dith:0,rfrpo:True,rfrpn:Refraction,coma:15,ufog:False,aust:True,igpj:False,qofs:0,qpre:2,rntp:3,fgom:False,fgoc:False,fgod:False,fgor:False,fgmd:0,fgcr:0.5,fgcg:0.5,fgcb:0.5,fgca:1,fgde:0.01,fgrn:0,fgrf:300,stcl:False,stva:128,stmr:255,stmw:255,stcp:6,stps:0,stfa:0,stfz:0,ofsf:0,ofsu:0,f2p0:False,fnsp:False,fnfb:False;n:type:ShaderForge.SFN_Final,id:3138,x:32719,y:32712,varname:node_3138,prsc:2|emission-5813-RGB,clip-5813-A;n:type:ShaderForge.SFN_Tex2d,id:5813,x:32417,y:32868,ptovrint:False,ptlb:Base,ptin:_Base,varname:_Base,prsc:2,glob:False,taghide:False,taghdr:False,tagprd:False,tagnsco:False,tagnrm:False,tex:d6fcda50c6589554380ee9dbb4c702ea,ntxv:0,isnm:False;proporder:5813;pass:END;sub:END;*/

Shader "AlphaToCoverage/SFUnlit-AlphaCoverageTest" {
    Properties {
        _Base ("Base", 2D) = "white" {}
        [HideInInspector]_Cutoff ("Alpha cutoff", Range(0,1)) = 0.5
    }
    SubShader {
        Tags {
            "Queue"="AlphaTest"
            "RenderType"="TransparentCutout"
        }
       
        AlphaToMask On
       
        Pass {
            Name "FORWARD"
            Tags {
                "LightMode"="ForwardBase"
            }
           
           
            CGPROGRAM
            #pragma vertex vert
            #pragma fragment frag
            #define UNITY_PASS_FORWARDBASE
            #include "UnityCG.cginc"
            #pragma multi_compile_fwdbase_fullshadows
            #pragma exclude_renderers gles gles3 metal d3d11_9x xbox360 xboxone ps3 ps4 psp2
            #pragma target 3.0
            uniform sampler2D _Base; uniform float4 _Base_ST;
            struct VertexInput {
                float4 vertex : POSITION;
                float2 texcoord0 : TEXCOORD0;
            };
            struct VertexOutput {
                float4 pos : SV_POSITION;
                float2 uv0 : TEXCOORD0;
            };
            VertexOutput vert (VertexInput v) {
                VertexOutput o = (VertexOutput)0;
                o.uv0 = v.texcoord0;
                o.pos = mul(UNITY_MATRIX_MVP, v.vertex );
                return o;
            }
            float4 frag(VertexOutput i) : COLOR {
                float4 _Base_var = tex2D(_Base,TRANSFORM_TEX(i.uv0, _Base));
                clip(_Base_var.a - 0.5);
////// Lighting:
////// Emissive:
                float3 emissive = _Base_var.rgb;
                float3 finalColor = emissive;
                return fixed4(finalColor,1);
            }
            ENDCG
        }
        Pass {
            Name "ShadowCaster"
            Tags {
                "LightMode"="ShadowCaster"
            }
            Offset 1, 1
           
            CGPROGRAM
            #pragma vertex vert
            #pragma fragment frag
            #define UNITY_PASS_SHADOWCASTER
            #include "UnityCG.cginc"
            #include "Lighting.cginc"
            #pragma fragmentoption ARB_precision_hint_fastest
            #pragma multi_compile_shadowcaster
            #pragma exclude_renderers gles gles3 metal d3d11_9x xbox360 xboxone ps3 ps4 psp2
            #pragma target 3.0
            uniform sampler2D _Base; uniform float4 _Base_ST;
            struct VertexInput {
                float4 vertex : POSITION;
                float2 texcoord0 : TEXCOORD0;
            };
            struct VertexOutput {
                V2F_SHADOW_CASTER;
                float2 uv0 : TEXCOORD1;
            };
            VertexOutput vert (VertexInput v) {
                VertexOutput o = (VertexOutput)0;
                o.uv0 = v.texcoord0;
                o.pos = mul(UNITY_MATRIX_MVP, v.vertex );
                TRANSFER_SHADOW_CASTER(o)
                return o;
            }
            float4 frag(VertexOutput i) : COLOR {
                float4 _Base_var = tex2D(_Base,TRANSFORM_TEX(i.uv0, _Base));
                clip(_Base_var.a - 0.5);
                SHADOW_CASTER_FRAGMENT(i)
            }
            ENDCG
        }
    }
    FallBack "Diffuse"
    CustomEditor "ShaderForgeMaterialInspector"
}

Thanks again!

[bgolus]

So I use two tricks for this.

The simple one is to use a hardcoded clip of 0.1 on the forward pass, and 0.8 on the shadowcaster pass. It doesn’t fix the issue but mitigates the most egregious error of the bright edge.

The more complex method requires overriding some of the cginc files to change how it samples the screen space shadows. See:

"Fixing" Screen Space Directional Shadows and Anti-Aliasing

For a project I’m working on I stipple the alpha in the shadow casting pass and the above bit of code produces slightly cleaner shadows with less fringing.

For the SF shader try putting the line within the forward pass {} instead of above it and change the clip() in the forward pass to not be 0.5

[brisck1]

Thanks again for your help!

I tried moving the AlphaToMask On to the Forward section of the shader but unfortunately it still does not work:

Shader "AlphaToCoverage/SFUnlit-AlphaCoverageTest" {
    Properties {
        _Base ("Base", 2D) = "white" {}
        _Cutoff ("Alpha cutoff", Range(0,1)) = 0.01
    }
    SubShader {
        Tags {
            "Queue"="AlphaTest"
            "RenderType"="TransparentCutout"
        }
      
        Pass {
            Name "FORWARD"
            Tags {
                "LightMode"="ForwardBase"
            }
          
            AlphaToMask On
          
            CGPROGRAM
            #pragma vertex vert
            #pragma fragment frag
            #define UNITY_PASS_FORWARDBASE
            #include "UnityCG.cginc"
            #pragma multi_compile_fwdbase_fullshadows
            #pragma exclude_renderers gles gles3 metal d3d11_9x xbox360 xboxone ps3 ps4 psp2
            #pragma target 3.0
            uniform sampler2D _Base; uniform float4 _Base_ST;
            struct VertexInput {
                float4 vertex : POSITION;
                float2 texcoord0 : TEXCOORD0;
            };
            struct VertexOutput {
                float4 pos : SV_POSITION;
                float2 uv0 : TEXCOORD0;
            };
            VertexOutput vert (VertexInput v) {
                VertexOutput o = (VertexOutput)0;
                o.uv0 = v.texcoord0;
                o.pos = mul(UNITY_MATRIX_MVP, v.vertex );
                return o;
            }
            float4 frag(VertexOutput i) : COLOR {
                float4 _Base_var = tex2D(_Base,TRANSFORM_TEX(i.uv0, _Base));
                clip(_Base_var.a - 0.01);
////// Lighting:
////// Emissive:
                float3 emissive = _Base_var.rgb;
                float3 finalColor = emissive;
                return fixed4(finalColor,1);
            }
            ENDCG
        }
        Pass {
            Name "ShadowCaster"
            Tags {
                "LightMode"="ShadowCaster"
            }
            Offset 1, 1
          
            CGPROGRAM
            #pragma vertex vert
            #pragma fragment frag
            #define UNITY_PASS_SHADOWCASTER
            #include "UnityCG.cginc"
            #include "Lighting.cginc"
            #pragma fragmentoption ARB_precision_hint_fastest
            #pragma multi_compile_shadowcaster
            #pragma exclude_renderers gles gles3 metal d3d11_9x xbox360 xboxone ps3 ps4 psp2
            #pragma target 3.0
            uniform sampler2D _Base; uniform float4 _Base_ST;
            struct VertexInput {
                float4 vertex : POSITION;
                float2 texcoord0 : TEXCOORD0;
            };
            struct VertexOutput {
                V2F_SHADOW_CASTER;
                float2 uv0 : TEXCOORD1;
            };
            VertexOutput vert (VertexInput v) {
                VertexOutput o = (VertexOutput)0;
                o.uv0 = v.texcoord0;
                o.pos = mul(UNITY_MATRIX_MVP, v.vertex );
                TRANSFER_SHADOW_CASTER(o)
                return o;
            }
            float4 frag(VertexOutput i) : COLOR {
                float4 _Base_var = tex2D(_Base,TRANSFORM_TEX(i.uv0, _Base));
                clip(_Base_var.a - 0.5);
                SHADOW_CASTER_FRAGMENT(i)
            }
            ENDCG
        }
    }
    FallBack "Diffuse"
    CustomEditor "ShaderForgeMaterialInspector"
}

Thanks for directing me to your own solution for the shadow problem. I’m rather new to writing shader code manually (I usually use Shader Forge to make most of my custom shaders) so could you explain in a little more detail how I should use the example shader code in that thread? I’m guessing there is more to it than just copying and pasting the code into my project?

[bgolus]

If you don’t understand how to use that code I recommend you not use it.

As for why it still doesn’t work I missed the fact the forward pass is returning 1 for the alpha. It needs to use the alpha value of the texture for the alpha of return fixed4().

[brisck1]

Awesome! So I changed

return fixed4(finalColor,1);

to

return fixed4(finalColor, _Base_var.a);

and that fixed it! Thanks so much for your help, I really hope Unity implement proper support for this with shadows soon.

[Pangamini_1]

You can actually use the AlphaToMask with surface shaders without editing the compiled one, just add keepalpha to your surface shader #pragma

[bgolus]

That works, as long as you don’t need custom shadow casting, yes. The problem there actually comes from the fact that the shadow caster’s fragment shader always outputs a zero, thus alpha to coverage is clipping the depth write. Using a shadow caster from a Fallback should work though.

[tjholleran]

Thanks for this tread. Has anything happened in unity 2018.x that makes the shadow workarounds easier for alpha to cover?

[bgolus]

Nope.

My solution these days has been to write a custom vertex fragment shadow pass in the shader instead of relying on the addshadow or fallback.

[tjholleran]

Hola bgolus,
I have followed down the rabbit hole, with basic custom shadow pass, and applied shadow in forward base.
Shadows received on the surface look ok, but, shadows cast don’t respect our alpha … and though it casts an essentially opaque shadow, I con’t see that thru the transparent portions of our alpha test gag.

Is there s special version of SHADOW_CASTER_FRAGMENT(i) that I should be calling to respect the alpha, … or do I need to set up samplers for the alpha texture in this pass?
Pass
{
Tags{ “LightMode” = “ShadowCaster” }
CGPROGRAM
#pragma vertex vert
#pragma fragment frag
#pragma multi_compile_shadowcaster
#include “UnityCG.cginc”
struct v2f {
V2F_SHADOW_CASTER;
};
v2f vert(appdata_base v)
{
v2f o;
TRANSFER_SHADOW_CASTER_NORMALOFFSET(o)
return o;
}
float4 frag(v2f i) : SV_Target
{
SHADOW_CASTER_FRAGMENT(i)
}
ENDCG
}

Here you can see a quad with the Naive version of the Alpha to Cover, … and you can see geo thru the clipped alpha, but not the shadow. Yet, you can see the shadow of the entire quad.

Thanks in advance,
TJ

[bgolus]

The shadow caster still needs to use traditional alpha testing.

[tjholleran]

Thank you much … I am in the ballpark I think … have the custom textured shadow pass in.
I ran into an interesting issue. My main pass mults the texture alpha with a facing ratio alpha.
I don’t want the facing fade to impact the shadows as I move the view, so I exclude any facing gag from the shadow pass. But, I shadows give a ‘ghost’ version of the textured alpha card in areas where the facing ratio nixes the main pass. This is a test not using the alphaToCover … just so I can get the facing gag in.

Pass
{
Tags{ “LightMode” = “ShadowCaster” }

CGPROGRAM
#pragma vertex vert
#pragma fragment frag
#pragma multi_compile_shadowcaster
#include “UnityCG.cginc”
#include “UnityStandardShadow.cginc”

half _AlphaTextureBias;
half _AlphaTextureBiasShadowOffset;

struct appdata
{
float4 vertex : POSITION;
float3 normal : NORMAL;
float4 texcoord : TEXCOORD0;
float4 color : COLOR;

UNITY_VERTEX_INPUT_INSTANCE_ID
};

struct v2f {
V2F_SHADOW_CASTER;
float2 uv : TEXCOORD0;
};

//v2f vert(appdata_base v)
v2f vert(appdata v)
{
v2f o;
o.uv = TRANSFORM_TEX(v.texcoord, _MainTex);

TRANSFER_SHADOW_CASTER_NORMALOFFSET(o)
return o;
}

float4 frag(v2f i, fixed facing : VFACE) : SV_Target
{
half alpha = tex2D(_MainTex, i.uv.xy).a;
alpha += _AlphaTextureBias + _AlphaTextureBiasShadowOffset;

clip(alpha - _Cutoff);

SHADOW_CASTER_FRAGMENT(i)
}
ENDCG
}

[bgolus]

The shadow caster pass is used for both shadow casting and the main directional light’s shadow receiving. If you’re fading out based on view direction in the main pass, you will need to also fade out in the shadow caster pass, but only when rendering the main scene depth.

Unfortunately there’s no clean way to do that. The best solution I have thus far found is to check to see if the current projection matrix is a perspective matrix or orthographic matrix.

Differentiating between shadow caster and shadow receiver in 5.2

Has anyone figured out a clean way (ie: some combination of defines to #ifdef between) to discern between the main directional light shadow depth rendering and the shadow receiver depth rendering? Since 5.0 merged the two it's made it difficult to handle each case differently (useful for screen space alpha test effects and billboard shadow casting). The only test I've been able to come up with is to test for an orthographic projection to differentiate between the scene depth and shadow depth, …

[tjholleran]

Awesome, … that did the trick.
I took unity_LightShadowBias.z as the hook you were left with as the best, from that thread.

Would not have found that for years … so, thank u much for all the awesome info on this topic.
Now, I am on to tackle some less naive version of my alphaTocCover.

[tjholleran]

Back at this again.
Am I correct in my understanding that sampling an individual cascade (furthest) is not easily achieved, … and that SHADOW_ATTENUATION() is effectively a screen space shadow result of combining the cascades?

Thanks in advance.