Alpha channel when using transparent cutout shader

Unity Engine
1

Hi guys,

I have a problem and I can’t figure out how to solve it. Maybe someone here can help me.

I want to use a matte shadow shader that only displays shadows on an object but doesn’t display the object itself. I have found a couple shaders that do the trick, but unfortunately they all have a problem: The shadow is not visible in the alpha channel.

I have reduced the problem to the Transparent cutout shader. When using such a cutout shader, the part that will be cut off won’t be visible in the alpha channel. Can someone help me to figure out how to show this cutoff in the alpha channel?

Below you will find an image that shows the problem.

837993--31242--$unity_cutoff_problem.jpg
837993--31242--$unity_cutoff_problem.jpg800×283 42.1 KB

2

There are two issues here. First of all, by default most built-in unity shaders do not write their alpha to the frame buffer (for their interaction with the glow post-effect). To have a transparent shader write alpha, get the code for your specific shader (all built-in shader code can be found from a sticky in this forum) add it to your project and replace any occurrence of “ColorMask RGB” by “ColorMask RGBA”. Make sure to rename the shader and use the renamed shader instead of the built-in for your material.

Secondly, the very nature of cutout makes it impossible to draw color or alpha where the cutout is active, as any pixel that meets the cutout criterium will not be rendered at all. This means that cutout pixels will show the alpha of whatever is behind it rather than alpha 0. This would not be a problem for the example from your 3 pictures, but it might be a problem when multiple masks overlap (or not, depending on what kind of effect you’re going for).

3

Thanks tomvds, that did the trick indeed for the Transparent Cutout shader but unfortunately will not work for the Matte Shadow shader. This shader doesn’t use the ColorMask.

Below you will find an image that shows the problem. The plane in the background doesn’t receive the shadow as an alpha.

I have embedded the source code of the shader. Unfortunately I don’t have much experience with shaders and their code, but maybe you or someone else can give me a direction where I should look.

Shader "FX/Matte Shadow V3" {

	Properties {
	    _Color ("Main Color", Color) = (1,1,1,1)
		_MainTex ("Base (RGB) Trans (A)", 2D) = "white" {}
	 	_Cutoff ("Alpha cutoff", Range(0,1)) = 0.5
	}
	
	SubShader {
	
	    Tags {"Queue"="AlphaTest" "IgnoreProjector"="True" "RenderType"="TransparentCutout"}
		LOD 200
		Blend Zero SrcColor
		AlphaToMask True ColorMask RGBA
	
	
		CGPROGRAM
		
		
		#pragma surface surf ShadowOnly alphatest:_Cutoff
		
		
		fixed4 _Color;
		
		struct Input {
			float2 uv_MainTex;
		};
		
		
		inline fixed4 LightingShadowOnly (SurfaceOutput s, fixed3 lightDir, fixed atten)
		{
			fixed4 c;
			c.rgb = s.Albedo*atten;
			c.a = s.Alpha;
			return c;
		}
		
		
		void surf (Input IN, inout SurfaceOutput o) {
			fixed4 c = _Color;
			o.Albedo = c.rgb;
			o.Alpha = 1;
		}
		
		 
		ENDCG
		
		
		}
	
	Fallback "Transparent/Cutout/VertexLit"

}

838042--31243--$Screen shot 2012-02-22 at 3.29.01 PM.png
838042--31243--$Screen shot 2012-02-22 at 3.29.01 PM.png800×416 48.2 KB

4

This is probably related to my second point, although it’s not easy to tell as I’m not quite clear on what the overall purpose of your shaders is.
Is there any specific reason you need cutout? What happens if you remove the alphaTest:_Cutoff from line 20?

5

What I am triyng to manage is to send my Unity output to Resolume (VJ software) through Syphon. I want to bring the shadows to Resolume without the object that receives the shadow.

Removing “alphatest:_Cutoff” doesn’t change anything.

Tomorrow I will give it a new try. Maybe someone else with suggestions?