Mask Shader and Shadow

Hello Developers!

I working on a team based game with fog of war / line of sight.

I use 2 shader :

  • Enemy shader, what is not visible as default
  • Mask shader, what reveals the Enemy shader materials

The problem is, that the hidden enemy objects have visible shadows…

I like to hide / mask out the hidden object shadows too and I hope that I can still use shaders, without scripting if it possible.
61440-mask.jpg

Please help me, to solve this. Im very beginner with shaders.

The Shaders:

Enemy shader (this is not visible without the mask):

Shader "Example/Diffuse Texture Enemy" {
    Properties {
      _MainTex ("Texture", 2D) = "white" {}
    }
    SubShader {
      Tags { "RenderType" = "Opaque" }
      
      Stencil {
			Ref 1
			Comp equal
		}
      
      CGPROGRAM
      #pragma surface surf Lambert
      struct Input {
          float2 uv_MainTex;
      };
      sampler2D _MainTex;
      void surf (Input IN, inout SurfaceOutput o) {
          o.Albedo = tex2D (_MainTex, IN.uv_MainTex).rgb;
      }
      ENDCG
    } 
    Fallback "Diffuse"
  }

Mask shader (on a mesh, this make visible the Enemy shader materials):

Shader "Custom/Stencil Mask" {
	Properties {
		_Color ("Color", Color) = (1,1,1,1)
		_MainTex ("Albedo (RGB)", 2D) = "white" {}
		_Glossiness ("Smoothness", Range(0,1)) = 0.5
		_Metallic ("Metallic", Range(0,1)) = 0.0
	}
	SubShader {
		Tags { "RenderType"="Opaque" "Queue"="Geometry-100" }
		ColorMask 0
		ZWrite off
		LOD 200

		Stencil {
			Ref 1
			Pass replace
		}

		CGPROGRAM
		// Physically based Standard lighting model, and enable shadows on all light types
		#pragma surface surf Standard fullforwardshadows

		// Use shader model 3.0 target, to get nicer looking lighting
		#pragma target 3.0

		sampler2D _MainTex;

		struct Input {
			float2 uv_MainTex;
		};

		half _Glossiness;
		half _Metallic;
		fixed4 _Color;

		void surf (Input IN, inout SurfaceOutputStandard o) {
			// Albedo comes from a texture tinted by color
			fixed4 c = tex2D (_MainTex, IN.uv_MainTex) * _Color;
			o.Albedo = c.rgb;
			// Metallic and smoothness come from slider variables
			o.Metallic = _Metallic;
			o.Smoothness = _Glossiness;
			o.Alpha = c.a;
		}
		ENDCG
	}
	FallBack "Diffuse"
}

Possible SOLUTION: For anyone who bumps into this post and is having trouble with shadows on their invisible objects. I fiddled around while using stencil shaders to setup a portal effect and got something pretty decent as a workaround. Here are the steps taken below.

  1. Mark environment and/or other items which will NOT be invisible as ‘static’ so we can bake a lightmap of area.

2)On first directional light in scene go to its component and change to ‘baked’ (must do this first before next steps OR process will not work correctly!)

3)Go to Unity ‘Light Settings’ and bake your lightmap. Afterwards you should see that shadows have baked ONLY onto items marked as static. This will make your invisible items lose their shadow entirely also. We still likely want to see shadows on those object when the object itself is not invisible intentionally… we’ll fix this in the next few steps.

4)Overlap other area (if your going for a portal effect) or duplicate current area on top of existing one (if you just want shadows to appear when viewed through the plane you are using as a stencil filter). Note: Be sure that everything in the newly overlapping area, which you do want to have a shadow, is using your shader which allows objects to be viewed when seen through the stencil filter plane. So basically as you would normally set this sort of thing up anyway, ie one shader on object to be seen and different shader on object to do the seeing. Also KEEP this environment as NOT STATIC.

5)Now, add second directional light to scene and keep this set as ‘realtime’. You will notice that now a light version of a shadow is appearing on invisible objects yet again, this is momentary. We have added this second light so that we can have shadow on objects when they are intentionally viewed.

6)Now, select first environment (all objects which were marked static) and create and add them to a new layer. I called mine ‘No Shadow’ just for easy use.

  1. Go back to the DirectionalLight (your second one) and in its component uncheck the ‘No Shadow’ Option… causing it to cull just that layer.

Hopefully this will help others… seemed to work for me :slight_smile: