Soft Edge and Stencils

TLDR : Shader which lets you see the skybox through other objects.
I am trying to create a stencil shader which exposes a discovered area for an AR application. I have tried quite a few solutions but am not at all experienced with working with shaders. Im certain there must be a simple solution I have missed. In the example below I am using an Alpha Mask for what to draw, but I cannot get the black to be the shade of the skybox instead, I approached this hoping that I could set a value to 0 late into the que in order to prevent it rendering over the skybox but have had no luck. Most recent shader attempts are below, in varying levels of abandonment and frustration.
There is a fairly complex heirarchy needed, so the mask needs to be a seperate object/material from what is being rendered.
Ive been running in circles a bit, help would be appreciated.
5072759--498758--upload_2019-10-16_20-28-43.png
5072759--498770--upload_2019-10-16_20-52-1.png
SurfaceFademask

Shader "SurfaceFadeMask"
{
     Properties {
        _MainTex("Base (RGB) Trans (A)", 2D) = "white" {}
        _Color ("Main Color", Color) = (0,0,0,0)
        _Stencil("Stencil Texture (RGB)", 2D) = "white" {}
     }
     Subshader {
        Tags {
          "Queue"="Transparent"
          "RenderType"="Transparent"
        }
                Stencil
        {
            Ref 1
            Comp Equal
            Pass Keep
        }
        ZTest Greater
        CGPROGRAM
        #pragma surface surf Lambert alpha
         
          struct Input {
           float2 uv_MainTex;
          };
  
          half4 _Color;
          sampler2D _MainTex;
          sampler2D _Stencil;
         
            void surf (Input IN, inout SurfaceOutput o)
            {
             half4 c = tex2D(_MainTex, IN.uv_MainTex) * _Color;
             o.Emission = c.rgb;
             o.Alpha = c.a - tex2D(_Stencil, IN.uv_MainTex).a;
            
            }
        ENDCG
     }
}

OcclusionShader

Shader "OcclusionShader"
{
    SubShader {
            Pass {
                // Render the Occlusion shader before all
                // opaque geometry to prime the depth buffer.
                Tags { "Queue"="Geometry" }

                ZWrite On
                ZTest LEqual
                ColorMask RGBA
                Blend SrcAlpha OneMinusSrcAlpha
                CGPROGRAM
                #pragma vertex vert
                #pragma fragment frag

                #include "UnityCG.cginc"

                struct appdata
                {
                    float4 vertex : POSITION;
                };

                struct v2f
                {
                    float4 position : SV_POSITION;
                };

                v2f vert (appdata input)
                {
                    v2f output;
                    output.position = UnityObjectToClipPos(input.vertex);
                    return output;
                }

                fixed4 frag (v2f input) : SV_Target
                {
                    return 0.5;
                }
                ENDCG
            }
    }
}

A soft edge and stencil masks are mutually exclusive. You can have one or the other, not both.

However if all you need really is just to show the skybox through objects, then I’d setup a second camera that only renders the skybox to a render texture, then have your “hole” be an object using screen space UVs to show the skybox render texture.

2 Likes

Thanks for the quick reply mate. I think you were somewhere in half the threads I read on this stuff :stuck_out_tongue:
If I was able to sort out a method without stencils, would there be a way to do a soft occlusion material which blends all the way to the skybox or am I going back to the drawing board?