Deferred shader, modify depth without it affecting ZTest

Hey everybody, im trying to write a shader for cutting out spheres out of geometry. Heres my process:
a) Render all faces clipping out the pixels inside a sphere
b) Render only back faces, doing the ray sphere intersection, finding normal and world position of the intersection, and writing that to the G-Buffer.
There is a problem however, that i illustrated here:
3373500--264586--problem.png
The red ray intersects with the sphere and sets its depth closer than the front face, which causes it to be rendered on top of the front face, which leads to incorrect looking results:
3373500--264587--problem3d.PNG

So my question is, can i write to the depth buffer, but somehow use the original depth for the ZTesting?
I think it would also be possible with stencil buffer, but im on deferred, so thats not an option. And the worst case scenario is rendering a separate render texture, i would like to avoid that if possible.

Heres the shader code btw:
Shader code

Shader "Custom/CharacterShader" {
    Properties {
        _Color ("Color", Color) = (1,1,1,1)
        _MainTex ("Albedo (RGB)", 2D) = "white" {}
        _NoiseTex("Noise", 2D) = "white" {}
        _Glossiness ("Smoothness", Range(0,1)) = 0.5
        _Metallic ("Metallic", Range(0,1)) = 0.0
        _SpherePos("Sphere pos", Vector) = (0,0,0,0)
    }
    SubShader {

        Tags { "RenderType"="Opaque" }
        LOD 200
        Cull Off

        CGPROGRAM
        // Physically based Standard lighting model, and enable shadows on all light types
        #pragma surface surf Standard fullforwardshadows vertex:vert addshadow

        // Use shader model 3.0 target, to get nicer looking lighting
        #pragma target 3.0

        sampler2D _MainTex;
        sampler2D _NoiseTex;

        struct Input {
            float2 uv_MainTex;
            float3 localPos;
        };

        half _Glossiness;
        half _Metallic;
        fixed4 _Color;
        float4 _SpherePos;

        void vert (inout appdata_full v, out Input o) {
            UNITY_INITIALIZE_OUTPUT(Input,o);
            o.localPos = v.vertex.xyz;
        }

        // Add instancing support for this shader. You need to check 'Enable Instancing' on materials that use the shader.
        // See https://docs.unity3d.com/Manual/GPUInstancing.html for more information about instancing.
        // #pragma instancing_options assumeuniformscaling
        UNITY_INSTANCING_BUFFER_START(Props)
            // put more per-instance properties here
        UNITY_INSTANCING_BUFFER_END(Props)

        void surf (Input IN, inout SurfaceOutputStandard o) {
            float noise = tex2D(_NoiseTex, IN.localPos.xy);
            clip(distance(_SpherePos.xyz, IN.localPos)-0.5+noise*0.25*0);
            // Albedo comes from a texture tinted by color
            fixed4 c = tex2D (_MainTex, IN.uv_MainTex) * _Color;
            o.Albedo = c.rgb;
            // Metallic and smoothness come from slider variables
            o.Metallic = _Metallic;
            o.Smoothness = _Glossiness;
            o.Alpha = c.a;
        }
        ENDCG

        Pass {      
            Tags { "LightMode" = "Deferred" }
            Cull Front

            CGPROGRAM
            #pragma vertex vert
            #pragma fragment frag
            #pragma exclude_renderers nomrt
            #pragma multi_compile ___ UNITY_HDR_ON
            #pragma target 3.0

            #include "UnityPBSLighting.cginc"

            float4 _SpherePos;

            float raySphereIntersect(float3 r0, float3 rd, float3 s0, float sr) {
                // - r0: ray origin
                // - rd: normalized ray direction
                // - s0: sphere center
                // - sr: sphere radius
                // - Returns distance from r0 to first intersecion with sphere,
                //   or -1.0 if no intersection.
                float a = dot(rd, rd);
                float3 s0_r0 = r0 - s0;
                float b = 2.0 * dot(rd, s0_r0);
                float c = dot(s0_r0, s0_r0) - (sr * sr);
                if (b*b - 4.0*a*c < 0.0) {
                    return -1.0;
                }
                return (-b - sqrt((b*b) - 4.0*a*c))/(2.0*a);
            }

            float CalcDepth(float3 vert) {
                float4 pos_clip = mul(UNITY_MATRIX_VP, float4(vert,1));
                return pos_clip.z / pos_clip.w;
            }

            struct structureVS
            {
                float4 screen_vertex : SV_POSITION;
                float4 world_vertex : TEXCOORD0;
                float4 local_vertex: TEXCOORD2;
                float3 normal : TEXCOORD1;
            };
        
            struct structurePS
            {
                half4 albedo : SV_Target0;
                half4 specular : SV_Target1;
                half4 normal : SV_Target2;
                half4 emission : SV_Target3;
                float depth: SV_DEPTH;
            };
        
            structureVS vert (float4 vertex : POSITION,float3 normal : NORMAL)
            {
                structureVS vs;
                vs.local_vertex = vertex;
                vs.screen_vertex = UnityObjectToClipPos( vertex );
                vs.world_vertex = mul(unity_ObjectToWorld, vertex);            
                vs.normal = UnityObjectToWorldNormal(-normal);
                return vs;
            }
        
            structurePS frag (structureVS vs)
            {
                //clip(distance(_SpherePos.xyz, vs.local_vertex)-0.5);
                structurePS ps;
                float3 normalDirection = normalize(vs.normal);
                half3 specular;
                half specularMonochrome;
                half3 diffuseColor = DiffuseAndSpecularFromMetallic(half3(1,0,0), 0, specular, specularMonochrome );
                ps.albedo = half4( diffuseColor, 1.0 );
                ps.specular = half4( specular, 0.7 );
                ps.normal = half4( normalDirection * 0.5 + 0.5, 1.0 );
                ps.emission = half4(0,0,0,1);
                float3 dir = normalize(_WorldSpaceCameraPos - vs.world_vertex);
                float3 centerPos = mul(unity_ObjectToWorld, float4(0,0,0,1));
                float3 spherePos = centerPos+_SpherePos.xyz;
                float intersectDist = raySphereIntersect(
                                    vs.world_vertex,
                                    dir,
                                    spherePos,
                                    0.5);
                intersectDist = max(intersectDist, 0);
              
                float3 norm = normalize(spherePos - vs.world_vertex);
                ps.normal = half4( norm * 0.5 + 0.5, 1.0 );

                ps.depth = CalcDepth(vs.world_vertex+dir*intersectDist);

                #ifndef UNITY_HDR_ON
                    ps.emission.rgb = exp2(-ps.emission.rgb);
                #endif
                return ps;
            }
            ENDCG
        }
    }
    FallBack "Diffuse"
}

The stencil buffer would be what I would usually use for something like this, but just reversing the ZTest should also do the trick in this case.

So just ZTest GEqual in step b. That actually also removes the need to do any per pixel clipping in step a.

Thanks for the reply! Unfortunately reversing ZTest doesnt really help here, cause then the cutout part will be visible through sides that it shouldnt be visible through, like this for example:
3373650--264612--problemRev.png

And as i said, stencil buffer sadly is not an option since im on deferred.

Stencils are kind of the only way to do this, apart from actually cutting out geometry (or analytical shader solutions like cross section shaders which I’ll get back to).

There’s also nothing preventing you from using stencils with deferred. You just can’t use Surface Shaders that use stencils with deferred. Luckily deferred shaders, as long as they’re not using light maps, are stupendously simple. They can be summed up as “Read material’s textures, sample ambient lighting, done”. The reason why Unity disables stencils on Surface Shaders is because the stencil is used by deferred for light mask layers so things will show up unlit if you don’t re-render objects you’ve already rendered to reset the stencil back to the original state.

However there is one big problem with using Stencils with deferred. Unity doesn’t respect manual draw ordering. You can’t use the material queue or even Renderer.sortingOrder to control which order meshes are rendered in, and there’s no equivalent of DrawMeshNow in command buffers and the existing DrawMesh commands don’t retain their order. If the Queue is less than 2500 every object is sorted however Unity wants. This actually got worse in later versions of Unity 5.

So, back to that first comment I made. Analytical shader solutions!

The short version is you need to render out the faces of your object(s) with a shader that does clip() based on the distance to a point in space (ie: a sphere). To handle the interior of the sphere, you’ll actually be rendering the back faces of your object(s) and raytracing against the interior of the sphere. The hardest part here is outputting proper depth values.

Thanks for the reply! But isnt that exactly what im already doing? Or am i missing something?
Ill try to ditch the surface shader and try to use raw frag shader with stencil instead.

Doh, you’re right. I didn’t look at the shader code.

Stencils really are the only way to solve this, but it also appears that Unity has changed the deferred renderer once again. When I last played around with this a year or two ago I could force stencil ops using a vertex fragment shader. Now it looks like all significant stencil operations really are ignored now. The comparison function for example is explicitly stuck on Always, which removes all of it’s usefulness.

If you can get away with holes that don’t go all the way through the object you could do the sphere intersection test on the front faces. But I suspect you’re trying to do L4D style holes in enemies so that’s not an option.

The only solution I can think of that would work with an arbitrary mesh in deferred would require manually rendering out a depth pass before the deferred pass and using that to clamp the out depth. That’s cumbersome at best.

The other option would be render the character in deferred, with solid black for the back faces, then do the interior rendering as a separate set of forward passes. Not exactly efficient, but ultimately there’s no good solution here with Unity’s existing deferred rendering path.

1 Like

Yeah, tried the raw frag shader, and as you said, even with that, unity hardcodes the stencil values for you. So i had to resort to manually rendering a depth map for it. There is still a caveat here though - i cant do shadowcaster pass, cause i would need a custom depthmap per each shadowcasting light, which sounds like a really problematic thing to do. This causes the inner parts to not be able to drop any shadows, on self or otherwise, but i think its still looks decent as is, so ill probably just leave it like that.
Thanks for the help!

3380093--265271--Снимок.PNG

1 Like