Rendering deferred fog only where useful

Unity Engine
1

Hello,

The deferred fog, in our implementation, is currently a full screen quad pass that blends with the skybox for nice effects. But on most pixels, it actually does nothing since there is no fog effect to perform as pixels are too close. I’d like to use the depth buffer to avoid executing the fragment shader on those pixels.

For this purpose I started modifying the vertex shader so that the Z value corresponds to the start of the fog in clip space. I thought it would be: (fogStart - nearClip) / (farClip - nearClip). But it doesn’t work! I wonder if there is something involved due to perspective division, reverse Z or something.

I started poking around values just to see if the concept would work and it does. For a fog start of 300, a near clip of 5 and a far clip of 2500, FOV at 45°, a Z value of 0.0063 properly culls all the pixels just as if I had clipped in the fragment shader. But I have no idea how to come up with that value from the parameters that I know of (near clip, far clip, fog start)

5312085--534009--upload_2019-12-24_13-28-32.png
5312085--534009--upload_2019-12-24_13-28-32.png701×390 345 KB

In this screenshot, we see that the fragment shader is not executed on pixels too close. The green/orange is debugging values. Green shows where the fog computation is supposed to occur and red shows how much fog to apply.

Looking at the cameraProjection matrix in the frame debugger, I notice that it is not affected at all by near and far clip values. It’s quite an unconventional projection matrix. Further digging shows that the actual projection UNITY_MATRIX_P is actually called glstate_projection_matrix and it’s also quite weird. It looks like the usual A and B values of the matrix are A = near / (far - near) and B = near. What kind of perspective projection matrix is Unity using?

5312085--534012--upload_2019-12-24_13-32-58.png

Any idea how to compute a proper Z value for depth testing to work for fog?

2

You want to convert a view space depth to clip space depth? Mostly you just need to apply the UNITY_MATRIX_P and you’re done… correcting for clip space w.

// view space fog start position, x&y don’t matter, w must be 1
float4 fogStartView = float4(0,0, _FogStart, 1);

// apply projection transform to view space position to transform into clip space
float4 fogStartClip = mul(UNITY_MATRIX_P, fogStartView);

// do perspective divide (transform from clip to normalized device coordinates, aka NDC)
float fogStartNDCZ = fogStartClip.z / fogStartClip.w;

// calculate clip space position of vertex
o.pos = UnityObjectToClipPos(v.vertex);

// convert NDC z back to clip space using the vertex clip pos w.
o.pos.z = fogStartNDCZ * o.pos.w;
3

A bit of explanation about why the “real” projection matrix looks like it does. First, matrixes in c# are defined in OpenGL form, and Unity transforms that into whatever from is needed for the given platform. So that means if you’re using Windows, it’ll be transformed into a Direct3D projection matrix, but the unity_CameraProjection matrix is the original OpenGL form. In addition Unity uses a reversed Z depth when rendering with anything other than OpenGL, which requires inverting the projection matrix. Look up “reversed z depth” to find out why.

The funny thing is no platform except Android uses OpenGL by default anymore. Windows is D3D11. Linux is Vulkan. MacOS & iOS are both Metal. Android currently defaults to OpenGLES, but you can use Vulkan on most new phones from the last few years. In a few more years I suspect no platform that Unity supports will still use the original OpenGL projection matrix on the GPU.

This all stems from Unity originally being an OpenGL only engine btw.

4

Yes, that’s what I also thought initially but surprisingly it doesn’t work. This is a post process shader ran from a commandbuffer in BeforeImageEffectsOpaque so it might change things a bit. Here’s my code by the way:

Shader "Hidden/PostProcessing/DeferredFog"
{
    HLSLINCLUDE

        #pragma multi_compile __ FOG_LINEAR FOG_EXP FOG_EXP2
        #include "../StdLib.hlsl"
        #include "Fog.hlsl"

        TEXTURE2D_SAMPLER2D(_MainTex, sampler_MainTex);
        TEXTURE2D_SAMPLER2D(_CameraDepthTexture, sampler_CameraDepthTexture);

        #define SKYBOX_THREASHOLD_VALUE 0.9999

        struct AttributesFog
        {
            float4 vertex : POSITION;
            float4 texcoord : TEXCOORD0;
            float4 texcoord1 : TEXCOORD1;
        };

        struct VaryingsFogFade
        {
            VaryingsDefault deffault;
            float3 ray : TEXCOORD2;
        };

        CBUFFER_START(UnityPerFrame)
            float4x4 glstate_matrix_projection;
        CBUFFER_END

        VaryingsFogFade VertFogFade(AttributesFog v)
        {
            VaryingsFogFade o;

            float near = _ProjectionParams.y;

            AttributesDefault defAttribs;
            defAttribs.vertex = v.vertex;
            o.deffault = VertDefault(defAttribs);

#if FOG_LINEAR
            float4 projected = mul(glstate_matrix_projection, float4(0, 0, FOG_START, 1));
            o.deffault.vertex.z = projected.z / projected.w * o.deffault.vertex.w;
            //o.deffault.vertex.z = 0.0063;
#else
            o.deffault.vertex.z = 0;
#endif

            float tanHalfFovX = 1 / unity_CameraProjection[0][0];
            float tanHalfFovY = 1 / unity_CameraProjection[1][1];
            float3 right = unity_WorldToCamera[0].xyz * near * tanHalfFovX;
            float3 top = unity_WorldToCamera[1].xyz * near * tanHalfFovY;
            float2 corner = v.vertex.xy;
#if UNITY_UV_STARTS_AT_TOP
            corner.y = -corner.y;
#endif
            float3 origin = unity_WorldToCamera[2].xyz * near;
            o.ray = origin + corner.x * right + corner.y * top;

            float _SkyRotation = FOG_SKYBOX_ROTATION;
            o.ray = RotateAroundYAxis(o.ray, _SkyRotation);

            return o;
        }

        float4 Frag(VaryingsDefault i) : SV_Target
        {
            half4 color = SAMPLE_TEXTURE2D(_MainTex, sampler_MainTex, i.texcoordStereo);

            float depth = SAMPLE_DEPTH_TEXTURE(_CameraDepthTexture, sampler_CameraDepthTexture, i.texcoordStereo);
            depth = Linear01Depth(depth);
            float dist = ComputeFogDistance(depth);
            half fog = 1.0 - ComputeFog(dist);

            return lerp(color, _FogColor, fog);
        }

        float4 FragExcludeSkybox(VaryingsDefault i) : SV_Target
        {
            half4 color = SAMPLE_TEXTURE2D(_MainTex, sampler_MainTex, i.texcoordStereo);

            float depth = SAMPLE_DEPTH_TEXTURE(_CameraDepthTexture, sampler_CameraDepthTexture, i.texcoordStereo);
            depth = Linear01Depth(depth);
            float skybox = depth < SKYBOX_THREASHOLD_VALUE;
            float dist = ComputeFogDistance(depth);
            half fog = 1.0 - ComputeFog(dist);

            return lerp(color, _FogColor, fog * skybox);
        }

        float4 FragFadeToSkybox(VaryingsFogFade i) : SV_Target
        {
            half4 color = SAMPLE_TEXTURE2D(_MainTex, sampler_MainTex, i.deffault.texcoordStereo);

            float depth = SAMPLE_DEPTH_TEXTURE(_CameraDepthTexture, sampler_CameraDepthTexture, i.deffault.texcoordStereo);
            depth = Linear01Depth(depth);
            float dist = ComputeFogDistance(depth) - FOG_START;
            half fog = 1.0 - ComputeFog(dist);
            float skybox = depth > SKYBOX_THREASHOLD_VALUE;

            half factor = saturate(fog + skybox);
            //clip(factor - 0.001);

            return half4(factor, (factor > 0) * 0.2, 0, 1);

            // Look up the skybox color.
            half3 skyColor = texCUBE(_SkyCubemap, i.ray);
            skyColor *= _FogColor.rgb * _FogColor.a * 4.59479380; // _FogColor.a contains exposure

            // Lerp between source color to skybox color with fog amount.
            half4 sceneColor = SAMPLE_TEXTURE2D(_MainTex, sampler_MainTex, i.deffault.texcoordStereo);
            return lerp(sceneColor, half4(skyColor, 1), factor);
        }

    ENDHLSL

    SubShader
    {
        Cull Off
        ZWrite Off
        ZTest Less

        Pass
        {
            HLSLPROGRAM
                #pragma vertex VertDefault
                #pragma fragment Frag
            ENDHLSL
        }

        Pass
        {
            HLSLPROGRAM
                #pragma vertex VertDefault
                #pragma fragment FragExcludeSkybox
            ENDHLSL
        }

        Pass
        {
            HLSLPROGRAM
                #pragma vertex VertFogFade
                #pragma fragment FragFadeToSkybox
            ENDHLSL
        }
    }
}

Running the math manually shows the value ends up a lot greater than 0.0063. Hence why I started this thread.
EDIT: actually, it’s not greater, my memories were wrong. It’s less than 0!

5

Whoops, view space is negative Z forward on the GPU, an positive view space Z is guaranteed to behind the camera. Try using -_FogStart.

6

Wow, I tried many things but nothing so simple haha! Thank you, it works!