World normal from scene depth

Unity Engine
,
1

Hey,
I am trying to compute world normals using only scene depth. I am using URP so there is no normal buffer available. In my shader I have access to the scene depth so I tried computing the normals like so:

    float vectorLength = 0.001;
    float2 y = float2(0, vectorLength);
    float2 x = float2(vectorLength,0);

    float depth1 = SampleDepth(_CameraDepthTexture, sampler_ScreenTextures_linear_clamp, uv + y).r;
    float depth2 = SampleDepth(_CameraDepthTexture, sampler_ScreenTextures_linear_clamp, uv + x).r;

    float3 p1 = float3(y, depth1 - depth);
    float3 p2 = float3(x, depth2 - depth);

    float3 normal = cross(p1, p2);
    normal.z = -normal.z;

    return normalize(normal);

However I realized this is creating more like screenspace normal than world normal. Anyone knows how I could do that?

Thanks!

2

The camera depth texture is by definition in screen space. To get a world space normal you need to first reconstruct the world position from the depth texture. If you search for “unity world position from depth texture” you’ll find this project or one of many forks others have made.

But to get a normal you need to sample the depth from multiple offset positions to get the normal. That’s something you already know. The method used in keijiro’s example to get the world position can’t be used for that, as the interpolated ray only works for the current pixel and not the neighbors you actually care about. Really you want to calculate the inverse view projection matrix in a c# script and pass it to the shaders. Like this project (which is doing it for both eyes so it supports VR, but that’s probably not something you need):

After that, you’ll probably notice really terrible artifacts on the edges of objects. These two articles go into methods of improving the values you get from the normal reconstruction, when possible.

Lastly, be aware that the normals you’re going to get are the normals of the actual geometry you’re using. They will not be smooth unless you have your mesh density way, way too high.

3

Hey, thanks for the reply!

So I should’ve mentioned this is not for a post process shader, it’s a regular transparent shader. I don’t really need super accurate stuff and I noticed about the normals being not smooth, but that will be good enough for my use case.

that being said, I tried using worlposition like you said.

what I used that function to convert the depth to world position (pixel depth is the view Z)

float3 ProjectedWorldPos(float3 worldPos, float sceneDepth, float pixelDepth)
{
    float3 pos = worldPos - _WorldSpaceCameraPos;
    float depthDiff = sceneDepth / pixelDepth;

    pos.xyz *= depthDiff;
    pos.xyz += _WorldSpaceCameraPos;

    return pos;
}

then I try to sample this multiple times with offsets like I previously did

    float2 uv = data.screenUV.xy;
    float vectorLength = 0.001;
    float2 y = float2(0, vectorLength);
    float2 x = float2(vectorLength, 0);

    float depth1 = SampleDepth(_CameraDepthTexture, sampler_ScreenTextures_linear_clamp, uv + y).r;
    float depth2 = SampleDepth(_CameraDepthTexture, sampler_ScreenTextures_linear_clamp, uv + x).r;

    float3 worldUV_c = ProjectedWorldPos(data.worldPosition, distortedData.a, data.pixelDepth);
    float3 worldUV_x = ProjectedWorldPos(data.worldPosition + float3(vectorLength,0,0), depth2, data.pixelDepth);
    float3 worldUV_z = ProjectedWorldPos(data.worldPosition + float3(0,0,vectorLength), depth1, data.pixelDepth);
   

    float3 p1 = worldUV_z - worldUV_c;
    float3 p2 = worldUV_x - worldUV_c;

    float3 normal = normalize(cross(p1, p2));

It’s a bit closer, but still not right… any idea about what is wrong?

4

Using the interpolated world position of the current fragment will let you get the world position of the depth sampled for that screen position. It won’t let you get the world position for other positions. The interpolated world position gets you the direction from the camera to that pixel position, and you don’t know the offset to get to other pixel positions. That arbitrary vectorLength variable is just that, completely arbitrary, and are totally different offsets when in world space vs UV space. “0.01” is a different distance in world space depending on far away the position is. And “0.01” in normalized screen space is different in the x and y depending on the aspect ratio.

Ultimately it doesn’t matter if you’re doing this as a post process or on an object, you still need to calculate the inverse view projection matrix in a c# script and pass it to the shaders to calculate the world position of pixel positions within the depth texture.

5

Maybe it would be helpful. C# script and post processing shader which shows proper world space positions, calculated from depth buffer. Add script to Main Camera and assign shader.

using UnityEngine;

public class DepthBufferToWorldSpace : MonoBehaviour
{
    public Shader DepthBufferToWorldSpaceShader;

    private Camera _MainCamera;   
    private Material _Material;

    void Start()
    {
        _Material = new Material(DepthBufferToWorldSpaceShader);
        _MainCamera = GetComponent<Camera>();
        _MainCamera.depthTextureMode = DepthTextureMode.Depth;
    }

    void OnRenderImage (RenderTexture source, RenderTexture destination)
    {
        Matrix4x4 m = GL.GetGPUProjectionMatrix(_MainCamera.projectionMatrix, false);
        m[2, 3] = m[3, 2] = 0.0f; m[3, 3] = 1.0f;
        Matrix4x4 ProjectionToWorld = Matrix4x4.Inverse(m * _MainCamera.worldToCameraMatrix) * Matrix4x4.TRS(new Vector3(0, 0, -m[2,2]), Quaternion.identity, Vector3.one);
        _Material.SetMatrix("unity_ProjectionToWorld", ProjectionToWorld);
        Graphics.Blit (source, destination, _Material);
    }
}

Shader "DepthBufferToWorldSpace"
{
    SubShader
    {
        Pass
        {
            CGPROGRAM
            #pragma vertex VSMain
            #pragma fragment PSMain
            #pragma target 5.0

            float4x4 unity_ProjectionToWorld;
            sampler2D _CameraDepthTexture;

            void VSMain (inout float4 vertex:POSITION, inout float2 uv:TEXCOORD0, out float3 direction:TEXCOORD1)
            {
                vertex = UnityObjectToClipPos(vertex);
                direction = mul(unity_ProjectionToWorld, float4(vertex.xy, 0.0, 1.0)) - _WorldSpaceCameraPos;
            }

            void PSMain (float4 vertex:POSITION, float2 uv:TEXCOORD0, float3 direction:TEXCOORD1, out float4 fragColor:SV_TARGET)
            {
                float depth = 1.0 / (_ZBufferParams.z * tex2D(_CameraDepthTexture, uv.xy) + _ZBufferParams.w);
                float3 worldspace = direction * depth + _WorldSpaceCameraPos;
                fragColor = float4(worldspace, 1.0);
            }

            ENDCG
        }
    }
}
2 Likes
6

That c# script is useful for getting a projection to world matrix which, if I’m not misunderstanding it, allows you to use the normalized screen position as the input. But the shader itself isn’t since it’s using the interpolated ray method which isn’t applicable to this use case for the reasons I mentioned above. Unless you’ve got some tricks for that @Przemyslaw_Zaworski . I guess it might be plausible with some derivative hackery but I remember trying that before and couldn’t get it to work reliably.

7

There now seems to be unity_CameraInvProjection built-in variable accessible from the shader which is Inverse of camera’s projection matrix. Is it the same as the one generated in the C# script?

8

That’s always existed, but it never actually matched the inverse of the “real” projection matrix. I’ve honestly never been entirely sure what it is. As best I can tell it’s the perspective matrix but with a totally different near & far plane. However I realized there was a way to merge the techniques from several of the above shaders to actually get something that works without any script.

6878867--803429--upload_2021-2-25_16-46-39.png
6878867--803429--upload_2021-2-25_16-46-39.png780Ă—630 122 KB

Shader "Unlit/DepthToWorldNormal"
{
    SubShader
    {
        Tags { "RenderType"="Transparent" "Queue"="Transparent" }
        LOD 100

        Pass
        {
            Cull Off
            ZWrite Off

            CGPROGRAM
            #pragma vertex vert
            #pragma fragment frag

            #include "UnityCG.cginc"

            struct appdata
            {
                float4 vertex : POSITION;
            };

            struct v2f
            {
                float4 pos : SV_POSITION;
            };

            sampler2D _MainTex;
            float4 _MainTex_ST;

            v2f vert (appdata v)
            {
                v2f o;
                o.pos = UnityObjectToClipPos(v.vertex);
                return o;
            }

            Texture2D _CameraDepthTexture;
            float4 _CameraDepthTexture_TexelSize;

            float3 rayFromScreenUV(in float2 uv, in float4x4 InvMatrix)
            {
              float x = uv.x * 2.0 - 1.0;
              float y = uv.y * 2.0 - 1.0;
              float4 position_s = float4(x, y, 1.0, 1.0);
              return mul(InvMatrix, position_s * _ProjectionParams.z);
            }

            float3 viewSpacePosAtPixelPosition(float2 pos)
            {
                float rawDepth = _CameraDepthTexture.Load(int3(pos, 0)).r;
                float2 uv = pos * _CameraDepthTexture_TexelSize.xy;
                float3 ray = rayFromScreenUV(uv, unity_CameraInvProjection);
                return ray * Linear01Depth(rawDepth);
            }

            fixed4 frag (v2f i) : SV_Target
            {
                float3 vpl = viewSpacePosAtPixelPosition(i.pos.xy + float2(-1, 0));
                float3 vpr = viewSpacePosAtPixelPosition(i.pos.xy + float2( 1, 0));
                float3 vpd = viewSpacePosAtPixelPosition(i.pos.xy + float2( 0,-1));
                float3 vpu = viewSpacePosAtPixelPosition(i.pos.xy + float2( 0, 1));

                float3 viewNormal = normalize(-cross(vpu - vpd, vpr - vpl));
                float3 WorldNormal = mul((float3x3)unity_MatrixInvV, viewNormal);

                // if needed, this will detect the sky
                // float rawDepth = _CameraDepthTexture.Load(int3(i.pos.xy, 0)).r;
                // if (rawDepth == 0.0)
                    // WorldNormal = float3(0,0,0);

                return WorldNormal.xyzz;
            }
            ENDCG
        }
    }
}

Basically I’m computing a ray direction like Keijiro & @Przemyslaw_Zaworski 's examples, but for each sample position rather than in the vertex shader.

However I can’t guarantee it’ll work on OpenGL platforms, but it should work everywhere else, including with the built in rendering paths!

5 Likes
9

It’s quite a bit more calculations that what I expected from the start, but I must say it works quite well and not having to rely on a script is a big plus for me!

Big thanks for the help!

Btw, you say it might not work on OpenGL, is it because of the _CameraDepthTexture.Load ?

10

Yep.

11

Minor improvement over the previous. This implements the “improved normal reconstruction” from one of the links above, and adds some comments. And should work with OpenGL.

Shader "WorldNormalFromDepthTexture"
{
    SubShader
    {
        Tags { "RenderType"="Transparent" "Queue"="Transparent" }
        LOD 100

        Pass
        {
            Cull Off
            ZWrite Off

            CGPROGRAM
            #pragma vertex vert
            #pragma fragment frag

            #include "UnityCG.cginc"

            struct appdata
            {
                float4 vertex : POSITION;
            };

            struct v2f
            {
                float4 pos : SV_POSITION;
            };

            v2f vert (appdata v)
            {
                v2f o;
                o.pos = UnityObjectToClipPos(v.vertex);
                return o;
            }

            UNITY_DECLARE_DEPTH_TEXTURE(_CameraDepthTexture);
            float4 _CameraDepthTexture_TexelSize;

            // inspired by keijiro's depth inverse projection
            // https://github.com/keijiro/DepthInverseProjection
            // constructs view space ray at the far clip plane from the vpos
            // then multiplies that ray by the linear 01 depth
            float3 viewSpacePosAtPixelPosition(float2 vpos)
            {
                float2 uv = vpos * _CameraDepthTexture_TexelSize.xy;
                float3 viewSpaceRay = mul(unity_CameraInvProjection, float4(uv * 2.0 - 1.0, 1.0, 1.0) * _ProjectionParams.z);
                float rawDepth = SAMPLE_DEPTH_TEXTURE_LOD(_CameraDepthTexture, float4(uv, 0.0, 0.0));
                return viewSpaceRay * Linear01Depth(rawDepth);
            }

            // inspired by János Turánszki's improved normal reconstruction technique
            // https://wickedengine.net/2019/09/22/improved-normal-reconstruction-from-depth/
            // this is a minor optimization over the original, using only 2 comparisons instead of 8
            // at the cost of two additional vector subtractions
            half4 frag (v2f i) : SV_Target
            {
                // get current pixel's view space position
                half3 viewSpacePos_c = viewSpacePosAtPixelPosition(i.pos.xy + float2( 0.0, 0.0));

                // if depth is at the far plane, then assume skybox
                // if (abs(viewSpacePos_c.z) >= _ProjectionParams.z)
                    // return 0;

                // get view space position at 1 pixel offsets in each major direction
                half3 viewSpacePos_l = viewSpacePosAtPixelPosition(i.pos.xy + float2(-1.0, 0.0));
                half3 viewSpacePos_r = viewSpacePosAtPixelPosition(i.pos.xy + float2( 1.0, 0.0));
                half3 viewSpacePos_d = viewSpacePosAtPixelPosition(i.pos.xy + float2( 0.0,-1.0));
                half3 viewSpacePos_u = viewSpacePosAtPixelPosition(i.pos.xy + float2( 0.0, 1.0));

                // get the difference between the current and each offset position
                half3 l = viewSpacePos_c - viewSpacePos_l;
                half3 r = viewSpacePos_r - viewSpacePos_c;
                half3 d = viewSpacePos_c - viewSpacePos_d;
                half3 u = viewSpacePos_u - viewSpacePos_c;

                // pick horizontal and vertical diff with the smallest z difference
                half3 h = abs(l.z) < abs(r.z) ? l : r;
                half3 v = abs(d.z) < abs(u.z) ? d : u;

                // get view space normal from the cross product of the two smallest offsets
                half3 viewNormal = normalize(cross(h, v));

                // transform normal from view space to world space
                half3 WorldNormal = mul((float3x3)unity_MatrixInvV, viewNormal);

                // visualize normal (assumes you're using linear space rendering)
                return half4(GammaToLinearSpace(WorldNormal.xyz * 0.5 + 0.5), 1.0);
            }
            ENDCG
        }
    }
}
5 Likes
12

Can this be reproduced using only ShaderGraph or ASE ?
I am working on a custom decal shader and I guess this would be the way to override the projector mesh’s vertex normals

13

Any Luck? I am also making a custom decal shadergraph, as the built-in URP decal projector is lacking in performance and features.

14

Sorry for reviving an old post but it seems Unity has a function called SampleSceneNormals but it doesn’t seem to do anything in shadergraph.

I tried with a custom function node:
8647485--1163292--upload_2022-12-9_3-34-33.png

But apparently its an undeclared identifier but it does exist so its a bit confusing.

They also have this node:

8647485--1163295--upload_2022-12-9_3-38-36.png

But this just gives fully black output when i linked it to the base color…none of this is documented anywhere but it seems like there is some built in scene normal texture and don’t require complex calculations on the depth texture but i cannot figure it out at all its frustrating!

15

Hi, it seems that the normal texture does not exist.

You can try enabling depth normal texture by adding SSAO or this empty renderer feature to the renderer feature list.