Environmental Occlusion and Depth in ARFoundation

[j0schm03]

In a portrait app, the environmentDepthTexture is rotated 90 degrees. Is there a way to automatically adjust this to portrait?

It’s my understanding that the values returned by the environmentDepthTexture are in meters, the distance from the camera to the pixel in the environmentDepthTexture. According to this comment:

@todds_unity

The pixel values are in meters.

We are noticing that when using getPixel on the environmentDepthTexture at a given location the values returned for the r value are only returning between 0 and 1. Is this the correct behavior? With the distance supposedly in meters we anticipated values over 1 meter. Or are the Color.r values returned by the environmentDepthTexture actually not the distance in meters and we then have to calculate the distance based on the clipping planes etc?

For example, doing the following never gives us a value over 1:

Color depthValue = m_OcclusionManager.environmentDepthTexture.GetPixel((int)touchPos.x,(int)touchPos.y);
Debug.Log(depthValue.r)

[Tomzai]

@j0schm03

In a portrait app, the environmentDepthTexture is rotated 90 degrees. Is there a way to automatically adjust this to portrait?

It’s my understanding that the values returned by the environmentDepthTexture are in meters, the distance from the camera to the pixel in the environmentDepthTexture. According to this comment:

We are noticing that when using getPixel on the environmentDepthTexture at a given location the values returned for the r value are only returning between 0 and 1. Is this the correct behavior? With the distance supposedly in meters we anticipated values over 1 meter. Or are the Color.r values returned by the environmentDepthTexture actually not the distance in meters and we then have to calculate the distance based on the clipping planes etc?

For example, doing the following never gives us a value over 1:

Color depthValue = m_OcclusionManager.environmentDepthTexture.GetPixel((int)touchPos.x,(int)touchPos.y);
Debug.Log(depthValue.r)

From what I can tell, the depth values are in meters but the format of the texture means that if you use GetPixel you get a value clamped to 1 meter max. To get full-range values what I ended up doing is blitting the depth texture into a render texture (format RenderTextureFormat.ARGBFloat) then render that into a new Texture2D, which I could then use GetPixel on. I’m sure there’s a more efficient way but it worked. I also use a shader when blitting the initial texture that rotates the texture based on Screen.orientation so I don’t have to handle that later on - the initial texture is always in ‘Landscape Right’ orientation.

[j0schm03]

@Tomzai

From what I can tell, the depth values are in meters but the format of the texture means that if you use GetPixel you get a value clamped to 1 meter max. To get full-range values what I ended up doing is blitting the depth texture into a render texture (format RenderTextureFormat.ARGBFloat) then render that into a new Texture2D, which I could then use GetPixel on. I’m sure there’s a more efficient way but it worked. I also use a shader when blitting the initial texture that rotates the texture based on Screen.orientation so I don’t have to handle that later on - the initial texture is always in ‘Landscape Right’ orientation.

UPDATE:
This worked perfectly, thank you again for your help! I’ve been stuck on this for a week and this finally got me over the hump. Thank you!

ORIGINAL POST:
Thank you so much for this information! I tried blitting into a rendertexture, but was using a different texture format. I will give that a try and see what results we get.

Any chance you’d be willing to share the shader you used for rotating? I’ve been using the shader provided with the DisplayDepthImage example, with some modifications. This may work with the change you recommended, but if not it would be awesome if you were able to share. If not, I definitely understand.

Thanks for your help!

[Tomzai]

No problem. This is the shader I’ve been using - pass in Screen.orientation in “_Orientation” -

Shader "Unlit/CopyShader"
{
    SubShader
    {
        Tags { "RenderType"="Opaque" }
        LOD 100
        ZTest Always
        Cull Off
        ZWrite Off
       
        Pass
        {
            Name "Unlit"
               
            HLSLPROGRAM
            #pragma vertex vert
            #pragma fragment frag
           
            #include "UnityCG.cginc"

            struct appdata
            {
                float4 vertex : POSITION;
                float2 uv : TEXCOORD0;
            };

            struct v2f
            {
                float2 uv : TEXCOORD0;
                float4 vertex : SV_POSITION;
            };
           
            Texture2D _MainTex;
            SamplerState sampler_MainTex;
            float4 _MainTex_ST;
            int _Orientation;

            v2f vert (appdata v)
            {
                v2f o;
                o.vertex = UnityObjectToClipPos(v.vertex);
               
                // Flip X
                o.uv = float2(1.0 - v.uv.x, v.uv.y);
               
                if (_Orientation == 1) {
                    // Portrait
                    o.uv = float2(1.0 - o.uv.y, o.uv.x);
                }
                else if (_Orientation == 3) {
                    // Landscape left
                    o.uv = float2(1.0 - o.uv.x, 1.0 - o.uv.y);
                }
               
                o.uv = TRANSFORM_TEX(o.uv, _MainTex);
                return o;
            }

            fixed4 frag (v2f i) : SV_Target
            {
                return  _MainTex.Sample(sampler_MainTex, i.uv);
            }
            ENDHLSL
        }
    }
}

[j0schm03]

@todds_unity

AROcclusionManager.descriptor.supportsEnvironmentDepthImage

Would this also let you know if an iOS device has lidar? Trying to figure out how to detect the lidar hardware on iOS as well.

Thank you!

[DreamPower]

@j0schm03

Would this also let you know if an iOS device has lidar? Trying to figure out how to detect the lidar hardware on iOS as well.

Thank you!

Right now it would work, because only the LIDAR supports creating depth images in ARKit (or mesh reconstruction for that matter). But that’s not guaranteed for the future, it’s possible that the next ARKit could add depth sensing and mesh reconstruction to the normal camera for older devices.

Another option is to check for the device model, and check against devices you know have a LIDAR. For that you would parse the string returned from SystemInfo.deviceModel.

[j0schm03]

@DreamPower

Right now it would work, because only the LIDAR supports creating depth images in ARKit (or mesh reconstruction for that matter). But that’s not guaranteed for the future, it’s possible that the next ARKit could add depth sensing and mesh reconstruction to the normal camera for older devices.

Another option is to check for the device model, and check against devices you know have a LIDAR. For that you would parse the string returned from SystemInfo.deviceModel.

Thanks!

[prono82_develop]

@todds_unity

iOS beta 1 and beta 2 had an issue with the interaction between the human stencil texture and the environment depth texture.

iOS beta 3 resolved this issue. Do note that you will need the 4.1.0-preview.6 set of AR Foundation packages to work with iOS 14 beta 3.

Does the environment depth features of AR Foundation 4.1.0-preview.7 work with the latest iPadOS 14 beta 7 and Xcode 12.0 beta 6?
I’m worried that similar above problems will occur.

[serjightu]

If I use ARKore Deptx API in my project. I cannot see the entire 3D model. I see only part of the 3D model and the rest is hidden. How to make me see the entire 3D model completely?

[unity_8OaaYiLvKPqJSg]

@Tomzai

From what I can tell, the depth values are in meters but the format of the texture means that if you use GetPixel you get a value clamped to 1 meter max. To get full-range values what I ended up doing is blitting the depth texture into a render texture (format RenderTextureFormat.ARGBFloat) then render that into a new Texture2D, which I could then use GetPixel on. I’m sure there’s a more efficient way but it worked. I also use a shader when blitting the initial texture that rotates the texture based on Screen.orientation so I don’t have to handle that later on - the initial texture is always in ‘Landscape Right’ orientation.

Hello ! I’ve started testing the Depth API, but I have the same problem as j0schm03, could’nt retrieve depth values.
I tried to understand the tagged post, but I’m not sure how you did it.

I tried to Convert the Texture2D using ARGBFloat and GetPixels on the converted texture, but I’m still getting values between 0 and 1.

What I am doing wrong ?

Thanks in advance !

[todds_unity]

Use the AROcclusionManager.TryAcquireEnvironmentDepthCpuImage() to get a XRCpuImage of the environment depth texture.

[thorikawa1]

On iOS, does environmental occlusion automatically consider the confidence map provided by ARKit? If so can is there any way to set the threshold of that?

[todds_unity]

@thorikawa1

On iOS, does environmental occlusion automatically consider the confidence map provided by ARKit? If so can is there any way to set the threshold of that?

The AR Foundation background shader does not consider the confidence map.

You would need to have your own custom shader to achieve this behavior.

[dimitris_baud]

For those spending time trying to understand why Depth is not working for them on the iPad Pro Lidar / ARKit / iOS:

It seems to be broken on AR Foundation 4.1.0-preview9 and AR Foundation 4.1.0-preview10.

It works on AR Foundation 4.1.0-preview5.

Maybe it works on some in-between versions too.

[dimitris_baud]

Is anyone else having issues with this specifically on iOS (and iPadOS) 14.0.1?

[KyryloKuzyk]

@dimitris_baud

For those spending time trying to understand why Depth is not working for them on the iPad Pro Lidar / ARKit / iOS:

It seems to be broken on AR Foundation 4.1.0-preview9 and AR Foundation 4.1.0-preview10.

It works on AR Foundation 4.1.0-preview5.

Maybe it works on some in-between versions too.

Just tested this scene in all these setups and everything seems to be working fine:
iPad running iOS 14.0.1
Unity 2019.4.12/2020.1.8
AR Foundation 4.1.0-preview.7/4.1.0-preview.9/4.1.0-preview.10
macOS 10.15.5

Could you please share what exactly is not working and describe your setup? Have you tried the DepthImages scene from samples?

[MassiveTchnologies]

For some reason human occlusion doesn’t work on iOS 14.0 and iPhone 11 Pro Max. That’s very strange.

[tdmowrer]

@MassiveTchnologies

For some reason human occlusion doesn’t work on iOS 14.0 and iPhone 11 Pro Max. That’s very strange.

This feature works on my iPhone 11 Pro running iOS 14.0.1 and ARFoundation 4.1.0-preview.11. Can you post the exact versions of iOS and ARFoundation you are using?

[MassiveTchnologies]

I tried again on iPhone 11 Pro Max with ARFoundation 4.1.0-preview.11 and iOS 14.1, human occlusion still doesn’t work, could be an issue with the project / package / temp files. Human Occlusion does work on iPad Pro (with LiDAR) running iOS 14.1 though, same exact build.

[tdmowrer]

@MassiveTchnologies

I tried again on iPhone 11 Pro Max with ARFoundation 4.1.0-preview.11 and iOS 14.1, human occlusion still doesn’t work, could be an issue with the project / package / temp files. Human Occlusion does work on iPad Pro (with LiDAR) running iOS 14.1 though, same exact build.

The two devices have different capabilities, and depending on what other features you have requested, the iPhone 11 could be choosing a different configuration. In a development build, there is debug output (look in the Xcode console) that tells you what features were requested, which are enabled, and which could not be satisfied. This should help you identify the issue.