Iphone Depth Image for hand stencil does not line up with camera feed vs Ipad

[ROBYER1]

When testing the AR Foundation Samples repo on an ipad and iphone (as well as using the depth images in our own apps), we noticed that the Hand stencil never lines up with the camera feed in AR properly on Iphones, but on Ipad the hand stencil matches the camera feed perfectly.

I have made a video of this below, you can easily see this if you overlay the hand stencil texture with the AR Camera background itself or if you simply pay attention to where my fingers are near the edges of the screen, the stencil is quite far off on Iphone and it is causing us issues in our apps.

https://drive.google.com/file/d/1WGngAOJJz9dz-FjtnCaweb78nPT8ulYW/view?usp=sharing

Github Bug: [Bug] Iphone Depth Image for hand stencil does not line up with camera feed vs Ipad · Issue #796 · Unity-Technologies/arfoundation-samples · GitHub

[KyryloKuzyk]

I’m almost sure this is related:

ArBackgroundCamera & raw missalignment

Hello, I have some ARFoundation questions, despite reading extensively, I have missed the information if it is in the forum already. Q1 : I have noticed that on an Ipad pro2020, the ARCameraBackground image and the Raw CameraImage are not aligned. There is a stretch and crop factor once we superimpose the raw. See here : [image]Pasteboard - Uploaded Image [Unity 2020.1.9, ARfoundation 4.1.0p12, ARkit 4.1.0p12] The question is how to match a pixel in viewport or screen coordinates in Unity…

ARFoundation Camera Intrinsic Params

I think what you’re looking for is displayMatrix. ARKit and ARCore use it to display the camera image in correct orientation/crop. Here is a link that describes the anatomy of the displayMatrix.

[ROBYER1]

@KyryloKuzyk

I’m almost sure this is related:
ArBackgroundCamera & raw missalignment
ARFoundation Camera Intrinsic Params

Did you have any suggestions on a good workaround? I currently have a shader graph where I nab the AR Camera Background renderer texture using:

Graphics.Blit(null, targetRenderTexture, m_ARCameraBackground.material);

Then separately, using the DisplayDepthImage.cs script from ARF samples, I save out that onscreen hand stencil texture to the same shader graph and overlay them buuut… obviously things do not line up on iphone. It works fantastic on ipad which is annoying.

[KyryloKuzyk]

@ROBYER1_1

Did you have any suggestions on a good workaround? I currently have a shader graph where I nab the AR Camera Background renderer texture using:

You’ll have to replicate this line from the ARKitBackground.shader in Shader Graph:

// Remap the texture coordinates based on the device rotation.
float2 texcoord = mul(float3(v.texcoord, 1.0f), _UnityDisplayTransform).xy;

@ROBYER1_1

obviously things do not line up on iphone. It works fantastic on ipad which is annoying.

I assume that the iPad has a similar aspect ratio to the camera video, so the displacement is unnoticeable.

[ROBYER1]

@KyryloKuzyk

You’ll have to replicate this line from the ARKitBackground.shader in Shader Graph:

// Remap the texture coordinates based on the device rotation.
float2 texcoord = mul(float3(v.texcoord, 1.0f), _UnityDisplayTransform).xy;

I assume that the iPad has a similar aspect ratio to the camera video, so the displacement is no unnoticeable.

Thanks, am bogged down in new functionality work for this app I am working on but I will post back next week if I find a way to recreate that, I did see that line in the background camera shader but it is not very clear how to translate that from shader code to a shader graph

I have seen there is a node for the value but I have no idea what the values in a matrix 4x4 actually mean!
https://docs.unity3d.com/Packages/com.unity.shadergraph@6.9/manual/Matrix-4x4-Node.html

[ROBYER1]

As the AR Foundation team didn’t seem to have a good explanation of how to do this and advised me to do so, I am chasing this up more in this separate thread in the Shader Graph forum: How to replicate ARKitBackground.shader from AR Foundation in Shader Graph + displayMatrix 4x4?