AR Raycast from point on XRCpuImage?

[vl4dimir]

I’m processing an XRCpuImage using OpenCV and detecting a number of points of interest on its texture. Then I want to take those points and perform camera.ScreenPointToRay(…) raycasts through them, but I’m trying to figure out the exact transform from XRCpuImage’s coordinate space back into screen coordinates (as the phone camera’s aspect ratio is rarely exactly the same as the screen’s).

Is there a standard way to do this, perhaps using ARCameraManager or ARCameraBackground? I can cook up a transform myself but I’m worried that it might be slightly off on some phones, as I don’t know exactly what algorithm ARCameraBackground uses to stretch the camera texture across the screen. I’m guessing it’s “stretch to fill”?

[KyryloKuzyk]

The ARCameraFrameEventArgs.displayMatrix is what you’re probably looking for.
Here is a more detailed answer:

ArBackgroundCamera & raw missalignment

I have a suggestion for Q1. Recently, I played around with ARCameraFrameEventArgs.displayMatrix. ARKitBackground.shader uses the displayMatrix to remap the texture coordinates: // Remap the texture coordinates based on the device rotation. float2 texcoord = mul(float3(v.texcoord, 1.0f), _UnityDisplayTransform).xy; But when I tried to multiply this matrix by a vector (bound to [0, 1]), the result was NOT bound by [0, 1]. It seems like this matrix contains not only the rotation information bu…

[vl4dimir]

Cheers, I’ll look into that!