Basic understanding of LiDAR support within ARFoundation.

Unity Engine
, ,
1

TLDR: When using LiDAR iPhone or iPad, is the sensor input from LiDAR factored in when generating feature points? Which „Trackable“ in ARFoundation represents the LiDAR Mesh? Is it TrackableType.Depth? Is it TrackableType.Face? (Home Page. | AR Subsystems | 4.2.10

Asked differently: ARKit and ARCore use some form of feature tracking algorithm, for that they mostly use visual clues but also probably weigh in other sensors like gyro. In that sense, LiDAR is just one more sensor in the array of available sensors that might be factored in for the slam algorithm.

However, when I build a simple ARFoundation FeaturePointCloud Visualizer Scene, iPhone XS and iPad Pro (with LiDAR) generate the exact same feature points. That’s indicating to me, that the feature point detection algorithm does not factor in LiDAR data at all.

Is my observation correct? If so, why would ARFoundation not integration LiDAR data? Is it not yet implemented due to differences between ARCore and ARKit, or does even ARKit not use LiDAR Data for feature tracking?

A usecase for factoring in LiDAR data is for example when the device looks at a white wall or similiar homogenous area. Regular visual feature tracking algorithms are unable to detect any features in that scenario. However LiDAR devices might generate featurepoints even there. I don’t see why ARFoundation (or ARKit) wouldn’t use the sensors, if they are available.

Also basically everything in ARFoundation, that is located and tracked in 3d Space is represented by a „Trackable“ (eg. Feature points, Planes, Images, Faces …). Which „Trackable“ in ARFoundation represents the LiDAR Mesh? Is it TrackableType.Depth? Is it TrackableType.Face? (Enum TrackableType | AR Subsystems | 4.2.10)

2

Hello there! Please refer to the meshing samples in GitHub - Unity-Technologies/arfoundation-samples: Example content for Unity projects based on AR Foundation.

While on a supported device, the trackables generated here are using the lidar sensor.

3

Thank you very much for your answer Treyk-47! Much appreciated!

I read the meshing sample, build several apps based on that and think to have at least a basic understanding of working with ARFoundation for LiDAR.
Still this one question bugs me, which I probably formulated way to complicated above:

Why dont we have a trackable type for the LiDAR Mesh? I dont see any way of using the ARRaycastManager to raycast against LiDAR Mesh. I allways work around that by using regular physics raycast.

The second thing I dont quite get is why the ARPlaneManager uses LiDAR to generate planes quicker (especially on homogoneus surfaces) but the ARPointCloudManager doesnt seem to take LiDAR into account at all. I think it would be logical to use LiDAR on the ARPointCloudManager whenever the user tries to find featurePoints on homogenous surfaces.

4

ARKit uses ARRaycastQuery to raycast against ARRaycastTarget which doesn’t expose a type for LIDAR mesh.