Getting access to the ARKit ARFrame pointer with ARFoundation

Unity Engine
,
1

As I understand, the official way to do computer vision with ARFoundation is to use the mechanisms described in “Accessing the Camera Image on the CPU”, but when using CoreML it’s much more convenient to pass the ARFrame captured image to a Vision request. I used to do this with the ARKit plug-in by passing the
UnityARSessionNativeInterface.GetNativeFramePtr() to my Objective-C plug-in, which worked great.

With ARFoundation, there is no obvious way to get access to this ARFrame pointer, but it’s still defined in the plug-in, so I tried accessing it via reflection like this :

var arKitApiClass = typeof(UnityEngine.XR.ARKit.ARWorldMap).Assembly.GetType("UnityEngine.XR.ARKit.Api");
var getNativeFramePtrMethod = arKitApiClass.GetMethod("UnityARKit_getNativeFramePtr", BindingFlags.NonPublic | BindingFlags.Static);

This works, and I can invoke it statically to get access to the pointer, but when I try to use it from Objective-C (by making a bridged-cast back to an ARFrame*), it dies with a BAD_ACCESS exception.
I assume that this is because the ARFrame was created from protected memory and I don’t have access to it…

Is there a way around this? It would be hugely convenient to have access to the ARKit session objects from Objective-C or Swift plug-ins.

Thanks!

2

You want XRCameraExtensions.GetNativePtr. The reason it didn’t work is that this isn’t the raw ARFrame, but rather a pointer to a struct of pointers. We’ll be releasing a C header soon which will let you interpret them properly, but for now you just need to know that each type of native ptr is a pointer to a struct whose first member is an int and the second is the native pointer you want. For example, the native pointer you get on iOS points to a struct similar to

struct NativeCameraPtr {
    int version;
    ARFrame* frame;
};

However, I wouldn’t recommend using the ARFrame directly on iOS, since there is nothing retaining this pointer and ARKit has its own frame rate independent of Unity, which means the pointer may become invalidated at any point (there are stronger guarantees on ARCore, so this isn’t as much of an issue there). It might be better to get the ARSession pointer, and use the currentFrame from that. Again, ARFoundation gives you a pointer to a struct of version and native pointer pair.

2 Likes
3

Awesome, that worked!
For now I’m just defining the struct like you posted, I’ll update with the official header once it’s out.

Thanks for the help!

4

Hi again,

I’m trying this method in another project just now, and XRSessionSubsystem.GetNativePtr() is returning null on iOS within a working ARKit session…
Is there anything that would cause the register hook not to hook itself? I see that the ARKit module uses [RuntimeInitializeOnLoadMethod] to register its GetNativePtr() method, what could cause this method not to run?

Thanks for any info! It’s pretty hard to debug this on-device since XCode is stepping through delegates and method pointers, and it’s nearly impossible to follow.

Edit : I just found out that calling GetNativePtr() ends up in the DefaultGetNativePtr() implementation on device, which returns 0. So that means the register failed or never ran?

5

Update on my issue, it looks like just touching the UnityEngine.XR.ARKit assembly by doing a typeof() on any type inside it will load the assembly and register the method, which makes GetNativePtr() work as expected. It’s surprising that it wasn’t loaded otherwise… I did have tracking working with points and planes on iOS.

6

That sounds like an issue with IL2CPP stripping an extension method. That can be addressed by adding a link.xml, but the latest ARKit package should prompt to do this for you. What versions of the packages do you have?

7

I’ve used the method of grabbing the ARSession successfully with ARKit and am trying to setup a similar pipeline with Android. With regards to ARCore, it seems like we can call GetNativePtr for the XRSessionSubsystem as well, does it point to something equivalent to the ARSession? Alternatively is there a proper way to access the current ARCore Frame through ARFoundation?

Thanks!

8

Yes you can do the same trick on Android. It’s a similar struct which stores a pointer to the ArSession.

The frame is also available. I believe it’s the one on the camera subsystem.

1 Like
9

Hi! Is this broken in the latest ARFoundation ? I can’t seem to find XRSessionSubsystem.GetNativePtr() since ARExtensions are deprecated and XRSessionSubsystem.nativePtr returns BAD_EXCEPTION error when I try to use it in Objective C. I made sure to disable stripping for XR.ARKit.

Any help is appreciated.

10

EDIT: Seems that the pointer is always returning nil in objective C. even though the AR session is running.

11

See Extending AR Foundation | AR Foundation | 3.0.1

12

I’m trying to access the cvPixelBuffer from ar foundation to pass it over for some coreML stuff. Where do i get the UnityEngine.XR.ARKit.ARKitSessionSubsystem object? Is it even the correct class to use?

13

To get access to the ARKitSessionSubsystem, downcast the XRSessionSubsystem:

ARKitSessionSubsystem arkitSessionSubsystem = ((m_ARSession != null) && (m_ARSession.subsystem is ARKitSessionSubsystem)) ? (ARKitSessionSubsystem)m_ARSession.subsystem : null;

However, if you are trying to access the camera images, just follow the Accessing the Camera Image on the CPU documentaion.

1 Like
14

Hey, trying to get access to the Front TrueDepth data with ARFoundations, just wondering whats the best approach to getting that? I see the ARFrame has captureDepthData, but not sure which path to take to access that infomation. Any help would be great! Cheers

1 Like
15

I don’t see an option to fetch the ARframe from ARSession(Session) on Android. Can you please point through the right direction?
On iOS, I see native library offers currentFrame but on ARCore its not provided. In this case, how can I fetch the ARFrame native instance?