I am currently trying to create a new AR Plug-in, like ARCore or ARkit, using Unity XR SDK. From my understanding, Unity XR SDK should be used when creating a dynamic linking library used in AR plug-in package, but the official documentation from Unity provides limited information on this. I was wondering if anyone has any relevant experience to share or knows where I can find more information. It would be even better if there are specific examples I can refer to. Thank you.
Yes, the documentation is rather scarce.
Currently, three AR Foundation subsystems have to be written in C++: XRInputSystem, XRMeshSubsystem, and XRDisplaySubsystem. You only have to use the XR SDK if you wish to implement those in your AR provider. All other subsystems can be implemented from the C# side.
Also, supporting the XRInputSystem is only necessary if you wish to support Input Manager (Old). If you’re targeting only for Input System (New), then implementing XRInputSystem is not necessary.
You can find examples of XRInputSystem and XRDisplaySubsystem by requesting the XR SDK.
There are not many examples of XRMeshSubsystem, but this one helped me a lot: com.unity.xr.openxr@1.2.8. Also, I just found another one inside the com.unity.mars-ar-foundation-providers@1.4.1.
Unity XR SDK has a collection of C++ headers that you have to use to compile your native library. The hard part for me was finding a cross-compiler to compile the library for all target platforms.
XR SDK comes with Bee compiler, but I wasn’t able to get it running. Instead, I found a hidden gem inside Zig language. It can cross-compile from any architecture to any architecture, which is sick.
2 Likes
On the C# side, note that you will also need to register subsystem descriptors and implement an XR Loader. I wrote up the key steps here: Implement a provider | AR Foundation | 5.0.7
See the ARKit and/or ARCore plug-ins for examples. Specifically:
ARKitLoader.cs : This is the ARKit XRLoader implementation you can use as a reference when implementing your XRLoader
UnitySubsystemsManifest.json : This is how you tell Unity that you implemented Meshing and/or Input subsystems
ARKitPackageMetadata.cs : This is how you tell Unity your supported build targets
ARKitSettings.cs : The XRConfigurationData attribute is how you render settings under Project Settings > XR > <Your Provider>
1 Like
If you come across more specific questions feel free to open new threads in this forum
Thank you, this is really helpful.
My AR plug-in plans to implement face tracking and body tracking functions. It looks like I won’t need to use the XR SDK, but now I have a new question. How can I obtain the frames captured by the camera in order to perform calculations on the face detection algorithm located in my dynamic linking library, if I was not using any Unity XR SDK functions?
Thank you again for your reply. It helps a lot.
https://docs.unity3d.com/Packages/com.unity.xr.arfoundation@4.2/manual/cpu-camera-image.html
(I’m linking to the 4.2 version of the docs here because we have new Camera docs coming early next year to AR Foundation 5.0 and 5.1 preview that will break any links to those versions.)
Thank you for taking the time to respond promptly. It is quite helpful.
I know how to get image in C# now, but what I concerned is that how face tracking function in dynamic linking library get the camera image? Using ARCore for example:
On line 51, the “UnityARCore_faceTracking_TryGetFaceData” located in the ARCore dll is obtained, and on line 30, this function is used to get the related face mesh data.
However, this function does not get any camera image data, so how does the face tracking algorithm of ARCore calculate these face mesh data?
All I want to know is that how the functions in the ARCore dll get the camera image to calculate the result, are there any key points that I have missed?
Our ARCore DLL does not contain a face tracking implementation. Face tracking is implemented at the platform layer within ARCore itself (ie, Guia do desenvolvedor de rostos aumentados para Android | ARCore | Google for Developers). Google gives us tracking data but does not expose the camera image via the face tracking API.
To get the camera image for yourself at the application layer for your own tracking implementation, you must create an XRCpuImage following the steps in my post above.
Thank you! I know how to do it now. This is really helpful!
1 Like
Hi @KyryloKuzyk and @andyb-unity @andybiar
The link posted earlier to request XR SDK access is broken, and it seems no one at Unity knows how to access it. Any hints?
I’m very stuck trying to set custom view for single pass stereo. Thread here.
I don’t know the answer to this question. I’ve pinged some folks internally and will get back to you. Thanks for bringing this to my attention.
These header files are all you need to build your own native XR subsystem:
You need to compile a C++ native library with these headers, and you should be good to go. The XR SDK contained a couple of extra samples, but this post references two open-source samples that are helpful too. Also, my plugin implements XRInputSystem and XRMeshSubsystem, and comes with full source code inside.
1 Like
@KyryloKuzyk is technically correct, although what I’ve learned since last week is that we don’t officially support the XR SDK for third-party use anymore, which is why we have removed the signup link. Our preferred path is for runtimes to get data into Unity via OpenXR.
1 Like
@andyb-unity Interesting news. Perhaps this should be noted directly in the documentation so that others are not also mislead to believe they should start writing a XR Provider Plugin.
I am so not familiar with OpenXR, other than it is used for VR headsets. But in my case, where I’m just trying to achieve the equivalent of the Camera.SetStereoViewMatrix() (EDIT: for single pass stereo with custom matrices) in HDRP, do I understand correctly that writing a OpenXR provider, similar to Steam, is the only viable way fourth? I am not targeting a specific HW device. I’m not even targeting a headset – I want to output stereoscopic SBS images to 3D projectors and 3D displays.
I found a hack, but it involves misusing MockHMD and altering both SRP Core and HDRP. So not a solution I am happy with.
I think if you can live with the performance overhead the easiest way would be to just use 2 cameras and render half the vewport for each.
Otherwise there are hints of an API to override the matrices although this does not seem to be properly supported without modifying the Render Pipeline (XRSystem.SetLayoutOverride)
That is really bad news after working more than a year on fixing Unitys missing Quadbuffer Stereo support via the XR SDK. Does this mean that the API will stop working in the future or just that it isnt officially supported (as in getting support) anymore?
Thanks for pointing out, but yes, I need single pass stereo with custom matrices. That is the whole point. Hence the hack.
We are bound by semantic versioning, so we can’t break the XR SDK until the next major version of Unity at the earliest. I’m unaware of any plans to do so on any timescale at the moment. I was surprised to learn that we shut down the sign-up form.
If we do break the XR SDK sometime in the future, I would hope that this would be widely communicated with ample lead time (ie, years).
Your use case is very far outside the realm of AR Foundation. I would recommend creating a topic for this and tagging Graphics.
Glasses-free 3D displays (like the Samsung Odyssey 3D) are hitting the market this year, and this means developers will need single pass stereo with off-axis perspective views. The only solution I’ve seen so far uses two cameras (as does the Unity SDK for Odyssey 3D), which means performance is terrible – especially in HDRP where each camera is doing a ridiculous amount of work.
I have a research thread here: Custom single pass stereo matrices in HDRP. How?