Hello, I’d like to start a discussion about how the AR tracked images are referenced in the new AR Foundation system. I’m using Unity 2019.2.0b1 with AR preview packages installed (AR Extensions + Foundation + Subsystems and ARCore+ARKit plugins).
As far as I know there’s no documentation yet on this feature but I managed to create a simple scene with a static library of images that, once detected by the phone camera, are used as anchors for 3D AR content. Perfect. The problem is that my aim was to have a dynamic library of images. With “dynamic” I mean a library that is not included in the app build but it is dynamically downloaded from a remote server (through an AssetBundle) and injected in the ARTrackedImageManager. After many tests I came to the conclusion that this is not possible with the current AR Foundation implementation. First of all, the manager dies if it does not find a library on initialization. My workaround was using a placeholder library with just one image (an empty library is not considered valid) and then replacing it at runtime:
var manager = gameObject.GetComponent<ARTrackedImageManager>();
manager.referenceLibrary = myDynamicLibrary;
This approach does not generate any error and looked fair enough to me but it just doesn’t work: the manager kept looking for the old library images.
After further investigation I came across the ARCoreImageTrackingProvider, which acts as a bridge between the Foundation and the low-level ARCore (i’m on Android) image tracking functionalities. From my understanding of this code, using a dynamic library for image tracking is just impossible: the provider expects to find a file .imgdb (which is the ARCore format for the images library) in the build itself. Unity creates this file just before building and ships it along the internal Android stuff (see ARCorePreprocessBuild).
Is there any solution for a dynamic approach? Maybe using Google ARCore implementation?
Today I tried to go low-level and directly call ARCore library method to update the AR images library. To make it compile I had to flag it as unsafe. I also had to recover the .imgdb file generated by the ARCore preprocessing routine and pass it to my function (second parameter):
public static unsafe void SetARCoreLibrary(XRReferenceImageLibrary lib, byte[] arcoreLib)
{
if (arcoreLib == null || arcoreLib.Length == 0)
{
throw new InvalidOperationException(string.Format(
"Failed to load image library '{0}' - file was empty!", lib.name));
}
var guids = new NativeArray<Guid>(lib.count, Allocator.Temp);
try
{
for (int i = 0; i < lib.count; ++i)
{
guids[i] = lib[i].guid;
}
fixed (byte* blob = arcoreLib)
{
UnityARCore_imageTracking_setDatabase(
blob,
arcoreLib.Length,
guids.GetUnsafePtr(),
guids.Length);
}
}
catch (Exception e)
{
Debug.LogError(string.Format("Error while loading '{0}': {1}", lib.name, e));
}
finally
{
guids.Dispose();
}
}
[DllImport("UnityARCore")]
static unsafe extern void UnityARCore_imageTracking_setDatabase(
void* databaseBytes, int databaseSize, void* sourceGuidBytes, int sourceGuidLength);
Aaaand it doesn’t work. I see no errors and the ARTrackerImageManager looks correctly configured (from the logs I can see that the number of images and their names is updated and correct)… but the system keeps detecting only the images of the old library.
Yeah, this is a nightmare.
My guess is that by replicating the [DllImport(“UnityARCore”)] attribute I’m actually calling another instance of that library… I’m not sure btw. I just know that a public keyword in the right place would have solved this problem…
Interesting, thanks for sharing your efforts! Although it looks discouraging at the moment, I’m hopeful there will be a way eventually. I am sure that Vuforia will be grateful for the delay, because the capability we’re looking for here is what continues to make them relevant. But the worst case is that Unity intends to restrict this feature, seeing it as a potential profit center and locks down “Cloud Reco” in the same way Vuforia does. That would suck.
Quoting: The reference image library can be set at runtime, but as long as the tracked image manager component is enabled, the reference image library must be non-null. The reference image library is an instance of the ScriptableObject XRReferenceImageLibrary. This object contains mostly Editor data. The actual library data (containing the image data) is provider-specific. Refer to your provider’s documentation for details.
I’m quite confused… why are we allowed to set the library at runtime if the actual library data is created at build time?
Other details in the ARCore plugin documentation: When building the Player for Android, each reference image library is used to generate a corresponding imgdb file, ARCore’s representation of the reference image library. These files are placed in your project’s StreamingAssets folder in a subdirectory called HiddenARCore to allow them to be accessible at runtime.
The StreamingAssets folder is read-only and this path is hardcoded in the provider’s code… any idea on how to override this behaviour?
I’ve been using EasyAR. It can pick up images directly from the StreamingAssets folder. I wanted to explore using ARFoundation but this is an issue for me also. Of course not running in the editor directly or using a working Remote also make me hold off on ARFoundation.
Guys I solved.
It works.
And I’m proud of my esoteric solution!
Basically I’m using reflection to call the private, static, unsafe method that updates the ARCore images database: ARCoreImageTrackingProvider.Provider.UnityARCore_imageTracking_setDatabase.
I know, this is so wrong but I guess it is the only way by now.
(WARNING, this C# code could make you cry).
#if UNITY_ANDROID
private static unsafe void ChangeARCoreImagesDatabase(XRReferenceImageLibrary library, byte[] arcoreDB)
{
if (arcoreDB == null || arcoreDB.Length == 0)
{
throw new InvalidOperationException(string.Format(
"Failed to load image library '{0}' - file was empty!", library.name));
}
var guids = new NativeArray<Guid>(library.count, Allocator.Temp);
try
{
for (int i = 0; i < library.count; ++i)
guids[i] = library[i].guid;
fixed (byte* blob = arcoreDB)
{
// Retrieve the ARCore image tracking provider by reflection
var provider = typeof(ARCoreImageTrackingProvider).GetNestedType("Provider", BindingFlags.NonPublic | BindingFlags.Instance);
// Destroy the current image tracking database
var destroy = provider.GetMethod("UnityARCore_imageTracking_destroy", BindingFlags.NonPublic | BindingFlags.Static);
destroy.Invoke(null, null);
// Set the image tracking database
var setDatabase = provider.GetMethod("UnityARCore_imageTracking_setDatabase", BindingFlags.NonPublic | BindingFlags.Static);
setDatabase.Invoke(null, new object[]
{
new IntPtr(blob),
arcoreDB.Length,
new IntPtr(guids.GetUnsafePtr()),
guids.Length
});
}
}
catch (Exception e)
{
Debug.LogError(string.Format("Error while loading '{0}': {1}", library.name, e));
}
finally
{
guids.Dispose();
}
}
#endif
TLDR: I found a way to change the ARCore tracked images database in an AR Foundation based application at runtime. The AR Foundation on Android expects to find this file in the app assets (it is automatically created and included at build time by Unity) making impossible to use downloaded or imported libraries. With my method you can swap between libraries easly, you just need to read the .imgdb file and pass the bytes to the function
Hey guys, at the moment I’m trying to do the same on ARKit… and oh my god.
Here’s the signature of the native call used to set the image library in iOS.
As you can see the parameter that matters is just the NAME of the library, string that it is internally used to find the file that Unity had created during the build processing… file that it won’t find, since the library has been downloaded/generated dynamically.
From my understanding, **[DllImport("__Internal")]** means that the actual implementation of that function is not in an external plugin but it is compiled in the engine itself… am I right? Is there a way to find the actual code?
Thanks for your efforts researching this and posting the code, I will need this soon and was planning on looking into it. Will post here if I have any new insights.
More on the internal statement here (but I’m sure you find this already):
Hi nilsdr,
Thank you, I hope for new insights aswell, I’ve no idea how to solve this
Yes I saw that page and I found the statically linked plugin that ARKit uses for the internal calls: “Library/PackageCache/com.unity.xr.arkit@2.0.1/Runtime/iOS/UnityARKit.a”. I tried to analyze its symbols (for the curious, you can do it using this unix command: nm UnityARKit.a) but I cannot see anything useful. I even tried to decompile it using Hopper
I admire your dedication, Lorenzo, but it’s clear that this capability (setting image targets at runtime) is not a design priority by Unity ARF team, at least not yet. It would be nice to have an official comment from someone on the AR Foundation team as to whether this omission is intentional, or if changes are coming that will make it possible.
I’ve managed to add reference image for ARKit, but only for now deprecated Unity ARKit Plugin. As far as I know, there is no way to add new reference image beside restarting native ARKit session. Check here for example.
So we have to change .a library to add new native call that restarting AR session with new reference images set.
Thank you Macoron!
Yes i saw that there was a way with the old ARKit plugin… but I was looking for a cleaner solution that didn’t break the Foundation packages, they should not be changed
Hey Lorenzo, thanks for trying to get dynamic image targets working in ARFoundation. It’s cool that you were able to get it working for the ARCore part of things (which makes sense, the official ARcore Unity plugin supports that feature, although the documentation is poor).
If you have any success in the future getting this working, please make another post here. I’ve started watching this thread.
PS - I did see that some people say they got at least a single dynamic image target working with ARKit 1.5 (not ARFoundation) via this thread. But since the Unity ARKit plugin is depreciated, we need a solution moving forward.
Hello! I was wondering if anyone has been able to find any solutions regarding dynamic image libraries on iOS? I have been working on a project for 6 months now that uses the dynamic loading of images, which I got working on the older versions of the ARKit and ARCore plugins. I’m stuck using those older, deprecated plugins until ARFoundation adds support for this feature, as dynamic image loading is an integral part of my project.
All the work of Lorenzo is very impressive, thanks for sharing it with us! It’s a shame that Unity aren’t adding dynamic image capability themselves, even though the feature is supported by both ARCore and ARKit.
Could you share the modified ARKit plugin code with us? That will help myself and others get this functionality working in at least the older versions of ARKit since it doesn’t look like it’s going to be brought over to ARFoundation.
Personally I’m trying to get the modified ARKit to support multiple dynamic image targets, added before the session starts.
Of course! There are 3 files that I modified, namely the UnityARCameraManager.cs, UnityARSessionNativeInterface.cs and ARSessionNative.mm. The first two are written in C#, the last on is written in Objective C.
This is the function that I added to ARSessionNative.mm. It allows you to start a session with multiple tracking images, which can be downloaded or retrieved from the device at runtime.
This is the code that I added to the UnityARSessionNativeInterface.cs. Don’t forget to import it with the [DllImport(“__Internal”)], else you won’t be able to call it.
And finally, this is the code that I added to the UnityARCameraManager. This is the function that you should call to start an ARKit session with your own, dynamically loaded tracking images. Simply provide the images, reference image sizes and the IDs for the images. The IDs are integers here, but they could also be reworked to be strings. You’ll be able to see which image has been detected by using the ARImageAnchor.referenceImageName function. This way you can have specific behavior when an image with a certain ID is tracked.
