ARFoundation - tracking loss : trackables don't recover from trackingState.limited

Target Platforms XR
, , ,
1

Hello.

I’m currently working on an AR project and i have a bug since a few days, i tried to fix it by myself but i’m running out of potential clues.

The project uses plane detection and is supposed to be used by untrained users. So i expect them to open android popups and hide the camera with their hands, etc… and so, they WILL provoke detected planes’s trackingStates to switch to “limited”.
ArFoundation anticipates these use cases by making those limited tracking trackables recover their position in world space as soon as the ARSession recalculates the environment based on feature points, if i understand well.

The problem is that my trackables with the limited trackingState are not replaced in the environment after tracking loss. I started a blank project to get it working, made a few tests in different situations : the environment is always automatically put back in place, except on my app. So i figured it could be due to one of my scripts. I disabled and deleted components and gameobjects in copies of my scene until i narrowed it down to a few leads.

It turns out the features gets working when i remove a certain ScriptableObject’s reference from a MonoBehaviour script instance. This reference contains prefabs needed to my simulation. Those might be heavy but i’ve had other similar references that didn’t break the feature.

I could try rearranging my SO, store those prefabs in another way or rewriting the script but since i don’t understand what causes the problem, i don’t know how to address it and i would like to prevent this problem from happening again when the project eventually gets larger without having to reorganize my entire workflow.

Did anyone have a similar problem ? And if yes, how did you solve it and/or find the reason why this is happening ?

Here is the structure of my scriptable :

using System;
using System.Collections.Generic;
using UnityEngine;

[CreateAssetMenu(fileName = "UIRenderingShelterTemplates", menuName = "Scriptables/UIRenderingShelterTemplates")]
public class UIRenderingShelterTemplates : ScriptableObject
{
    [Serializable]
    public struct UIRendModel
    {
        public string modelName;
        public List<GameObject> models;
    }

    public List<UIRendModel> modelTemplates;
}

I would gladly appreciate any contribution, thanks in advance.

2

Update :

I kept researching and confirmed the cause. To reproduce the problem :

  • Create a blank project or use the ar mobile template
  • Create a script with a simple List of GameObjects (public or field serialized)
  • Import a high poly model into your project and duplicate it several times (or import many different models) - i also tired with large images and it still bugs, so any large referencable file should work
  • Reference all copies of your model into the list from your script
  • Build the app on an android device
  • Detect some planes and scan the environment
  • Occlude the camera with your hand, lock your phone, do anything to make the trackables loose their “tracked” trackingState (visually it’s still in camera view even when moving the device)
  • Scan the environment again. ArCore can’t recognize it.

Do the same with fewer references to heavy assets and the last step will put the planes back in their original world position.

So, is there any way to prevent this issue ?

3

So i ended up using LoadResourcesAsync(). But if called in Start() or Awake(), the bug still occurs.

Though after loading the scene and letting the ARSession store the environment data into memory for like 10 secs, i can load my Assets without breaking the ability to put the trackables back in their place after camera occlusion or android popups.

This is not that great of a solution because i have to block almost every functionnality before the user scanned enough environment, then load the resources (it will take like 10 secs also) while keeping the user waiting. This adds approximatively 20/30 secs of just idling before the user can start to use the app.

I will keep doing this atm but if there is a way to load the scene along with the needed resources without breaking the ARSession i would appreciate learning about it.

4

I’m not sure about your ScriptableObject situation specifically, but we have noticed that ARCore has difficulty relocalizing anchors if the tracking is lost. (There was another thread about this recently: ARAnchor stuck with Limited trackingState - #12 by andyb-unity)

5

I don’t believe this is the case. AR Foundation reports the tracking state, but ARCore controls it.

6

Finally this had nothing to do with the scriptable, only the references in it. There was a list of dozens of heavy models loaded into memory. Turns out a simple Mono script with a list of referenced assets weighing a lot of space provokes tha same problem.

Ok, i don’t know which one to blame, my best guess from what i tested is that when the assets are loaded at runtime after some recognition of the environment, it is able to recognize it, but if loaded too soon, ARCore seems to stop either storing environment data or being able to process it. Maybe there is a way to allocate a fixed amount of memory to the ARCore system ? Or to prevent unity from steping on ARCore’s memory’s toes ?

7

It sounds like your app is heavily CPU-bound on the model loading task. This can affect the performance of ARCore (or any AR platform).

In general ARCore can have difficulty relocalizing anchors on some devices. Your app must handle the case where tracking is lost and never regained.

ARCore seems to stop either storing environment data or being able to process it

Neither Unity nor any program outside of ARCore itself has access to this level of internal information. Whether your theory is true or not, there’s no way to change the internal behavior of ARCore.

8

Yes, i have a lot of heavy models that can need to be loaded but only a few at the same time.

I had a warning modal to tell the user not to do stuff that could cause the issue but it’s not needed anymore, i used Resources.LoadAsync<T>(resourcePath); (instead of referencing all my prefabs in a list from a MonoBehaviour script instantiated in the scene) to prevent overloading the memory used for storing the models and i didn’t have the problem since then, environment recongition never breaks anymore afaik.