Time has passed since you had your problem. I had the same one recently and found a way around it :
My trackables not being able to recover from the limited trackingState was caused by a script that referenced too many heavy models (after some testing, it can be due to any kind of asset, not just models). So my solution was to load the models at runtime using Resources.LoadAsync<T>(resourcePath); (doesn’t need to be async but its to prevent a huge screenfreeze).
A warning though, don’t load your assets too soon or ARCore will stop being able to process the environment.
With that said, maybe your problem has nothing to do with mine but that was my “solution”. Even though this problem seems more like a bug than a wrong use of unity or ARCore to me.
Although its a late answer, i hope it can help.
