Deadlock in AssetBundleManager

Unity Engine
1

Some players of our game are running into what seems to be a deadlock in AssetBundleManager:

>    ntdll.dll!NtWaitForSingleObject()    Unknown
     KERNELBASE.dll!WaitForSingleObjectEx()    Unknown
     UnityPlayer.dll!Baselib_SystemSemaphore_Acquire()    Unknown
     UnityPlayer.dll!AssetBundleManager::AddAssetBundleLoadAssetOperation(class AssetBundleLoadAssetOperation *)    Unknown
     UnityPlayer.dll!AssetBundleLoadAssetOperation::AssetBundleLoadAssetOperation(struct MemLabelId,class AssetBundle &,class core::basic_string<char,class core::StringStorageDefault<char> > const &,class ScriptingSystemTypeObjectPtr,bool)    Unknown
     UnityPlayer.dll!AssetBundleLoadAssetOperation::LoadAsset(class AssetBundle &,class core::basic_string<char,class core::StringStorageDefault<char> > const &,class ScriptingSystemTypeObjectPtr,bool)    Unknown
     UnityPlayer.dll!AssetBundle::LoadAssetAsync_Internal(class core::basic_string<char,class core::StringStorageDefault<char> >,class ScriptingSystemTypeObjectPtr,class ScriptingExceptionPtr *)    Unknown
     UnityPlayer.dll!AssetBundle_CUSTOM_LoadAssetAsync_Internal(class ScriptingBackendNativeObjectPtrOpaque *,class ScriptingBackendNativeStringPtrOpaque *,class ScriptingBackendNativeObjectPtrOpaque *)    Unknown
     [Frames may be missing, no binary loaded for GameAssembly.dll]   
     GameAssembly.dll!00007ffbd9481a38()    Unknown

Not sure what other thread is holding the lock but could be this one

>    ntdll.dll!NtWaitForAlertByThreadId()    Unknown
     ntdll.dll!RtlpWaitOnAddressWithTimeout()    Unknown
     ntdll.dll!RtlpWaitOnAddress()    Unknown
     ntdll.dll!RtlpWaitOnCriticalSection()    Unknown
     ntdll.dll!RtlpEnterCriticalSectionContended()    Unknown
     ntdll.dll!RtlEnterCriticalSection()    Unknown
     UnityPlayer.dll!PersistentManager::GetSerializedTypesAndIdentifiers(struct dynamic_array<int,0> const &,struct dynamic_array<class Unity::Type const *,0> &,struct dynamic_array<struct SerializedObjectIdentifier,0> &)    Unknown
     UnityPlayer.dll!SortPreloadObjects(struct dynamic_array<int,0> &,bool)    Unknown
     UnityPlayer.dll!AssetBundleManager::CollectPreloadDataDependencies(class PPtr<class AssetBundle> &,struct dynamic_array<int,0> &,bool)    Unknown
     UnityPlayer.dll!AssetBundleLoadAssetOperation::Perform(void)    Unknown
     UnityPlayer.dll!PreloadManager::ProcessSingleOperation(void)    Unknown
     UnityPlayer.dll!PreloadManager::Run(void)    Unknown
     UnityPlayer.dll!PreloadManager::Run(void *)    Unknown
     UnityPlayer.dll!Thread::RunThreadWrapper(void *)    Unknown
     kernel32.dll!BaseThreadInitThunk()    Unknown
     ntdll.dll!RtlUserThreadStart()    Unknown

Unfortunately, we are unable to reproduce this locally but quite a few players are affected. Does anybody have an idea what we could do about this? We are on Unity 2019.4.15f1.

2

Is it normal that co-routines can run during a synchronous Bundle.LoadAsset call? This can cause some unexpected bugs because code is interrupted that shouldn’t be interruptible.

My guess is that Bundle.LoadAsset → Some Coroutine → Bundle.LoadAsset causes a deadlock but I still have to prove that.

     [Inline Frame] GameAssembly.dll!InterfaceFuncInvoker0<bool>::Invoke(unsigned int declaringInterface, Il2CppClass *) Line 195    C++
     GameAssembly.dll!SetupCoroutine_InvokeMoveNext_m9106BA4E8AE0E794B17F184F1021A53F1D071F31(Il2CppObject * ___enumerator0, __int64 ___returnValueAddress1, const MethodInfo * method) Line 27734    C++
     GameAssembly.dll!RuntimeInvoker_FalseVoid_t22962CB4C05B1D89B55A6E1139F0E87A90987017_RuntimeObject_Nullable_1_t0D03270832B3FFDDC0E7C2D89D4A0EA25376A1EB(void(*)() methodPointer, const MethodInfo * methodMetadata, void * obj, void * * args) Line 146437    C++
     GameAssembly.dll!il2cpp::vm::Runtime::Invoke(const MethodInfo * method, void * obj, void * * params, Il2CppException * * exc) Line 545    C++
     UnityPlayer.dll!scripting_method_invoke(class ScriptingMethodPtr,class ScriptingObjectPtr,struct ScriptingArguments &,class ScriptingExceptionPtr *,bool)    Unknown
     UnityPlayer.dll!ScriptingInvocation::Invoke(class ScriptingExceptionPtr *,bool)    Unknown
     UnityPlayer.dll!Coroutine::Run(bool *)    Unknown
     UnityPlayer.dll!AsyncOperation::InvokeCoroutine(void)    Unknown
     UnityPlayer.dll!UnityWebRequestAsyncOperation::InvokeCoroutine(void)    Unknown
     UnityPlayer.dll!UnityWebRequestProto<class UnityWebRequestTransport,class AtomicRefCounter,class RedirectHelper,class ResponseHelper,class DownloadHandler,class UploadHandler,class CertificateHandler,class HeaderHelper,class AsyncOperation>::Job_InvokeCoroutine(class AsyncOperation *)    Unknown
     UnityPlayer.dll!BackgroundJobQueue::ExecuteMainThreadJobs(void)    Unknown
     UnityPlayer.dll!Texture::VerifyFileTextureUploadCompletion(void)    Unknown
     UnityPlayer.dll!Texture2D::VerifyFileTextureUploadCompletion(void)    Unknown
     UnityPlayer.dll!Texture2D::AwakeFromLoad(enum AwakeFromLoadMode)    Unknown
     UnityPlayer.dll!AwakeFromLoadQueue::InvokePersistentManagerAwake(struct AwakeFromLoadQueue::Item *,unsigned int,enum AwakeFromLoadMode)    Unknown
     UnityPlayer.dll!AwakeFromLoadQueue::PersistentManagerAwakeFromLoad(void)    Unknown
     UnityPlayer.dll!PersistentManager::IntegrateAllThreadedObjects(void)    Unknown
     UnityPlayer.dll!PersistentManager::LoadAndIntegrateAllPreallocatedObjects(enum PersistentManager::LockFlags)    Unknown
     UnityPlayer.dll!PersistentManager::LoadObjects(int const *,int)    Unknown
     UnityPlayer.dll!ProcessAssetBundleEntries()    Unknown
     UnityPlayer.dll!LoadNamedObjectFromAssetBundle(class AssetBundle &,class core::basic_string<char,class core::StringStorageDefault<char> > const &,class ScriptingSystemTypeObjectPtr const &)    Unknown
     UnityPlayer.dll!AssetBundle::LoadAsset_Internal(class core::basic_string<char,class core::StringStorageDefault<char> >,class ScriptingSystemTypeObjectPtr,class ScriptingExceptionPtr *)    Unknown
     UnityPlayer.dll!AssetBundle_CUSTOM_LoadAsset_Internal(class ScriptingBackendNativeObjectPtrOpaque *,class ScriptingBackendNativeStringPtrOpaque *,class ScriptingBackendNativeObjectPtrOpaque *)    Unknown
     GameAssembly.dll!AssetBundle_LoadAsset_TisRuntimeObject_m3209F473EEC424474B1053E66A1F09574719CB0B_gshared(AssetBundle_tCE287BAB693894C3991CDD308B92A8C9C4BD4C78 * __this, String_t * ___name0, const MethodInfo * method) Line 66687    C++
3

We have similair problem. During sync load from asset bundle unity invokes UnityWebRequests completion callbacks that haven’t been called yet. Async operations competions during synchronous operation can cause any number of unexpected bugs.

It is called side effect and should be fixed asap.

4

Could be the same cause as this: Unity Issue Tracker - UnityWebRequest coroutine is running when CanvasUpdateRegistry.IsRebuildingGraphics and CanvasUpdateRegistry.IsRebuildingLayout