Various unreproducible crashes caused by asset bundles

[Ed-Marty]

We recently released a game on iOS and Android, and a bunch of crash reports have started coming in on iOS that appear to be due to problems with shaders in asset bundles and other asset bundle problems. We haven’t been able to reproduce this problem, so we aren’t sure what we can do about them other than do a new release with new builds of the asset bundles and hope that they work. We’re using Unity 2108.2.12f1.

Is there anything we can do to try to resolve this problem?

For reference, here are some of the places where it has crashed. All of these are internal to Unity apparently. There’s nothing specific to our game in the stack trace

Crashed: UnityPreload
EXC_BAD_ACCESS KERN_INVALID_ADDRESS 0x0000000000000000

__bzero + 28

SerializeTraits.h - Line 488
SerializeTraits<dynamic_array<unsigned char, 0ul> >::ResizeSTLStyleArray(dynamic_array<unsigned char, 0ul>&, int)

StreamedBinaryRead.h - Line 77
void StreamedBinaryRead::TransferSTLStyleArray<dynamic_array<unsigned char, 0ul> >(dynamic_array<unsigned char, 0ul>&, TransferMetaFlags)

SerializeTraits.h - Line 481
void PackedFloatVector::Transfer(StreamedBinaryRead&)

CompressedMesh.h - Line 155
void CompressedMesh::Transfer(StreamedBinaryRead&)

SerializeTraitsBase.h - Line 54
void Mesh::Transfer(StreamedBinaryRead&)

SerializedFile.cpp - Line 2164
SerializedFile::ReadObject(long long, ObjectCreationMode, bool, TypeTree const**, bool*, Object&)

PersistentManager.cpp - Line 2330
PersistentManager::ReadAndActivateObjectThreaded(int, SerializedObjectIdentifier const&, SerializedFile*, bool, bool, PersistentManager::LockFlags)

PersistentManager.cpp - Line 1785
PersistentManager::LoadObjectsThreaded(int const*, int, LoadProgress&, bool)

dynamic_array.h - Line 146
LoadSceneOperation::Perform()

PreloadManager.cpp - Line 281
PreloadManager::ProcessSingleOperation()

PreloadManager.cpp - Line 338
PreloadManager::Run()

PreloadManager.cpp - Line 187
PreloadManager::Run(void*)

Thread.cpp - Line 42
Thread::RunThreadWrapper(void*)

And a dozen similar ones.

Crashed: UnityPreload
EXC_BAD_ACCESS KERN_INVALID_ADDRESS 0x0000000000000028

StringStorageDefault.h - Line 31
std::__1::vector<ShaderLab::SerializedSubProgram::StructParameter, std::__1::allocatorShaderLab::SerializedSubProgram::StructParameter >::__append(unsigned long)

type_traits - Line 3615
void resize_trimmed<std::__1::vector<ShaderLab::SerializedSubProgram::StructParameter, std::__1::allocatorShaderLab::SerializedSubProgram::StructParameter > >(std::__1::vector<ShaderLab::SerializedSubProgram::StructParameter, std::__1::allocatorShaderLab::SerializedSubProgram::StructParameter >&, unsigned int)

vector - Line 1467
void StreamedBinaryRead::TransferSTLStyleArray<std::__1::vector<ShaderLab::SerializedSubProgram::StructParameter, std::__1::allocatorShaderLab::SerializedSubProgram::StructParameter > >(std::__1::vector<ShaderLab::SerializedSubProgram::StructParameter, std::__1::allocatorShaderLab::SerializedSubProgram::StructParameter >&, TransferMetaFlags)

SerializeTraits.h - Line 260
void ShaderLab::SerializedSubProgram::ConstantBuffer::Transfer(StreamedBinaryRead&)

iterator - Line 1190
void StreamedBinaryRead::TransferSTLStyleArray<std::__1::vector<ShaderLab::SerializedSubProgram::ConstantBuffer, std::__1::allocatorShaderLab::SerializedSubProgram::ConstantBuffer > >(std::__1::vector<ShaderLab::SerializedSubProgram::ConstantBuffer, std::__1::allocatorShaderLab::SerializedSubProgram::ConstantBuffer >&, TransferMetaFlags)

SerializeTraits.h - Line 260
void ShaderLab::SerializedSubProgram::Transfer(StreamedBinaryRead&)

iterator - Line 1190
void StreamedBinaryRead::TransferSTLStyleArray<std::__1::vector<ShaderLab::SerializedSubProgram, std::__1::allocatorShaderLab::SerializedSubProgram > >(std::__1::vector<ShaderLab::SerializedSubProgram, std::__1::allocatorShaderLab::SerializedSubProgram >&, TransferMetaFlags)

SerializeTraits.h - Line 260
void ShaderLab::SerializedPass::Transfer(StreamedBinaryRead&)

iterator - Line 1190
void StreamedBinaryRead::TransferSTLStyleArray<std::__1::vector<ShaderLab::SerializedPass, std::__1::allocatorShaderLab::SerializedPass > >(std::__1::vector<ShaderLab::SerializedPass, std::__1::allocatorShaderLab::SerializedPass >&, TransferMetaFlags)

SerializeTraits.h - Line 260
void ShaderLab::SerializedSubShader::Transfer(StreamedBinaryRead&)

iterator - Line 1190
void StreamedBinaryRead::TransferSTLStyleArray<std::__1::vector<ShaderLab::SerializedSubShader, std::__1::allocatorShaderLab::SerializedSubShader > >(std::__1::vector<ShaderLab::SerializedSubShader, std::__1::allocatorShaderLab::SerializedSubShader >&, TransferMetaFlags)

SerializeTraits.h - Line 260
void ShaderLab::SerializedShader::Transfer(StreamedBinaryRead&)

SerializeTraits.h - Line 480
void Shader::Transfer(StreamedBinaryRead&)

SerializedFile.cpp - Line 2164
SerializedFile::ReadObject(long long, ObjectCreationMode, bool, TypeTree const**, bool*, Object&)

PersistentManager.cpp - Line 2330
PersistentManager::ReadAndActivateObjectThreaded(int, SerializedObjectIdentifier const&, SerializedFile*, bool, bool, PersistentManager::LockFlags)

PersistentManager.cpp - Line 1785
PersistentManager::LoadObjectsThreaded(int const*, int, LoadProgress&, bool)

dynamic_array.h - Line 146
LoadSceneOperation::Perform()

PreloadManager.cpp - Line 281
PreloadManager::ProcessSingleOperation()

PreloadManager.cpp - Line 338
PreloadManager::Run()

PreloadManager.cpp - Line 187
PreloadManager::Run(void*)

Thread.cpp - Line 42
Thread::RunThreadWrapper(void*)

And a dozen similar ones

Crashed: com.apple.main-thread
EXC_BAD_ACCESS KERN_INVALID_ADDRESS 0x00000000b253c034
libsystem_platform.dylib
_platform_memmove + 352

CachedReader.cpp - Line 98
CachedReader::Read(void*, unsigned long)

StreamedBinaryRead.h - Line 88
void StreamedBinaryRead::TransferSTLStyleArray<core::basic_string<char, core::StringStorageDefault > >(core::basic_string<char, core::StringStorageDefault >&, TransferMetaFlags)

SerializeTraits.h - Line 237
void NamedObject::Transfer(StreamedBinaryRead&)

SpriteFrame.cpp - Line 134
void Sprite::Transfer(StreamedBinaryRead&)

SerializedFile.cpp - Line 2164
SerializedFile::ReadObject(long long, ObjectCreationMode, bool, TypeTree const**, bool*, Object&)

PersistentManager.cpp - Line 2330
PersistentManager::ReadAndActivateObjectThreaded(int, SerializedObjectIdentifier const&, SerializedFile*, bool, bool, PersistentManager::LockFlags)

PersistentManager.cpp - Line 1663
PersistentManager::ReadObjectThreaded(int, PersistentManager::LockFlags)

PersistentManager.cpp - Line 1390
PersistentManager::ReadObject(int, AwakeFromLoadMode)

AssetBundleUtility.cpp - Line 143
ForcePreload(AssetBundle&, dynamic_array<int, 0ul>, bool)

dynamic_array.h - Line 114
ProcessAssetBundleEntries(AssetBundle&, std::__1::pair<std::__1::__map_iterator<std::__1::__tree_iterator<std::__1::__value_type<core::basic_string<char, core::StringStorageDefault >, AssetBundle::AssetInfo>, std::__1::__tree_node<std::__1::__value_type<core::basic_string<char, core::StringStorageDefault >, AssetBundle::AssetInfo>, void*>, long> >, std::__1::__map_iterator<std::__1::__tree_iterator<std::__1::__value_type<core::basic_string<char, core::StringStorageDefault >, AssetBundle::AssetInfo>, std::__1::__tree_node<std::__1::__value_type<core::basic_string<char, core::StringStorageDefault >, AssetBundle::AssetInfo>, void>, long> > > const&, ScriptingSystemTypeObjectPtr, dynamic_array<Object, 0ul>&, bool)

dynamic_array.h - Line 471
LoadNamedObjectFromAssetBundle(AssetBundle&, core::basic_string<char, core::StringStorageDefault > const&, ScriptingSystemTypeObjectPtr)

AssetBundle.bindings.gen.cpp - Line 265
AssetBundle_CUSTOM_LoadAsset_Internal(Il2CppObject*, Il2CppString*, Il2CppObject*)

unsigned long long flatbuffers::HashFnv1a(char const*) + 3860516

unsigned long long flatbuffers::HashFnv1a(char const*) + 7811508

unsigned long long flatbuffers::HashFnv1a(char const*) + 21918336

Il2CppInvokerTable.cpp - Line 4336374932
RuntimeInvoker_Void_t1185182177_RuntimeObject_IntPtr_t(void ()(), MethodInfo const, void*, void**)

Runtime.cpp - Line 521
il2cpp::vm::Runtime::Invoke(MethodInfo const*, void*, void**, Il2CppException**)

ScriptingApi_Il2Cpp.cpp - Line 245
scripting_method_invoke(ScriptingMethodPtr, ScriptingObjectPtr, ScriptingArguments&, ScriptingExceptionPtr*, bool)

ScriptingInvocation.cpp - Line 341
ScriptingInvocation::Invoke(ScriptingExceptionPtr*, bool)

ScriptingInvocation.h - Line 71
Coroutine::InvokeMoveNext(ScriptingExceptionPtr*)

Coroutine.cpp - Line 257
Coroutine::Run(bool*)

MonoBehaviour.cpp - Line 742
MonoBehaviour::TryCreateAndRunCoroutine(ScriptingObjectPtr, ScriptingMethodPtr, Coroutine**)

MonoBehaviour.cpp - Line 760
MonoBehaviour::StartCoroutineManaged2(ScriptingObjectPtr)

unsigned long long flatbuffers::HashFnv1a(char const*) + 3882300

unsigned long long flatbuffers::HashFnv1a(char const*) + 26739188

unsigned long long flatbuffers::HashFnv1a(char const*) + 1768320

unsigned long long flatbuffers::HashFnv1a(char const*) + 1768580

unsigned long long flatbuffers::HashFnv1a(char const*) + 2599928

unsigned long long flatbuffers::HashFnv1a(char const*) + 2602580

unsigned long long flatbuffers::HashFnv1a(char const*) + 2037840

unsigned long long flatbuffers::HashFnv1a(char const*) + 21782776

unsigned long long flatbuffers::HashFnv1a(char const*) + 21782776

unsigned long long flatbuffers::HashFnv1a(char const*) + 26101624

unsigned long long flatbuffers::HashFnv1a(char const*) + 22342724

unsigned long long flatbuffers::HashFnv1a(char const*) + 22341516

unsigned long long flatbuffers::HashFnv1a(char const*) + 22339732

Il2CppInvokerTable.cpp - Line 99434
RuntimeInvoker_Void_t1185182177(void ()(), MethodInfo const, void*, void**)

Runtime.cpp - Line 521
il2cpp::vm::Runtime::Invoke(MethodInfo const*, void*, void**, Il2CppException**)

ScriptingApi_Il2Cpp.cpp - Line 245
scripting_method_invoke(ScriptingMethodPtr, ScriptingObjectPtr, ScriptingArguments&, ScriptingExceptionPtr*, bool)

ScriptingInvocation.cpp - Line 341
ScriptingInvocation::Invoke(ScriptingExceptionPtr*, bool)

MonoBehaviour.cpp - Line 571
MonoBehaviour::CallUpdateMethod(int)

Behaviour.cpp - Line 176
void BaseBehaviourManager::CommonUpdate()

PlayerLoop.cpp - Line 340
ExecutePlayerLoop(NativePlayerLoopSystem*)

PlayerLoop.cpp - Line 357
ExecutePlayerLoop(NativePlayerLoopSystem*)

RecursionLimit.h - Line 45
PlayerLoop()

LibEntryPoint.mm - Line 299
UnityPlayerLoopImpl(bool)

UnityAppController+Rendering.mm - Line 280
UnityRepaint

UnityAppController+Rendering.mm - Line 77
-[UnityAppController(Rendering) repaintDisplayLink]
QuartzCore
CA:isplay:isplayLink::dispatch_items(unsigned long long, unsigned long long, unsigned long long) + 632

And again, a dozen similar ones.

[Ed-Marty]

I am seeing many more of these now. Still no idea how to track this down. Any ideas?

[dirty-rectangle]

@Ed-Marty Did you find a fix for this issue? we seeing similar crash callstacks, for example

iOS Unity 2018.4.0f1

Code Type: ARM-64
Exception Type: SIGSEGV
Exception Codes: SEGV_ACCERR at 0x0

Thread 29 Crashed:
0 libsystem_platform.dylib 0x00000001b570251c 0x1b56fa000 + 34076
1 appname 0x00000001052f3778 0x104934000 + 10221432 std::__1::enable_if<(__is_forward_iteratorShaderLab::SerializedSubProgram::ConstantBuffer*::value) && (is_constructible<ShaderLab::SerializedSubProgram::ConstantBuffer, std::__1::iterator_traitsShaderLab::SerializedSubProgram::ConstantBuffer*::reference>::value), void>::type std::__1::vector<ShaderLab::SerializedSubProgram::ConstantBuffer, std::__1::allocatorShaderLab::SerializedSubProgram::ConstantBuffer >::assignShaderLab::SerializedSubProgram::ConstantBuffer*(ShaderLab::SerializedSubProgram::ConstantBuffer*, ShaderLab::SerializedSubProgram::ConstantBuffer*) (in appname) (vector:1415)
2 appname 0x00000001052f1a7c 0x104934000 + 10214012 void resize_trimmed<std::__1::vector<ShaderLab::SerializedSubShader, std::__1::allocatorShaderLab::SerializedSubShader > >(std::__1::vector<ShaderLab::SerializedSubShader, std::__1::allocatorShaderLab::SerializedSubShader >&, unsigned int) (in appname) (vector_utility.h:38)
3 appname 0x00000001052f1a24 0x104934000 + 10213924 void resize_trimmed<std::__1::vector<ShaderLab::SerializedSubShader, std::__1::allocatorShaderLab::SerializedSubShader > >(std::__1::vector<ShaderLab::SerializedSubShader, std::__1::allocatorShaderLab::SerializedSubShader >&, unsigned int) (in appname) (vector_utility.h:27)
4 appname 0x0000000105123aec 0x104934000 + 8321772 void SafeBinaryRead::TransferSTLStyleArray<dynamic_array<long long, 0ul> >(dynamic_array<long long, 0ul>&, TransferMetaFlags) (in appname) (SafeBinaryRead.h:162)
5 appname 0x0000000105124cbc 0x104934000 + 8326332 void Transfer_Blittable_ArrayField<SafeBinaryRead, ColorRGBA32>(SafeBinaryRead&, ArrayInfo&, StaticTransferFieldInfo const&) (in appname) (Blittable.h:22)
6 appname 0x0000000105124ee4 0x104934000 + 8326884 void Transfer_Blittable_ArrayField<SafeBinaryRead, BitField>(SafeBinaryRead&, ArrayInfo&, StaticTransferFieldInfo const&) (in appname) (Blittable.h:32)
7 appname 0x000000010505d700 0x104934000 + 7509760 void std::__1::__insertion_sort_3<std::__1::lessBuiltinResourceManager::Resource&, BuiltinResourceManager::Resource*>(BuiltinResourceManager::Resource*, BuiltinResourceManager::Resource*, std::__1::lessBuiltinResourceManager::Resource&) (in appname) (algorithm:3733)
8 appname 0x0000000104d270c8 0x104934000 + 4141256 std::__1::__split_buffer<std::__1::pair<int, ConstantString>, std::__1::allocator<std::__1::pair<int, ConstantString> >&>::~__split_buffer() (in appname) (__split_buffer:340)
9 appname 0x000000010505f78c 0x104934000 + 7518092 bool std::__1::__insertion_sort_incomplete<ResourceManager:ependency::Sorter&, ResourceManager:ependency*>(ResourceManager:ependency*, ResourceManager:ependency*, ResourceManager:ependency::Sorter&) (in appname) (algorithm:3791)
10 appname 0x000000010505f54c 0x104934000 + 7517516 void std::__1::__insertion_sort_3<ResourceManager:ependency::Sorter&, ResourceManager:ependency*>(ResourceManager:ependency*, ResourceManager:ependency*, ResourceManager:ependency::Sorter&) (in appname) (algorithm:3746)
11 appname 0x000000010505f500 0x104934000 + 7517440 void std::__1::__insertion_sort_3<ResourceManager:ependency::Sorter&, ResourceManager:ependency*>(ResourceManager:ependency*, ResourceManager:ependency*, ResourceManager:ependency::Sorter&) (in appname) (ResourceManager.h:191)
12 appname 0x000000010509dc4c 0x104934000 + 7773260 dynamic_array<short, 0ul>::assign(short const*, short const*) (in appname) (dynamic_array.h:263)
13 libsystem_pthread.dylib 0x00000001b57102c0 0x1b5705000 + 45760
14 libsystem_pthread.dylib 0x00000001b5710220 0x1b5705000 + 45600
15 libsystem_pthread.dylib 0x00000001b5713cdc 0x1b5705000 + 60636

[Aurimas-Cernius]

Looks like a recently fixed race condition in AssetBundle loading. Yes, it’s very hard to reproduce. Check the release notes for 2019.1 or 2019.2. It’s possible that not all supported versions with the fix have been released yet.