IL2CPP build high crash rate on il2cpp::os::FastMutex::Lock()

Unity Engine
,
1

We are currently using Unity 2019.2.5f1 and have used 2019.1.14f1, 2019.1.10f1 before. Regardless what Unity version we are using, our game has a high crash rate reported in Google Play Console and for many of the crashes Unity Cloud Diagnostics shows a crash stack trace like this:

Native StackTrace:

Thread 0 (crashed)
0 libunity.so 0x00000000864dc02e CrashReporting::Android::SignalHandler(int, siginfo*, void*)
1 libunity.so 0x0000000086303bbe NativeRuntimeException::signal_handler(int, siginfo*, void*)
2 app_process32 0x0000000085277242 <system symbols missing>
3 libc.so 0x00000000a48669a8 <system symbols missing>
4 libil2cpp.so 0x0000000081c504c8 il2cpp::os::FastMutex::Lock()
5 libil2cpp.so 0x0000000081c73760 il2cpp::vm::ThreadStart(void*)
6 libil2cpp.so 0x0000000081c588b0 il2cpp::os::Thread::RunWrapper(void*)
7 libil2cpp.so 0x0000000081c57510 il2cpp::os::ThreadImpl::ThreadStartWrapper(void*)
8 libc.so 0x00000000a4895508 <system symbols missing>
9 libc.so 0x00000000a4868f76 <system symbols missing>

We build our game for iOS & windows too. They also have a high crash rate with the same stack trace.

On Windows

Thread 56 (crashed)
0 ntdll 0x00007fff11b902b4 ZwWaitForMultipleObjects
1 KERNELBASE 0x00007fff0eaccc7e WaitForMultipleObjectsEx
2 KERNELBASE 0x00007fff0eaccb6e WaitForMultipleObjects
3 UnityPlayer 0x00007ffeab830202 ?HandleCrash@ExternalCrashHandler@winutils@@AEAAXKKPEBDPEAU_CONTEXT@@PEAU_EXCEPTION_RECORD@@_N@Z
4 UnityPlayer 0x00007ffeab830292 ?HandleCrash@ExternalCrashHandler@winutils@@QEAA_NKKPEAU_EXCEPTION_POINTERS@@@Z
5 UnityPlayer 0x00007ffeab83cc38 ?ProcessInternalCrash@winutils@@YAHPEAU_EXCEPTION_POINTERS@@_N@Z
6 KERNELBASE 0x00007fff0eaf6b2c UnhandledExceptionFilter
7 ntdll 0x00007fff11b981cb RtlUserThreadStart$filt$0
8 ntdll 0x00007fff11b7fd56 _C_specific_handler
9 ntdll 0x00007fff11b9477f RtlpExecuteHandlerForException
10 ntdll 0x00007fff11af4bef RtlDispatchException
11 ntdll 0x00007fff11b934ee KiUserExceptionDispatch
12 GameAssembly 0x00007ffea0842840 ?Lock@FastMutex@os@il2cpp@@QEAAXXZ
13 GameAssembly 0x00007ffea0887704 il2cpp::vm::ThreadStart
14 GameAssembly 0x00007ffea0843e62 ?RunWrapper@Thread@os@il2cpp@@CAXPEAX@Z
15 GameAssembly 0x00007ffea0845493 il2cpp::os::ThreadStartWrapper
16 KERNEL32 0x00007fff11037974 BaseThreadInitThunk
17 ntdll 0x00007fff11b5a271 RtlUserThreadStart
18 ntdll 0x00007fff11b5a271 RtlUserThreadStart

On iOS

Thread 45 (crashed)
0 ios 0x000000010197c7a8 il2cpp::os::FastMutex::Lock()
1 ios 0x000000010199e774 il2cpp::vm::Thread::SetState(Il2CppThread*, il2cpp::vm::ThreadState)
2 ios 0x000000010199ee58 il2cpp::vm::ThreadStart(void*)
3 ios 0x000000010197c234 il2cpp::os::Thread::RunWrapper(void*)
4 ios 0x000000010197e5e8 il2cpp::os::ThreadImpl::ThreadStartWrapper(void*)
5 libsystem_pthread.dylib 0x0000000215f452c0 _pthread_body
6 libsystem_pthread.dylib 0x0000000215f45220 _pthread_start
7 libsystem_pthread.dylib 0x0000000215f48cdc thread_start

Does anyone have the same crash issue and any idea on what could have caused the problem?

2

We’ve had something like that, but couldn’t pinpoint the cause.
Some of the things that helped reduce crashes related to this issue:

  • Remove any use to ConcurrentDictionary
  • Check if your Semaphores are being awaited and released correctly
  • Check if there are any concurrent write to files
1 Like
3

Thanks for the suggestions. Were you able to eliminate these crashes after that or are there still such crashes but just in a smaller volume?

In our case we found the use of C# HttpClient can increase the crash rate significantly and removed it from our Android build. The crash rate dropped significantly after that but still there are about ~0.5% crashed play sessions with these stack traces.

4

I don’t think we’ve received new reports of this crash after the changes.
I would bet it was mainly the ConcurrentDictionary causing them.

5

We’re using gRPC and we think it might be using ConcurrentDictionary.

We’ll probably try modifying the code, recompiling the package and see if the crash is gone.

2 Likes
6

This is a slightly old thread, but we’re also having these crashes.

I suspect it may be caused by incremental garbage collection, possibly in combination with multithreaded code. In the log I’m seeing, the main thread blocking on GC_lock while a number of other threads are all showing as allocating memory. The crashing thread is in il2cpp::os::FastMutex::Lock.

We don’t use ConcurrentDictionary or the C# HttpClient (at least not in any of our code, or any of the code that’s likely to be active during the crash).

1 Like
7

We’re still suffering this. We released a build without Firebase and the crash disappeared. However we need Firebase and we’re still figuring out what’s wrong with it. It might be colliding with something else.

8

I’m still investigating this - we are using Firebase, though only for push notifications.

9

Recently we tried enabling the new Incremental GC and the crash rate spiked while all the logs were pointing to GC_lock or something like that.
We disabled it for the next version and the crashes were gone.
Are you guys using it?

1 Like
10

We are using it, we’ve disabled it and will release a new version asap. I’ll report back with results. Thanks for the tip!

11

Following on from the above - we’ve had incremental GC turned off and that seems to have resolved the issue, at least we’ve not had any further reports.

1 Like
12

We released a new version with incremnetal GC off and the crash went away! Thanks everyone.

3 Likes