BatchDeleteObjects Crash - Unity 5

nindim · April 24, 2015, 12:01am

Hi,

We are running 5.0.1P1 I’m not sure when this crash started happening, but it must be sometime within the last 3 weeks.

We have an intermittent crash that occurs when we are transitioning from some of our main game scenes to the FE / results screens etc. It occurs maybe 15% of the time and seems quite random.

I have tried to get some crash log information, I have only reproduced this problem with Release,IL2CPP, Arm 64 builds so far. It didn’t seem to occur when I was using Mono runtime… (from what I could tell). I have never seen this crash in the Editor, only on iOS builds.

The crash log doesn’t give me much other than:

Thread 3 name: BatchDeleteObjects
Thread 3 Crashed:
0 ABC 0x0000000100d195b4 0x100060000 + 13342132
1 ABC 0x0000000100d15ad0 0x100060000 + 13327056
2 ABC 0x0000000100d156d4 0x100060000 + 13326036
3 ABC 0x0000000100cf5dd4 0x100060000 + 13196756
4 ABC 0x0000000100886d4c 0x100060000 + 8547660
5 ABC 0x000000010085c844 0x100060000 + 8374340
6 ABC 0x000000010085c780 0x100060000 + 8374144
7 ABC 0x0000000100863874 0x100060000 + 8403060
8 ABC 0x0000000100601754 0x100060000 + 5904212
9 ABC 0x000000010071b828 0x100060000 + 7059496
10 ABC 0x000000010079c704 0x100060000 + 7587588
11 libsystem_pthread.dylib 0x0000000197847dc4 _pthread_body + 160
12 libsystem_pthread.dylib 0x0000000197847d20 _pthread_start + 156
13 libsystem_pthread.dylib 0x0000000197844ef4 thread_start + 0

At this point the main thread is doing this:

Thread 0 name: Dispatch queue: com.apple.main-thread
Thread 0:
0 libsystem_kernel.dylib 0x0000000197790e0c mach_msg_trap + 8
1 libsystem_kernel.dylib 0x0000000197790c84 mach_msg + 68
2 IOKit 0x000000018723deb0 io_connect_method + 360
3 IOKit 0x00000001871ee03c IOConnectCallMethod + 168
4 IOAccelerator 0x000000018f8beae8 ioAccelResourceFinalize + 112
5 CoreFoundation 0x0000000185f352b0 CFRelease + 324
6 libGPUSupportMercury.dylib 0x000000018eb50a54 gpulDeleteKernelTexture + 136
7 libGPUSupportMercury.dylib 0x000000018eb509a0 gldReclaimTexture + 40
8 libGPUSupportMercury.dylib 0x000000018eb50aa0 gldDestroyTexture + 40
9 libGFXShared.dylib 0x0000000189beab7c gfxDestroyPluginTexture + 64
10 GLEngine 0x0000000189b8749c gleFreeTextureObject + 44
11 GLEngine 0x0000000189b63480 gleUnbindDeleteHashNamesAndObjects + 228

It makes sense that it’s crashing during the cleanup of the scene as my main game scenes will be unloading at the point it crashes so we can load in the FE/Result scenes.

This is after selecting Re-Symbolicate log from the Device logs window. I have tried to make sure I have my XCode project setup to give me all the debug info, I am using this post process step to set that up for me:

Can anyone shed any light on what I could do to try and track this further? I can’t reproduce this in a simple project so reporting as a bug with an example is pretty much out of the question.

Thanks.

mjohnsonOOO · April 29, 2015, 7:04am

I’m running into the same issue. As far as I know, it started with 5.0.1p1. Using IL2CPP, targeting iOS 7 or 8 SDK.

I’ve been able to symbolicate my crash report for some more details:

Thread 3 name: BatchDeleteObjects
Thread 3 Crashed:
0 — 0x0000000101bd8704 il2cpp::os::ThreadImpl::GetCurrentThread() (ThreadImpl.cpp:189)
1 — 0x0000000101bd8700 il2cpp::os::ThreadImpl::GetCurrentThread() (ThreadImpl.cpp:189)
2 — 0x0000000101bd4c20 il2cpp::os::posix::PosixWaitObject::Wait(unsigned int, bool) (PosixWaitObject.cpp:0)
3 — 0x0000000101bd4824 il2cpp::os::MutexImpl::Lock(bool) (MutexImpl.cpp:39)
4 — 0x0000000101bb4b80 il2cpp::gc::GCHandle::GetTarget(unsigned int) (GCHandle.cpp:186)
5 — 0x00000001016bbb4c TrackedReferenceBase::~TrackedReferenceBase() (RefCounted.h:21)
6 — 0x000000010168bb70 Animation::ReleaseAnimationStates() (Animation.cpp:1886)
7 — 0x00000001016934b0 Animation::~Animation() (Animation.cpp:67)
8 — 0x000000010132037c delete_object_internal_step2(Object*) (BaseObject.cpp:1329)
9 — 0x0000000101486624 BatchDeleteStep2Threaded(void*) (BatchDeleteObjects.cpp:104)
10 — 0x000000010153aaac Thread::RunThreadWrapper(void*) (Thread.cpp:40)
11 libsystem_pthread.dylib 0x00000001938b3dc4 _pthread_body + 156
12 libsystem_pthread.dylib 0x00000001938b3d20 _pthread_start + 152
13 libsystem_pthread.dylib 0x00000001938b0ef4 start_wqthread + 4

This is occurring after destroying GameObjects that had Animations using AnimationClips.

I haven’t yet tried this on 5.0.1p2, but the patch log does not look like it would solve this issue. I also haven’t tried going back to 5.0.1f1, but that is likely my next step in debugging, which unfortunately means re-implementing all the workarounds for HTTPS, generic type reflection and non-colored vertex meshes.

nindim · April 29, 2015, 5:19pm

Good to know it’s not just us seeing this!

We also think it started in 5.0.1p1 (based on when I sent out a Unity upgrade e-mail and when we first got reports about crashes).

We are just about to try 5.0.1p3 which has a fix for:

(684129) - iOS/IL2CPP: Match the behavior of the Mono scripting backend by not stripping the Animation code from the engine when code stripping is enabled.

Could be related…

Let me know if you try 5.0.1f1 and it fixes the problem, I’ll do the same with 5.0.1p3.

nindim · April 29, 2015, 5:35pm

Just reproduced the crash in 5.0.1p3

The callstack in Xcode looks pretty much the same as what you are reporting mjohnsonOOO.

nindim · April 29, 2015, 6:04pm

@mjohnsonOOO

I don’t suppose you have any sort of repro project you could submit with a bug report? I mentioned this crash in the semi official IL2CPP support thread and they asked for a repro case. Unfortunately, we haven’t been able to repro outside of our project (which we can’t submit to Unity).

http://forum.unity3d.com/threads/4-6-ios-64-bit-beta.290551/page-35#post-2090877

mjohnsonOOO · April 30, 2015, 6:42am

I tried 5.0.1p3 today, and it worked for me. Then I saw your post on how you were still seeing the issue and that got me thinking on what I was doing differently. I had put in a few potential workarounds to try and get around the bug when building against 5.0.1p1, so I took those out and saw the crashing again immediately.

The workarounds I did were:

The AnimationClips I had were being re-used in multiple Animations. I was just instantiating one version of each clip and assigning them to whichever Animation needed to use it. I changed that to instantiate a new AnimationClip each time one was being added to an Animation.
Also, on any GameObject that had an Animation component, I added an OnDestroy which would call Animation.Stop and Animation.RemoveClip for every AnimationClip.

With this combination, I haven’t seen the crash again. Although I haven’t done a long term test to see if this just ends up masking the issue for longer. But it went from crashing within the first 2 times of creating and destroying a number of animating GameObjects, to being able to run that cycle tens of times without issue.

I still haven’t been able to put together a simple test project to replicate the issue, but have some better ideas now that these workarounds seem to be doing something. I’m also in a position where I cannot submit the full project to Unity for bug reporting.

Hope this can help.

AmirLevi · April 30, 2015, 12:32pm

I am getting the same errors, the problem is, it happen randomly and i cant reproduce the problem
I’ve tried it on 5.0.1p1 - 5.0.1p3 IOS builds

here is the log in xcode:

BatchDeleteObjects (10)
#0 0x01a3ccbb in il2cpp:s::ThreadImpl::GetCurrentThread() ()
#1 0x01a38ede in il2cpp:s::posix::posixWaitObject::Wait(unsigned int, bool) at /Users/builduser/buildslave/unity/build/Tools/il2cpp/il2cpp/libil2cpp/os/Posix/PosixWaitObject.cpp:90
#2 0x01a38a9f in il2cpp:s::MutexImpl::TryLock(unsigned int, bool) [inlined] at /Users/builduser/buildslave/unity/build/Tools/il2cpp/il2cpp/libil2cpp/os/Posix/MutexImpl.cpp:39
#3 0x01a38a7b in il2cpp:s::MutexImpl::Lock(bool) at /Users/builduser/buildslave/unity/build/Tools/il2cpp/il2cpp/libil2cpp/os/Posix/MutexImpl.cpp:26
#4 0x01a332cb in il2cpp:s::Mutex::Lock(bool) at /Users/builduser/buildslave/unity/build/Runtime/Threads/Mutex.cpp:30
#5 0x01a14530 in il2cpp::gc::GCHandle::GetTarget(unsigned int) at /Users/builduser/buildslave/unity/build/Tools/il2cpp/il2cpp/libil2cpp/gc/GCHandle.cpp:186
#6 0x01285006 in ~TrackedReferenceBase at /Users/builduser/buildslave/unity/build/./Runtime/BaseClasses/RefCounted.h:23
#7 0x012473a6 in ReleaseAnimationStates at /Users/builduser/buildslave/unity/build/Runtime/Animation/Animation.cpp:1886
#8 0x0124f85b in ThreadedCleanup [inlined] at /Users/builduser/buildslave/unity/build/Runtime/Animation/Animation.cpp:67
#9 0x0124f853 in ~Animation at /Users/builduser/buildslave/unity/build/Runtime/Animation/Animation.h:84
#10 0x00da2235 in delete_object_internal_step2 at /Users/builduser/buildslave/unity/build/Runtime/BaseClasses/BaseObject.cpp:1329
#11 0x00f48f57 in BatchDeleteStep2Threaded at /Users/builduser/buildslave/unity/build/Runtime/Misc/BatchDeleteObjects.cpp:105
#12 0x00ff3f2f in RunThreadWrapper at /Users/builduser/buildslave/unity/build/Runtime/Threads/Thread.cpp:40
#13 0x08ddce13 in _pthread_body ()
#14 0x08ddcd89 in _pthread_start ()
#15 0x08ddae52 in thread_start ()

Rajmahal · April 30, 2015, 2:28pm

I think I’m running into this as well. Happens randomly when I try to leave certain scenes and go to a new one. It doesn’t happen all the time on those scenes … just randomly. It used to happen a lot more frequently, but setting the iOS to 8.0 and Disabling stripping level in IL2CPP seems to have helped. However, it still seems to happen and I do get the same error message in Xcode around delete objects.

I’m going to try the animation workarounds that were suggested and see if that helps. I may also be able to provide a bug report and reproduction case.

mjohnsonOOO · May 1, 2015, 12:02am

So after doing some longer tests, I ended up still hitting the BatchDeleteObjects crash. I have gone back to 5.0.1f1, and have not gotten the crash since, so it seems to confirm that the bug was introduced with the 5.0.1p1 patch.

Rajmahal · May 1, 2015, 6:01pm

That’s unfortunate. I’m trying to create a reproduction project now. Hopefully I can reproduce it and submit a bug report.

mjohnsonOOO · May 1, 2015, 7:01pm

I have created a sample project that causes the crash and have submitted a bug report: Case 694004

Rajmahal · May 1, 2015, 7:32pm

Fantastic … I was trying to do the same but my sample project wasn’t replicating the crash.

Enkiduouo · May 4, 2015, 6:49pm

Also seeing this 5.0.1p1->p3 - can repro 100% of the time when trying to transition scenes, will attempt the animation fixes - sadly also cannot submit my project to unity - hopefully they will jump on the bug report - jumping from one broken version of unity to another is getting tiring.

Here is my stack :

#0 0x0242d0cc in il2cpp::os::ThreadImpl::GetCurrentThread() ()
#1 0x02428c54 in il2cpp::os::posix::PosixWaitObject::Wait(unsigned int, bool) at /Users/builduser/buildslave/unity/build/Tools/il2cpp/il2cpp/libil2cpp/os/Posix/PosixWaitObject.cpp:90
#2 0x024285f8 in il2cpp::os::MutexImpl::TryLock(unsigned int, bool) [inlined] at /Users/builduser/buildslave/unity/build/Tools/il2cpp/il2cpp/libil2cpp/os/Posix/MutexImpl.cpp:39
#3 0x024285d4 in il2cpp::os::MutexImpl::Lock(bool) at /Users/builduser/buildslave/unity/build/Tools/il2cpp/il2cpp/libil2cpp/os/Posix/MutexImpl.cpp:26
#4 0x023fb808 in il2cpp::gc::GCHandle::GetTarget(unsigned int) at /Users/builduser/buildslave/unity/build/Tools/il2cpp/il2cpp/libil2cpp/gc/GCHandle.cpp:186
#5 0x01f2dc90 in ~TrackedReferenceBase at /Users/builduser/buildslave/unity/build/./Runtime/BaseClasses/RefCounted.h:23
#6 0x01eef024 in ReleaseAnimationStates at /Users/builduser/buildslave/unity/build/Runtime/Animation/Animation.cpp:1886
#7 0x01eeef64 in ThreadedCleanup at /Users/builduser/buildslave/unity/build/Runtime/Animation/Animation.cpp:67
#8 0x01ef8034 in ~Animation at /Users/builduser/buildslave/unity/build/Runtime/Animation/Animation.h:84
#9 0x01ba2ca0 in delete_object_internal_step2 at /Users/builduser/buildslave/unity/build/Runtime/BaseClasses/BaseObject.cpp:1329
#10 0x01d15d70 in BatchDeleteStep2Threaded at /Users/builduser/buildslave/unity/build/Runtime/Misc/BatchDeleteObjects.cpp:105
#11 0x01dbd8b8 in RunThreadWrapper at /Users/builduser/buildslave/unity/build/Runtime/Threads/Thread.cpp:40
#12 0x38e22e66 in _pthread_body ()
#13 0x38e22dda in _pthread_start ()
#14 0x38e20b84 in thread_start ()

Enkiduouo · May 4, 2015, 7:18pm

Do a manual animation cleanup before doing Application.LoadLevelAsync transitions seems to be a functional workaround :
Thanks for the input though group, saved me a ton of work.

voidDoManualAnimationCleanup ()
{
foreach(AnimationainFindObjectsOfType(typeof(Animation))) {
a.Stop();

List clipNames = newList();
foreach(AnimationState v in a)
clipNames.Add(v.clip.name);

foreach(stringsinclipNames)
a.RemoveClip(s);
}
}

Rajmahal · May 7, 2015, 5:35pm

I’m starting to run into the crash again … I’m going to try your code. I take it is just a function you run right before you running Application.LoadLevelAsynch?

Enkiduouo · May 7, 2015, 6:52pm

Yes -

I was getting occasional crashes as well and exceptions in this line
a.RemoveClip(s);

So I wrapped that in a try / catch block so that it keeps trying to remove other animations if it fails on one - that seems to be working for me now.

Rajmahal · May 7, 2015, 9:54pm

What type of exception did you catch for? Was it a null reference exception … occurring if a was null, for example? Do you mind just adding that to the original code you shared?

Enkiduouo · May 8, 2015, 12:01am

Nothing exciting or clever :

void DoManualAnimationCleanup ()
{
foreach(Animation a in FindObjectsOfType(typeof(Animation))) {
a.Stop();

List clipNames = new List();
foreach(AnimationState v in a)
clipNames.Add(v.clip.name);

foreach(string s in clipNames) {
try {
a.RemoveClip(s);
} catch (Exception e) {
Debug.LogWarning("Failed to remove animation : " + e.ToString());
}
}
}
}

Rajmahal · May 8, 2015, 3:57pm

Thanks … appreciate the help.

Rajmahal · May 8, 2015, 6:42pm

Specifically what type of Exception do you try to catch for? If I try the generic Exception that you have in the code, Unity kicks up an error. Is it a UnityException or one of the other types?