2022.2.19f1 / 2023.1.0b17 native crashe when finalizing AnimationCurve

Hi!

We’ve encountered a problem while migrating our project from 2020.3.40f1 to 2022.2.19f1, it seems there’s a problem with AnimationCurve being attempted to be destroyed without it ever getting constructed in the first place, which evidently results in a crash. Also worth mentioning is that the problem manifests itself on Android platform only and no matter what configuration or settings those are assembled with, OS versions don’t affect the behavior as well.

The crash happens when we load a scene, exit that scene and then run garbage collection.

Something along these lines: main menulevelmain menurun gc.

The first thing we concentrated on is the removal of all of our explicit AnimationCurve uses case from the codebase just to be able to isolate the possibility of us doing something wonky with it, but it didn’t prove to be the case here.

Next, we decided to try other Unity 2022 releases just to exclude the possibility of a regression that could’ve been introduced at some point. We’ve even given a go to the 2023.1 beta release. No dice.

We have also tried disabling incremental GC, building in development mode, switching IL2CPP flags to produce a smaller build instead of a more optimized one, and using GLES renderer instead of a Vulkan-based one. Still no positive outcome.

The last resort for us was to export the Android project locally and dig into IL2CPP-generated code to see what was going under the hood. We added a lot of logging around the AnimationCurve functions to see if anything gets corrupted for some reason.

Our log output would look like this for proper AnimationCurve construction and destruction flows (
i.e. AnimationCurve_Finalize invocations that clearly have a matching construction call AnimationCurve__ctor before it):

AnimationCurve_Finalize_m803AC16166EE497C4DFA996B15692D91F4D04C3C: 0=482526307184 1=482526307184 2=482526307184 this=483584882176
memzero AnimationCurve_Finalize_m803AC16166EE497C4DFA996B15692D91F4D04C3C: 0=0 1=0 2=0 this=483584882176```

In case there's a mismatch, i.e. the finalization is performed on an `AnimationCurve`, which hasn't been constructed yet, we'd see this (preceded with a `=====>` just to make it easier to spot and filter out):

```=====> AnimationCurve_Finalize_m803AC16166EE497C4DFA996B15692D91F4D04C3C: 0=12970367422304525504 1=0 2=0 this=484505394416```

Every log message provides the following information:

- 3 copies of the pointer stored within `AnimationCurve_tCBFFAAD05CEBB35EF8D8631BD99914BE1A6BB354` the (`0=…, 1=…, 2=…`) parts.
- pointer to the `AnimationCurve_tCBFFAAD05CEBB35EF8D8631BD99914BE1A6BB354` instance itself (the `this=` part)

`AnimationCurve_tCBFFAAD05CEBB35EF8D8631BD99914BE1A6BB354` would look like this now:

```struct AnimationCurve_tCBFFAAD05CEBB35EF8D8631BD99914BE1A6BB354 : public RuntimeObject
{
intptr_t ___m_Ptr;
intptr_t ___m_Ptr2;
intptr_t ___m_Ptr3;
};```

Copies were introduced to see if there's some sort of memory corruption taking place, which somehow either reuses some older instances without ever re-constructing them properly or it's simply a case of double-freeing. They are assigned along the "primary" `__m_Ptr` at the same places in the code (like `AnimationCurve__ctor_mEABC98C03805713354D61E50D9340766BD5B717E`, etc).

Also, we extended the `AnimationCurve_Finalize` function to include a logic, which would check whether all of the pointer copies are identical and if there are not, then it would log the case and skip destruction of the `AnimationCurve` instance altogether:

```csharp
IL2CPP_EXTERN_C IL2CPP_METHOD_ATTR void AnimationCurve_Finalize_m803AC16166EE497C4DFA996B15692D91F4D04C3C (AnimationCurve_tCBFFAAD05CEBB35EF8D8631BD99914BE1A6BB354* __this, const RuntimeMethod* method)
{
  {
  }
  {
    auto __finallyBlock = il2cpp::utils::Finally([&]
    {

FINALLY_0010:
      {
        Object_Finalize_mC98C96301CCABFE00F1A7EF8E15DF507CACD42B2(__this, NULL);
        return;
      }
    });
    try
    {
      intptr_t L_0 = __this->___m_Ptr;
      if (L_0 == __this->___m_Ptr2 && L_0 == __this->___m_Ptr3)
      {
        AnimationCurve_Internal_Destroy_m240B298D0A13EEC1652955C4BDCDBE9B7B2EE296(L_0, NULL);

        // zero-out the struct to be sure there are no leftovers in case something reuses the object.
        memset(__this, 0, sizeof(AnimationCurve_tCBFFAAD05CEBB35EF8D8631BD99914BE1A6BB354));

        // log normally
      }
      else
      {
        // log the offending case
      }
      goto IL_0018;
    }
    catch(Il2CppExceptionWrapper& e)
    {
      __finallyBlock.StoreException(e.ex);
    }
  }

IL_0018:
  {
    return;
  }
}

Modifying the code this way solved our crashes, but of course, there’s no guarantee there won’t be any memory leaks associated with it or something even worse (latent segfaults that are waiting to happen, etc).

Currently, we’re sticking to the hack mentioned above, but hoping for a better solution, which wouldn’t force us to inject anything into C++ code right before compiling & linking it.

Segfault’s call stack, which might be useful:

Thread 0 Crashed:
0   libunity.so                     0x79b143c650        MemoryManager::VirtualAllocator::GetBlockInfoFromPointer
1   libunity.so                     0x79b14398b8        DualThreadAllocator<T>::Contains
2   libunity.so                     0x79b14396a8        DualThreadAllocator<T>::TryDeallocate
3   libunity.so                     0x79b143bc3c        MemoryManager::smile:eallocate
4   libil2cpp.so                    0x799dfe01f8        [inlined] AnimationCurve_Internal_Destroy_m240B298D0A13EEC1652955C4BDCDBE9B7B2EE296 (UnityEngine.CoreModule.cpp:9709)
5   libil2cpp.so                    0x799dfe01f8        AnimationCurve_Finalize_m803AC16166EE497C4DFA996B15692D91F4D04C3C (UnityEngine.CoreModule.cpp:9749)
6   libil2cpp.so                    0x799ba95f04        il2cpp::vm::Runtime::InvokeWithThrow (Runtime.cpp:604)
7   libil2cpp.so                    0x799ba95e50        il2cpp::vm::Runtime::Invoke (Runtime.cpp:590)
8   libil2cpp.so                    0x799ba8d7c8        il2cpp::gc::GarbageCollector::RunFinalizer (GarbageCollector.cpp:178)
9   libil2cpp.so                    0x799babee10        GC_invoke_finalizers (finalize.c:1314)
10  libil2cpp.so                    0x799ba8d710        il2cpp::gc::FinalizerThread (GarbageCollector.cpp:102)
11  libil2cpp.so                    0x799ba935b0        il2cpp::os::Thread::RunWrapper (Thread.cpp:200)
12  libil2cpp.so                    0x799bab4958        il2cpp::os::ThreadImpl::ThreadStartWrapper (ThreadImpl.cpp:123)
13  libc.so                         0x7cf0e52268        __pthread_start
14  libc.so                         0x7cf0de4a2c        __start_thread

A bug has been reported: case number IN-41806

1 Like

It’s possible there’s a bug in managed layer of AnimatioCurve, from which this cpp code is generated.

You can try asking Unity Engine - Unity Discussions, since AnimationCurve is handled by Animation team.

But if you can reproduce this locally, we would love to get a bug report with repro project attached.

Thank you!

We ran into a very similar problem when upgrading from 2022.2.11 to 2022.2.20, the stack trace is almost identical.

In our case 2022.2.13 turned out to be the last one working. Starting with .14 our app crashes, .21 which was just released did not fix it. The release notes for 2022.2.14 reveal a couple of optimizations related to animations.

The problem seems to be restricted to IL2CPP builds. A test with Mono as scripting backend on 2022.2.20 worked as expected

1 Like

I very much doubt so. Note that broken finalizer invocations do not have correspondent initialize invocations. If it was in the managed code, you’d see both sides.

I’d place my bets on some sort of GC bug.

I have this bug as well. It is funny that I tried all steps mentioned in first message and it was also unsuccessful. But I have this bug only on one device out of 6 I tested it.
I hope it would be fixed soon

Interesting, we tested on 2 devices, Galaxy Z Flip4 and Galaxy A12. The crash was reproducible on both of them

I got the same issue when migrating from 2020 LTS to 2022 LTS.
There are a lot of AnimationCurve in the project and it was not possible to find a specific case and highlight it in the test project.
Tell me, please, was someone able to send a bug report with playback on this problem or was it never reported to the developers? I didn’t find anything similar in issuetracker.

1 Like

I too was unable to file a bug report with a small test project. It happens reproducibly with the huge customer’s main project

Yep, I’ve got the same issue on 2022.3.2f1. Makes our Android version unshippable. I couldn’t find any further details on this one outside of this thread. Has anybody had any further luck tracking this one down?

We recently upgraded our project from 2021 LTS to 2022 LTS, and that seems to be a common issue of people reporting in this thread. Perhaps there’s something unexpected happening in the upgrade process…? shrugs I’m only seeing this in one particular portion of our game, so perhaps it’s an asset issue rather than a general code thing…

Ben

1 Like

If you can isolate this particular portion into a separate project and reproduce the bug there - please send a bug report with it, it will help us all in this thread a lot.

Or at least tell more about your particular portion so that you can try to reproduce this bug for others to bug report.
The more information - the more chances we have to reproduce and pass the problem to Unity

1 Like

I’ve got it narrowed down to a ParticleSystem. More specifically, a series of nested ParticleSystems. Calling Play() on that seems to cause the crash. It seems like it’s when the GC tries to collect the ParticleSystems that it falls over. This is Android-specific - other platforms are fine.

I’m still investigating, but thought I’d post my initial findings in case that sounds familiar to anyone else…?

Ben

2 Likes

Hey Ben,

I can confirm we are experiencing the exact issue (Unity 2022.3.3f1), only occurs on IL2CPP, doesn’t replicate on mono.
If particle system is allowed to run enough time, it seems to not reproduce.
Also seems that it’s not related to nested or not, since I flattened the hierarchy of our PS under 1 gameobject (non PS) and it still reproduced.

I’ll update once I find a workaround, this definitely looks like a bug in Unity 2022.

1 Like

@Wriggler

I have found the root cause for our issue, unsure if it’s the same for you.
We are using UI extensions, particularly UIParticleSystem.

The root cause was accessing the curve in OnPopulateMesh after it was disabled, caching the frameOverTime struct locally, and accessing it (rather than the actual PS one) - which solved the issue.

I hope this helps.

But this is definitely an issue with Unity, that for some reason accessing that struct in that function (lifecycle), causes a GC crash.

1 Like

Can you submit a bug report for this issue? This is definitely something that we want to address.

https://unity.com/releases/editor/qa/bug-reporting

Opened, case: 45835

2 Likes

Nice work! Yes, I’m seeing these in UI particles too. Sounds like we have the same issue.

Ben

1 Like

Thank you very much for the hints, I also found a crash in the project when accessing to ParticleSystem.MinMaxCurve. Now I can at least continue testing the transition to 2022 without this functionality

Special thanks for the bug report.
I really hope that they will quickly fix this problem

How do you guys even get the full stack trace for IL2CPP?

With Logcat, all I get is this:

#00 pc 00000000004b96fc  /data/app/~~ghQWRIe1Q9GgRlClVdSg1w==/<redacted>-2k7MKcJ8BTxxd_-Zp_54Eg==/lib/arm64/libunity.so (MemoryManager::VirtualAllocator::GetBlockInfoFromPointer(void const*)+12) (BuildId: 193a04e962461296)
#01 pc 00000000004b45e8  /data/app/~~ghQWRIe1Q9GgRlClVdSg1w==/<redacted>-2k7MKcJ8BTxxd_-Zp_54Eg==/lib/arm64/libunity.so (DualThreadAllocator<DynamicHeapAllocator>::Contains(void const*) const+64) (BuildId: 193a04e962461296)
#02 pc 00000000004b8bbc  /data/app/~~ghQWRIe1Q9GgRlClVdSg1w==/<redacted>-2k7MKcJ8BTxxd_-Zp_54Eg==/lib/arm64/libunity.so (MemoryManager::smile:eallocate(void*, MemLabelId const&, char const*, int)+216) (BuildId: 193a04e962461296)
#03 pc 00000000052d6504  /data/app/~~ghQWRIe1Q9GgRlClVdSg1w==/<redacted>-2k7MKcJ8BTxxd_-Zp_54Eg==/lib/arm64/libil2cpp.so (BuildId: 14f2531e46eafca2)
#04 pc 00000000060637b4  /data/app/~~ghQWRIe1Q9GgRlClVdSg1w==/<redacted>-2k7MKcJ8BTxxd_-Zp_54Eg==/lib/arm64/libil2cpp.so (BuildId: 14f2531e46eafca2)
#05 pc 0000000006063700  /data/app/~~ghQWRIe1Q9GgRlClVdSg1w==/<redacted>-2k7MKcJ8BTxxd_-Zp_54Eg==/lib/arm64/libil2cpp.so (BuildId: 14f2531e46eafca2)
#06 pc 00000000060ccf70  /data/app/~~ghQWRIe1Q9GgRlClVdSg1w==/<redacted>-2k7MKcJ8BTxxd_-Zp_54Eg==/lib/arm64/libil2cpp.so (BuildId: 14f2531e46eafca2)
#07 pc 00000000061070f4  /data/app/~~ghQWRIe1Q9GgRlClVdSg1w==/<redacted>-2k7MKcJ8BTxxd_-Zp_54Eg==/lib/arm64/libil2cpp.so (BuildId: 14f2531e46eafca2)
#08 pc 00000000060cceb8  /data/app/~~ghQWRIe1Q9GgRlClVdSg1w==/<redacted>-2k7MKcJ8BTxxd_-Zp_54Eg==/lib/arm64/libil2cpp.so (BuildId: 14f2531e46eafca2)
#09 pc 00000000060c2498  /data/app/~~ghQWRIe1Q9GgRlClVdSg1w==/<redacted>-2k7MKcJ8BTxxd_-Zp_54Eg==/lib/arm64/libil2cpp.so (BuildId: 14f2531e46eafca2)
#10 pc 00000000060be6bc  /data/app/~~ghQWRIe1Q9GgRlClVdSg1w==/<redacted>-2k7MKcJ8BTxxd_-Zp_54Eg==/lib/arm64/libil2cpp.so (BuildId: 14f2531e46eafca2)
#11 pc 00000000000b67a8  /apex/com.android.runtime/lib64/bionic/libc.so (__pthread_start(void*)+208) (BuildId: 39de735713a979219b391e7611970823)
#12 pc 000000000005340c  /apex/com.android.runtime/lib64/bionic/libc.so (__start_thread+64) (BuildId: 39de735713a979219b391e7611970823)
1 Like

Ah, okay, thanks to https://support.unity.com/hc/en-us/articles/115000292166-Symbolicate-Android-crash I found how to do it, turns out I have the same AnimationCurve issue:

2023.07.11 02:12:22.447 21946 22008 Error CRASH       #00 pc 00000000004b96fc (MemoryManager::VirtualAllocator::GetBlockInfoFromPointer(void const*) at ??:0)  /data/app/~~ghQWRIe1Q9GgRlClVdSg1w==/<redacted>-2k7MKcJ8BTxxd_-Zp_54Eg==/lib/arm64/libunity.so (MemoryManager::VirtualAllocator::GetBlockInfoFromPointer(void const*)+12) (BuildId: 193a04e962461296)
2023.07.11 02:12:22.447 21946 22008 Error CRASH       #01 pc 00000000004b45e8 (DualThreadAllocator<DynamicHeapAllocator>::Contains(void const*) const at ??:0)  /data/app/~~ghQWRIe1Q9GgRlClVdSg1w==/<redacted>-2k7MKcJ8BTxxd_-Zp_54Eg==/lib/arm64/libunity.so (DualThreadAllocator<DynamicHeapAllocator>::Contains(void const*) const+64) (BuildId: 193a04e962461296)
2023.07.11 02:12:22.447 21946 22008 Error CRASH       #02 pc 00000000004b8bbc (MemoryManager::smile:eallocate(void*, MemLabelId const&, char const*, int) at ??:0)  /data/app/~~ghQWRIe1Q9GgRlClVdSg1w==/<redacted>-2k7MKcJ8BTxxd_-Zp_54Eg==/lib/arm64/libunity.so (MemoryManager::smile:eallocate(void*, MemLabelId const&, char const*, int)+216) (BuildId: 193a04e962461296)
2023.07.11 02:12:22.447 21946 22008 Error CRASH       #03 pc 00000000052d6504 (AnimationCurve_Internal_Destroy_m240B298D0A13EEC1652955C4BDCDBE9B7B2EE296 at C:/Users/tomss/Projects/ROLLHILL/rollhill/Library/Bee/artifacts/Android/il2cppOutput/cpp\UnityEngine.CoreModule.cpp:12656)  /data/app/~~ghQWRIe1Q9GgRlClVdSg1w==/<redacted>-2k7MKcJ8BTxxd_-Zp_54Eg==/lib/arm64/libil2cpp.so (BuildId: 14f2531e46eafca2)
2023.07.11 02:12:22.447 21946 22008 Error CRASH       #04 pc 00000000060637b4 (il2cpp::vm::Runtime::InvokeWithThrow(MethodInfo const*, void*, void**) at C:/Program Files/Unity/Hub/Editor/2022.3.2f1/Editor/Data/il2cpp/libil2cpp/vm\Runtime.cpp:604)  /data/app/~~ghQWRIe1Q9GgRlClVdSg1w==/<redacted>-2k7MKcJ8BTxxd_-Zp_54Eg==/lib/arm64/libil2cpp.so (BuildId: 14f2531e46eafca2)
2023.07.11 02:12:22.447 21946 22008 Error CRASH       #05 pc 0000000006063700 (il2cpp::vm::Runtime::Invoke(MethodInfo const*, void*, void**, Il2CppException**) at C:/Program Files/Unity/Hub/Editor/2022.3.2f1/Editor/Data/il2cpp/libil2cpp/vm\Runtime.cpp:590)  /data/app/~~ghQWRIe1Q9GgRlClVdSg1w==/<redacted>-2k7MKcJ8BTxxd_-Zp_54Eg==/lib/arm64/libil2cpp.so (BuildId: 14f2531e46eafca2)
2023.07.11 02:12:22.447 21946 22008 Error CRASH       #06 pc 00000000060ccf70 (il2cpp::gc::GarbageCollector::RunFinalizer(void*, void*) at C:/Program Files/Unity/Hub/Editor/2022.3.2f1/Editor/Data/il2cpp/libil2cpp/gc\GarbageCollector.cpp:178)  /data/app/~~ghQWRIe1Q9GgRlClVdSg1w==/<redacted>-2k7MKcJ8BTxxd_-Zp_54Eg==/lib/arm64/libil2cpp.so (BuildId: 14f2531e46eafca2)
2023.07.11 02:12:22.447 21946 22008 Error CRASH       #07 pc 00000000061070f4 (GC_invoke_finalizers at C:/Program Files/Unity/Hub/Editor/2022.3.2f1/Editor/Data/il2cpp/external/bdwgc/extra/..\finalize.c:1315)  /data/app/~~ghQWRIe1Q9GgRlClVdSg1w==/<redacted>-2k7MKcJ8BTxxd_-Zp_54Eg==/lib/arm64/libil2cpp.so (BuildId: 14f2531e46eafca2)
2023.07.11 02:12:22.447 21946 22008 Error CRASH       #08 pc 00000000060cceb8 (il2cpp::gc::GarbageCollector::InvokeFinalizers() at C:/Program Files/Unity/Hub/Editor/2022.3.2f1/Editor/Data/il2cpp/libil2cpp/gc\BoehmGC.cpp:460)  /data/app/~~ghQWRIe1Q9GgRlClVdSg1w==/<redacted>-2k7MKcJ8BTxxd_-Zp_54Eg==/lib/arm64/libil2cpp.so (BuildId: 14f2531e46eafca2)
2023.07.11 02:12:22.447 21946 22008 Error CRASH       #09 pc 00000000060c2498 (il2cpp::os::Thread::RunWrapper(void*) at C:/Program Files/Unity/Hub/Editor/2022.3.2f1/Editor/Data/il2cpp/libil2cpp/os\Thread.cpp:201)  /data/app/~~ghQWRIe1Q9GgRlClVdSg1w==/<redacted>-2k7MKcJ8BTxxd_-Zp_54Eg==/lib/arm64/libil2cpp.so (BuildId: 14f2531e46eafca2)
2023.07.11 02:12:22.447 21946 22008 Error CRASH       #10 pc 00000000060be6bc (il2cpp::os::ThreadImpl::ThreadStartWrapper(void*) at C:/Program Files/Unity/Hub/Editor/2022.3.2f1/Editor/Data/il2cpp/libil2cpp/os/Posix\ThreadImpl.cpp:123)  /data/app/~~ghQWRIe1Q9GgRlClVdSg1w==/<redacted>-2k7MKcJ8BTxxd_-Zp_54Eg==/lib/arm64/libil2cpp.so (BuildId: 14f2531e46eafca2)
2023.07.11 02:12:22.447 21946 22008 Error CRASH       #11 pc 00000000000b67a8 (libc.so not found)  /apex/com.android.runtime/lib64/bionic/libc.so (__pthread_start(void*)+208) (BuildId: 39de735713a979219b391e7611970823)
2023.07.11 02:12:22.447 21946 22008 Error CRASH       #12 pc 000000000005340c (libc.so not found)  /apex/com.android.runtime/lib64/bionic/libc.so (__start_thread+64) (BuildId: 39de735713a979219b391e7611970823)

And looking into it, seems to be related with ParticleSystems/TrailRenderers (will continue debugging tomorrow). As disabling those, doesn’t crash the game.

1 Like

Thank you, guys! I was moving to the 2022.3.4 version in order for my iOS build to work, and it works only on this version somehow)
But then I faced this bug on Android.
Thanks to you I was able to find that one UI particle emitter that caused the problem! :smile:

Before this thread I was desperate. In my case, I was referring to the particles that were hidden.