[IN-43583] Possible memory corruption/overwrite with High Quality Line Rendering

Unity Engine
,
1

I’ve been having an issue with builds of my project hard crashing at random points (2023.1b14, and happening in 2023.1b20). It would happen seconds or minutes into running the build at seemingly random points.

From debugging the stack traces look to be at different origins. However, all of them have in common that it happens somewhere in either allocation or deallocation of memory. From this pattern I assume this is due to memory corruption from overwrite and/or double deallocation.

Here are some typical examples:

0x00007FFE0C33F222 (UnityPlayer) block_remove
0x00007FFE0C33FA3A (UnityPlayer) tlsf_free
0x00007FFE0B4378C3 (UnityPlayer) DynamicHeapAllocator::smile:eallocate
0x00007FFE0B43EC9B (UnityPlayer) DualThreadAllocator<DynamicHeapAllocator>::TryDeallocate
0x00007FFE0B43B360 (UnityPlayer) MemoryManager::smile:eallocate
0x00007FFE0B6BB13D (UnityPlayer) profiler_start_new_frame
0x00007FFE0B6A366E (UnityPlayer) `InitPlayerLoopCallbacks'::`2'::InitializationProfilerStartFrameRegistrator::Forward
0x00007FFE0B69AAD7 (UnityPlayer) ExecutePlayerLoop
0x00007FFE0B69AC84 (UnityPlayer) ExecutePlayerLoop
0x00007FFE0B69B034 (UnityPlayer) PlayerLoop
0x00007FFE0B8E36BB (UnityPlayer) PerformMainLoop
0x00007FFE0B8E5A1B (UnityPlayer) MainMessageLoop
0x00007FFE0B8E8D80 (UnityPlayer) UnityMainImpl
0x00007FFE0B8E8F8B (UnityPlayer) UnityMain

0x00007FFEBD6A7D32 (UnityPlayer) block_remove
0x00007FFEBD6A854A (UnityPlayer) tlsf_free
0x00007FFEBC7A7983 (UnityPlayer) DynamicHeapAllocator::smile:eallocate
0x00007FFEBC7ACEE2 (UnityPlayer) DelayedPointerDeletionManager::CleanupPendingMainThreadPointersInternal
0x00007FFEBC7AE8A9 (UnityPlayer) DualThreadAllocator<DynamicHeapAllocator>::Allocate
0x00007FFEBC7AAE88 (UnityPlayer) MemoryManager::Allocate
0x00007FFEBCADA630 (UnityPlayer) dynamic_array_detail::dynamic_array_data::resize_buffer
0x00007FFEBC6D6BBC (UnityPlayer) dynamic_array<double,0>::resize_buffer_nocheck
0x00007FFEBC9453EF (UnityPlayer) RenderingCommandBuffer::PPtrResolver<Material>::Resolve
0x00007FFEBC93EAA8 (UnityPlayer) RenderingCommandBuffer::PrepareState
0x00007FFEBC999445 (UnityPlayer) ScriptableRenderContext::ExecuteScriptableRenderLoop
0x00007FFEA2F3BCE2 (GameAssembly) [\Library\Bee\artifacts\WinPlayerBuildProgram\il2cppOutput\cpp\Unity.RenderPipelines.HighDefinition.Runtime__2.cpp:70710] HDRenderPipeline_Render_mD5A5DFADAEB2A8DB36E4261A8154AB6206D56557
0x00007FFEA32056CD (GameAssembly) [\Library\Bee\artifacts\WinPlayerBuildProgram\il2cppOutput\cpp\UnityEngine.CoreModule__4.cpp:27502] RenderPipelineManager_DoRenderLoop_Internal_mB646C8738F4A9859101F3BE94809E2E10BBDB1FB
0x00007FFE9FE70D16 (GameAssembly) [C:\Program Files\Unity\Hub\Editor\2023.1.0b14\Editor\Data\il2cpp\libil2cpp\vm\Runtime.cpp:638] il2cpp::vm::Runtime::InvokeWithThrow
0x00007FFE9FE707B9 (GameAssembly) [C:\Program Files\Unity\Hub\Editor\2023.1.0b14\Editor\Data\il2cpp\libil2cpp\vm\Runtime.cpp:623] il2cpp::vm::Runtime::Invoke
0x00007FFEBC99B1CD (UnityPlayer) ScriptableRenderContext::ExtractAndExecuteRenderPipeline
0x00007FFEBC83A4FC (UnityPlayer) RenderManager::RenderCamerasWithScriptableRenderLoop
0x00007FFEBC83AC5C (UnityPlayer) RenderManager::RenderCameras
0x00007FFEBCA0F45F (UnityPlayer) PlayerRender
0x00007FFEBCA06C37 (UnityPlayer) ExecutePlayerLoop
0x00007FFEBCA06DE4 (UnityPlayer) ExecutePlayerLoop
0x00007FFEBCA07194 (UnityPlayer) PlayerLoop
0x00007FFEBCC4E86B (UnityPlayer) PerformMainLoop
0x00007FFEBCC50BCB (UnityPlayer) MainMessageLoop
0x00007FFEBCC53F30 (UnityPlayer) UnityMainImpl
0x00007FFEBCC5413B (UnityPlayer) UnityMain

0x00007FFE0C33F222 (UnityPlayer) block_remove
0x00007FFE0C33FA3A (UnityPlayer) tlsf_free
0x00007FFE0B4378C3 (UnityPlayer) DynamicHeapAllocator::smile:eallocate
0x00007FFE0B43EC9B (UnityPlayer) DualThreadAllocator<DynamicHeapAllocator>::TryDeallocate
0x00007FFE0B43B360 (UnityPlayer) MemoryManager::smile:eallocate
0x00007FFE0B4380D4 (UnityPlayer) operator delete
0x00007FFE0B6ED36F (UnityPlayer) std::_Tree_val<std::_Tree_simple_types<HWND__ * __ptr64> >::_Erase_tree<std::allocator<std::_Tree_node<HWND__ * __ptr64,void * __ptr64> > >
0x00007FFE0B6EA70F (UnityPlayer) ContainerClear<std::map<DeprecatedFastPropertyNameSerialization,float,std::less<DeprecatedFastPropertyNameSerialization>,std::allocator<std::pair<DeprecatedFastPropertyNameSerialization const ,float> > > >
0x00007FFE0B6DC028 (UnityPlayer) UnityPropertySheet::operator=
0x00007FFE0B6DF777 (UnityPlayer) Material::CopyPropertiesFromMaterial
0x00007FFE0B25FDC4 (UnityPlayer) Material_CUSTOM_CopyPropertiesFromMaterial_Injected
0x00007FFE001B6049 (GameAssembly) [\Library\Bee\artifacts\WinPlayerBuildProgram\il2cppOutput\cpp\Unity.DemoTeam.Hair.Runtime.cpp:16394] HairInstance_UpdateRendererState_mDF3D80D342772090F4792EBA1923809AFDDF3EA9
0x00007FFE001B6A69 (GameAssembly) [\Library\Bee\artifacts\WinPlayerBuildProgram\il2cppOutput\cpp\Unity.DemoTeam.Hair.Runtime.cpp:16193] HairInstance_UpdateRenderingState_m960FD4D03064646506BBED4A8CED51AA88A46A23
0x00007FFE001B30E4 (GameAssembly) [\Library\Bee\artifacts\WinPlayerBuildProgram\il2cppOutput\cpp\Unity.DemoTeam.Hair.Runtime.cpp:14428] HairInstance_HandlePrerequisiteCompleted_m0CE6D3A6CA3FF2827B6961860CA7A58C7B92D6EB
0x00007FFE001B422D (GameAssembly) [\Library\Bee\artifacts\WinPlayerBuildProgram\il2cppOutput\cpp\Unity.DemoTeam.Hair.Runtime.cpp:14480] HairInstance_LateUpdate_mD2B573892A690C28007C6C6858C0E1D174E855E1

To debug this, I first began to deconstruct my scene, which was not very complex (just one animated character). Turning off the hair got rid of the crashes.

I investigated more, and turning off High Quality Line Rendering either on the hair or via HDRP settings also removed the crashes.

Further testing revealed that it was the composition mode the Line Rendering was set to.

Setting the HQ Line Rendering Composition mode to “After Temporal Antialiasing” in the HDRP settings is causing the issue (I had been using it since a custom pass was messing with the depth buffer creating artifacts on the hair. Turning off custom passes did not get rid of the crashes, however.)
(edit: see posts down below)

Here is some more information that might be useful:

  • Happens on both il2cpp and mono
  • Happens with incremental garbage collection and also without
  • Happens on Direct3D_11, Direct3D_12, but seemingly not Vulcan (which ran significantly slower)
  • Does not happen in editor (both edit and play mode). It only happens on a build
  • It didn’t seem to happen when building development builds. With release builds it would happen quickly (30 sec - 2 min).
  • Tested on Windows 10, Intel i9-12900KF, NVIDIA 3090 with latest drivers
  • Tested Unity versions: 2023.1b14 and 2023.1b20

I’ve ran the build with the ‘-debugallocator’ cmd line argument, but I couldn’t see any difference in the stack traces.

I’m not sure if I can quickly throw together and test a stripped down project for reproducing as I’m currently very busy, but I will try should it be necessary.

2

The crashes still seem to happen on other composition modes, though less frequently. I will try some things to see if it really is caused by line rendering, or if that just makes it easier for it to trigger.

EDIT:
It seems like the crashes are happening on all composition modes, but they happen a lot more frequently on “After Temporal Antialiasing”. I have gotten the crash on all modes, also while turning custom pass off.

It doesn’t seem to happen when turning off High Quality Line Renderer, still.

The crashes do not happen in a development build. I had it running for 30+ minutes, while without development build it crashes within 30 sec.

3

So… it doesn’t crash in development build.

But it does crash when I run the development build with the “-debugallocator” argument.

========== OUTPUTTING STACK TRACE ==================
0x00007FFE99189A4A (UnityPlayer) profiling::ProfilerManager::smile:isposeProfilerRecorder
0x00007FFE987FCE0B (UnityPlayer) ProfilerRecorder_CUSTOM_Control_Injected
0x00007FFE380BFD84 (GameAssembly) [\Library\Bee\artifacts\WinPlayerBuildProgram\il2cppOutput\cpp\UnityEngine.CoreModule.cpp:8952] ProfilerRecorder_Control_m523BF56A25C70F4A89D3D1FF79B2596E8F3A7873
0x00007FFE380C020B (GameAssembly) [\Library\Bee\artifacts\WinPlayerBuildProgram\il2cppOutput\cpp\UnityEngine.CoreModule.cpp:9103] ProfilerRecorder_Dispose_m6154715CD36B2CDE7B9DB390E0FEC335D75C96EB
0x00007FFE38208D0D (GameAssembly) [\Library\Bee\artifacts\WinPlayerBuildProgram\il2cppOutput\cpp\UnityEngine.CoreModule__3.cpp:32020] Recorder_Finalize_mBF7E97C65CF9B2082B073AC844783635EF623330
0x00007FFE38FC8D26 (GameAssembly) [C:\Program Files\Unity\Hub\Editor\2023.1.0b20\Editor\Data\il2cpp\libil2cpp\vm\Runtime.cpp:638] il2cpp::vm::Runtime::InvokeWithThrow
0x00007FFE38FC87C9 (GameAssembly) [C:\Program Files\Unity\Hub\Editor\2023.1.0b20\Editor\Data\il2cpp\libil2cpp\vm\Runtime.cpp:623] il2cpp::vm::Runtime::Invoke
0x00007FFE38FFAAED (GameAssembly) [C:\Program Files\Unity\Hub\Editor\2023.1.0b20\Editor\Data\il2cpp\libil2cpp\gc\GarbageCollector.cpp:200] il2cpp::gc::GarbageCollector::RunFinalizer
0x00007FFE3904FA08 (GameAssembly) [C:\Program Files\Unity\Hub\Editor\2023.1.0b20\Editor\Data\il2cpp\external\bdwgc\finalize.c:1318] GC_invoke_finalizers
0x00007FFE38FF98F0 (GameAssembly) [C:\Program Files\Unity\Hub\Editor\2023.1.0b20\Editor\Data\il2cpp\libil2cpp\gc\GarbageCollector.cpp:121] il2cpp::gc::FinalizerThread
0x00007FFE3903DA69 (GameAssembly) [C:\Program Files\Unity\Hub\Editor\2023.1.0b20\Editor\Data\il2cpp\libil2cpp\os\Thread.cpp:217] il2cpp::os::Thread::RunWrapper
0x00007FFE390286B3 (GameAssembly) [C:\Program Files\Unity\Hub\Editor\2023.1.0b20\Editor\Data\il2cpp\libil2cpp\os\Win32\ThreadImpl.cpp:29] il2cpp::os::ThreadStartWrapper
0x00007FFF66107614 (KERNEL32) BaseThreadInitThunk
0x00007FFF666A26A1 (ntdll) RtlUserThreadStart
========== END OF STACKTRACE ===========
4

Probably best to do a bug report , else chances are small to get this fixed. I already have submitted 5 critical bug reports for 2023.1, one of these is also about high quality lines. It seems to take a few weeks to month to get them fixed.

1 Like
5

I managed to reproduce the bug in a simple project with just one hair asset. It takes a bit longer for the bug to happen (probably due to less memory access), but it does happen after ~15 minutes for me.

I sent in the project with my bug report. I hope it gets addressed soon, as it pretty much makes the high quality hair unusable.

Incident number: IN-43583

1 Like
6

Could you please post here in case QA rejects the report (which often is the case, as they might not be able to do a build etc.) in that case I would also try to reproduce and submit a case, as this feature is very important for us.

7

Did you hear back from QA about the issue?

8

Nothing yet, unfortunately. Only the initial email saying that they received the report. it’s marked as “open” on atlassian.

Did you get a similar issue?

9

The bug has been rejected initially, since the reviewer wasn’t able to reproduce it. I expected as much since the build would sometimes only crash after 20 minutes, and judging from the emails between “under review” and the answer, only 30 minutes passed, so subtracting download/setup time, the build might not have been running long enough on their side.

I’ve reopened it with an improved version, in which I just copied the hair a couple of times, which as I expected did the trick.Non-development builds from this project will crash in under a minute for me.

I uploaded the project here: https://drive.google.com/file/d/18G8XO2o65B_cTzsr2CrRdVASdxzZ-owM/view?usp=sharing (there is also an already built version in there for reference)

so anyone interested can try and reproduce the issue. I used Unity 2023.1.0b20 for this.

Let me know if you can reproduce the issue, since this is pretty critical for anyone using the high quality lines.

10

Hey, I am testing the build right now. So far the build is running for like 10 minutes without a crash on a NVIDIA 3080 (not latest driver, this I’ll try next). I guess the build inside is a non dev build?

11

stopped it after running for nearly 30 minutes, no crash. I saw a slight increase in GPU memory, but this is usually not a reliable indicator (was like 200MB increase during the 30 minutes of running).

12

after updating drivers to latest, still running fine (did a 10 minutes run in fullscreen, around 24 fps, full GPU usage, around 2.2GB of GPU mem used, not really increasing over time).
Maybe it’s something with your GPU? I had issues with my first 3080, it crashed after running something full load for a while, after exchanging issue was gone.

13

last thing I tried was do a new build, this also running fine since a few minutes, no crash…

BTW, where did you get the hair asset from? I am always on the search for new hair assets, difficult to find good ones.

14

I will try to update my drivers and try again. It would be good if it was just that. I shouldn’t have ruled out the possibility of drivers being the issue. I guess I did because it was only happen on builds and not in editor, though I realize now that memory management for the hardware might be different as well.

I had the same issue as you with the hair assets. I eventually learned making hair in blender.

15

Unfortunately, updating the drivers didn’t help. Still getting crashes within like 30 sec.

========== OUTPUTTING STACK TRACE ==================

0x00007FFD40BD2965 (UnityPlayer) block_remove
0x00007FFD40BD312A (UnityPlayer) tlsf_free
0x00007FFD3FCBC193 (UnityPlayer) DynamicHeapAllocator::smile:eallocate
0x00007FFD3FCC356B (UnityPlayer) DualThreadAllocator<DynamicHeapAllocator>::TryDeallocate
0x00007FFD3FCBFC30 (UnityPlayer) MemoryManager::smile:eallocate
0x00007FFD3FF40E5D (UnityPlayer) profiler_start_new_frame
0x00007FFD3FF28C2E (UnityPlayer) `InitPlayerLoopCallbacks'::`2'::InitializationProfilerStartFrameRegistrator::Forward
0x00007FFD3FF1EEDB (UnityPlayer) ExecutePlayerLoop
0x00007FFD3FF1F078 (UnityPlayer) ExecutePlayerLoop
0x00007FFD3FF1F413 (UnityPlayer) PlayerLoop
0x00007FFD401721BB (UnityPlayer) PerformMainLoop
0x00007FFD4017451B (UnityPlayer) MainMessageLoop
0x00007FFD40177950 (UnityPlayer) UnityMainImpl
0x00007FFD40177B6B (UnityPlayer) UnityMain
0x00007FF647CE11F2 (TestHairHDRP) __scrt_common_main_seh
0x00007FFE014B7614 (KERNEL32) BaseThreadInitThunk
0x00007FFE02A426F1 (ntdll) RtlUserThreadStart

========== END OF STACKTRACE ===========
16

I just tested and it also crashed on a completely different second machine (with ATI instead of intel, and a 2080 instead of a 3090).

Did you run the build that was already in the build folder @Qleenie , or did you build it yourself. I wonder if there is something with my unity install that maybe creates builds with this bug. I’ll install and try building the project from the other machine.

17

I ran with the build unchanged in Build folder for like 30 minutes, then did a new build, which I again ran for like 10 minutes.
So it cannot be your Unity installation.

18

I’ll test your build on a few different machines with different GPU / CPU / Windows versions and see if any is giving same issue, which might help in narrowing down the reason.

1 Like
19

I tried to run it on the other machine again for like 10 minutes. Just when I thought the first crash was a fluke, it did crash again, in the same spot, though with a slightly different stack trace.

Not sure how to proceed from here. I have two machines that reproduce the crash.

One more question if you don’t mind. Are you running windows 10 or 11, since I’m still on 10. (grasping to straws right now)

20

I tested with windows 10, next test will be on laptop with windows 11 in a second.