Hey,
I am a bit worried that this issue or the severity of this issue of the topic might be overlooked, as the main thread is in a now archived forum:
and the description of the issue by QA seems to be rather cryptic:
The main issue is that builds with the new (2023.1+) feature “High Quality Lines” will crash after a certain amount of time, which can be less than a minute, depending on the hardware and the amount of lines on screen.
The nature of the crash seems mostly random and will only happen on release builds (not on development builds).
Because of this bug, it’s impossible to release any builds that use the new high definition lines feature. Furthermore, it will cost devs a lot of time trying to debug this bug, because there is no error or anything they can type into google. Initially, I only found the cause by painstakingly deactivating feature by feature. Thanks to Qleenie, we were able to confirm that it is reproducible on a lot of (but not all) hardware configurations.
I was able to reproduce on all of my hardware configs (4 different ones, including AMD + NVIDIA GPUs and AMD + Intel CPUs). Only thing was that on one configuration (3080 + Ryzen) it takes much longer until the crash occurs, for whatever reason this might be. On all other configs builds crashed in less than one minute.
I just got a crash on a dev-build. I let it run over night to see if at least dev-builds were stable. According to the last access to the log, it took around 6 hours for the crash to happen.
Here is the stack trace (the most common trace on tlsf_free):
========== OUTPUTTING STACK TRACE ==================
0x00007FFCB344D4CB (UnityPlayer) tlsf_free
0x00007FFCB1AC7CFD (UnityPlayer) DynamicHeapAllocator::smile:eallocate
0x00007FFCB1AC8306 (UnityPlayer) MemoryManager::smile:eallocate
0x00007FFCB1AD012E (UnityPlayer) free_alloc_internal
0x00007FFCB2118923 (UnityPlayer) profiling::ProfilerManager::StartNewFrame
0x00007FFCB21408B8 (UnityPlayer) profiler_start_new_frame
0x00007FFCB20DE5C4 (UnityPlayer) `InitPlayerLoopCallbacks'::`2'::InitializationProfilerStartFrameRegistrator::Forward
0x00007FFCB20CC63A (UnityPlayer) ExecutePlayerLoop
0x00007FFCB20CC70E (UnityPlayer) ExecutePlayerLoop
0x00007FFCB20D0A5A (UnityPlayer) PlayerLoop
0x00007FFCB2665CE9 (UnityPlayer) PerformMainLoop
0x00007FFCB26648CB (UnityPlayer) MainMessageLoop
0x00007FFCB266AD59 (UnityPlayer) UnityMainImpl
0x00007FFCB266CB6B (UnityPlayer) UnityMain
0x00007FF7220111F2 (D-Sim) __scrt_common_main_seh
0x00007FFE014B7614 (KERNEL32) BaseThreadInitThunk
0x00007FFE02A426F1 (ntdll) RtlUserThreadStart
========== END OF STACKTRACE ===========
Just crashed again in a dev build. This time after only a few minutes. It seems like my initial notion of it not happening on dev builds is definitely not true, even though it does happen more rarely.
Hey, @schema_unity !
We are seeing very similar stacktraces and random crashes in our build, but we do not use High Quality Lines feature, we use URP and out Unity version is 2022.2.2, and the same random crash happens in 2022.3.2 version as well. Are you sure that the crash completely disappears if you disable the feature or use a previous Unity version? Sometimes we got a crash after 2 minutes running, and sometimes its 2+ hours.
I’m attaching some callstacks we got in our crashes:
We looked at you description of the problem, it does look identical, but only our crashes are not related to High Quality Lines feature.
@schema_unity and myself did many in depth tests, and the crash we see is clearly related to the HDRP and 2023.1+ only feature “High Quality Lines”. If feature is turned off = no crash. Also some of the stack traces we collected in the other thread directly point to code of High Quality Lines. There is a slim chance that it’s an interaction between this feature and another Unity wide bug, but this sounds a bit unlikely.
Thank you for your swift reply! It so weird that we have the same stack traces and behavior of the problem, but different crashes reason. We will continue to follow your progress with the issue just in case the root of the problem is similar.
Those stack traces are generally an indication of memory allocation failing in some way, as if memory was allocated once and then the same memory is being freed twice. When I first researched the trace, I found issues of this happening in different unity versions on different features. It’s possible that multiple features might introduce similar issues though the root causes are not necessarily related (there is still a small chance as @Qleenie said, like it might be some new memory utility used for multiple features or something).
If you can get it to crash reliably, you could try disabling different features one by one until the crashes stop to narrow down the problem. If you find something, I’d be happy to test it on my machines to confirm.
I have some updates @schema_unity @Qleenie
Even though we had random memory-free stack traces when crashing release builds, we tried the thing you did with development build and debugallcator arguments ang got exactly the same result as you wrote in a previous thread and the same stack Imgur: The magic of the Internet
Then we narrowed the issue down to our custom render features, the code that was causing the problem was the “new ProfilingSampler” call we did every frame. Caching the call or removing the profiler calls from the code fixed the issue.
We managed to reproduce the crash in an empty project with a test code that does exactly this. I’m attaching the code we’ve used to crash the builds.
Also, we noticed that this code can even crash the editor after some time with the same stack, especially if the editor runs with attributes -debugallocator -systemallocator.
We will be reporting this bug today with the repo project. I will update the thread with the issue number, just in case those issues are related.
PS. I forgot to add that it is also a necessary step to reproduce the crash in the empty project - constantly allocate any garbage somewhere and invoke GC.Collect() and Resources.UnloadUnusedAssets.
that’s indeed very interesting! Seems that in the code pointed at by the stacktrace the HQ line renderer is doing the following:
using (new ProfilingScope(cmd, new ProfilingSampler("Vertex Setup")))
I hope Unity will fix the issue fast, if not I’ll probably do a branch of HDRP and remove the calls.
PS: That’s the code which probably causes the crash:
https://github.com/Unity-Technologies/Graphics/blob/master/Packages/com.unity.render-pipelines.high-definition/Runtime/RenderPipeline/LineRendering/Core/RenderPass/LineRendering.Pass.Geometry.cs
Very nice find @MariaKhomenko . And great work finding the possible cause in HDLines @Qleenie
It gives me quite a big relief knowing that I can now probably create a fix should the error not be fixed in time for release builds.
Hey @MariaKhomenko , did you report the bug, and if yes, do you have a link to the issuetracker yet?
Hey, yeah! We did report it right that day I wrote you, issue number IN-48616 but I think it is not open to public yet? Not sure the link will work. The status is still “in review”, but it was reproduced by unity internally in the editor.
Ah, thanks! You’ll get a public link as soon as it gets accepted. Might take a few more days, things are going slow due to summer probably.
almost one month and not a word from unity?
Hey, @Qleenie and @impheris , today we got a reply from Unity, stating “Your bug report has been confirmed and transferred to the appropriate internal development team at Unity. Your bug report has the following internal ID: UUM-44946”
There is still no public link so I will keep you posted.
Here is the public link
According to issuetracker it should be fixed with this weeks update of 2023.1.11.