I’m working on an online coop game, everything was mostly tested in editor until we started beta testing about 2 months ago. At that points testers started reporting crashes, upon inspection of the crash dumps, every crash comes from internal Unity code, seemingly on totally unrelated fonctions : animator, gameobject enable/disable, physx, URP, etc. All are memory access violation (I guess it’s the most common cause of crash)
I tried updating Unity, all packages, switching from mono to il2cpp, disabled code stripping as much as possible, no luck… I guess I’m doing something I shouldn’t in user code but I can’t see where I can start seaching for the cause.
There are also a lot of times where testing sessions went without crash at all for very long, then the next day the same user would crash every 30 minutes or so… Also worth noting the game never crashes in the editor during gameplay
We’re using Mirror and the steam API, here’s some of the dumps from the latest version, usable with the unity symbol server (http://symbolserver.unity3d.com)
I have absolutely no way of reproducing those crashes, and they all happened at seemingly very different state of the game. Any help welcome.
Crashes like that are usually indicative of memory corruption, especially if they happen after running the game for a while. Did you try running the game using the “-debugallocator” command line argument?
Thanks for the reply. Well it doesn’t happen specifically after running the game for a long time, it’s actually somewhat the opposite : sometimes it crashes very frequently, but most of the time everything is fine for hours. And it seem to be totally random, not associated with a particular computer or build version…
I didn’t know about -debugallocator, I can’t fine much information about it (it doesn’t seem to be listed anywhere here Unity - Manual: Command-line arguments). I’ll give it a try.
Here are two dumps with -debugallocator argument, I’m not sure what to make of it, it seems to have crashed at the same line (something in the GC finalizer?) but I have no clue how to identify the cause in my code from that. If you have any insight, that would be great ! https://drive.google.com/file/d/1S9qb17oUn8Uo6jv4syZyn15oT6eqnXFu/view?usp=drive_link
I’m using a lot of statics, delegates and events in my code, I’ve read those can cause memory leak if not handled properly, but I find very few information about memory corruption, could that be a possible cause?
Can you share GameAssembly.dll and also GameAssembly.pdb from the “_BackUpThisFolder_ButDontShipItWithYourGame” folder? You can send them to me privately if you wish.
I am not able to read the dump files properly without them - but your analysis about the finalizers may be correct. Given the callstack, you should be able to find the class name that is being finalized.
Yes, I use Profile.BeginSample and EndSample, that’s it.
I thought those were not compiled when building the game, is that not the case?
I’ll do some testing after removing them
No luck, crashes still happen with the same callstack…
Also, crashes seem to happen consistently after 10-15 minutes of gameplay with the -debugallogator argument, but without it, it crashes a lot less, sometimes not at all for hours. Is that expected behaviour? Could it be because debugallocator drastically increase memory usage (from 1Gb without to 5Gb with it)? Then could it be the debugger that is actually crashing?..
DebugAllocator works by making all out of bounds and freed memory accesses induce a crash, rather than creating a ticking time bomb that will crash at some point in the future (when the evidence of what went wrong is gone). It does increase memory usage and make memory allocation slower. So that is expected.
One thing to check: what happens if you run a non-development build with -debugallocator? Does it still crash in the same place?
I find it surprising you’re not using profiler recorder APIs (those are separate from Begin/end sample) and that this is what is causing a crash in your project. Let me talk with some folks internally.
I’m not sure how to identify if profiler recorder api is used in the project, I use several plugins that could use it. I’m thinking specifically of backtrace, a plugin that detects crashes and upload them. However, I installed this plugin because crashes were happening, so that’s not the main cause anyway, but maybe it’s doing weird things that triggers the debugallocator before the actual issue?
Edit : I tried a build without backtrace and it still crashes at the profilerrecorder finalizer, so I guess it’s not that particular plugin, I could not find any reference to ProfilerRecorder in the entire solution in visual studio, although I’m not sure that includes code in packages, and I guess it could still be used in a dll?
Actually it’s already the case, development build is not checked in the build window, and c++ compiler configuration is set to release. Is that a mistake? Should I try changing either / both of those?
I doubt it. I suspect this could be a bug in Unity itself at this point. But let me discuss with my peers first.
It could very well be, I just assumed development build and I can’t easily confirm through the dump file. If the development build watermark is not there, it’s not a development build.
I’ve tested in a dev build, with the c++ compiler config still on release, and I can confirm it crashes at the same place.
Thanks a lot for looking into it, let me know if I can test some other things to try to pin down the issue.
There is some usages of profiler recorder within URP: com.unity.render-pipelines.core\Runtime\Debugging\ProfilingScope.cs
Are you able to embed the package into your project for the time being and trying to comment out the code that constructs and operatates on the these recorder objects? Unity - Manual: Embedded dependencies
It would also be good to search the source code of all packages you’re using to figure out if any of them use the recorder too.
Alright, I just commented the #define UNITY_USE_RECORDER (lines 6 to 8) of ProfilingScope.cs, and no crash happened since.
Many thanks for helping to find a fix on the user side, before a global fix can be deployed !
I won’t be able to do that right now as I’m taking a 2 week vacation, also I can’t really attach the whole project or give step by step reprod, so I’m not sure what additionnal information I could give… I’ll try to setup a minimal project with the issue when I get back, but I have no idea if that will work.
I`m using 2022.3.50f1, and still face this problem! With your method and remove ProfilingScope usage in my custom render feature solved this random crash! Thanks a lot! @Tautvydas-Zilys@sixolar