Anyone else experience consistent editor crashes using burst/dots?

Unity Engine
, ,
1

My game is ~30k lines of code, probably 30+ burst jobs. I’ve been doing unity development for over a decade and never have gotten as many crashes as I do since I started using burst / DOTS. The stack traces in the editor logs are either nonexistent or incomprehensible to me. Most crashes occur either when I start or end the game in the editor. I average probably ~1.5 crashes per development hour, just enough to be annoying such that I’ve just lived with it the past several months. Just looking for any avenues to tackle this problem if possible.

Unity 2020.3.13f
Burst 1.4.8
Entities 0.17.0-preview.42
Jobs 0.8.0-preview.23

An example of a stack trace looks like this (I’ve also attached more logs):

Stack Trace of Crashed Thread 7820:
ERROR: SymGetSymFromAddr64, GetLastError: 'Attempt to access invalid address.' (Address: 00007FF69AFB072A)
0x00007FF69AFB072A (Unity) (function-name not available)
ERROR: SymGetSymFromAddr64, GetLastError: 'Attempt to access invalid address.' (Address: 00007FF69ADE4827)
0x00007FF69ADE4827 (Unity) (function-name not available)
ERROR: SymGetSymFromAddr64, GetLastError: 'Attempt to access invalid address.' (Address: 00007FF69E34E9E2)
0x00007FF69E34E9E2 (Unity) (function-name not available)
ERROR: SymGetSymFromAddr64, GetLastError: 'Attempt to access invalid address.' (Address: 00007FF69E71AA49)
0x00007FF69E71AA49 (Unity) (function-name not available)
ERROR: SymGetSymFromAddr64, GetLastError: 'Attempt to access invalid address.' (Address: 00007FF69E31C5C8)
0x00007FF69E31C5C8 (Unity) (function-name not available)
0x00007FFD855B217F (ntdll) _chkstk
0x00007FFD85561454 (ntdll) RtlRaiseException
0x00007FFD855B0CAE (ntdll) KiUserExceptionDispatcher

7403630–904904–Crashes.zip (1.24 MB)

2

We had this when the system did not have enough memory/powerful enough GPU but in genreal you’ll have crashes but not this many

3

Yes. We recently experimented with Burst & Jobs (not ECS) for Mirror.
Even with over a year of prior DOTS experience, Unity still crashes couple times a day.

If it crashes for me, it’ll crash for thousands of Mirror users with zero DOTS experience too.

Couple years ago I switched from C++ to Unity C# to avoid those types of crashes.
Not really interested in going down that road again.

1 Like
4

Yes, this is very similar to the experience I was having with DOTs.

5

I’ve noticed logging from jobs is pretty unstable if sustained. Not sure if that is the issue you are facing though. Otherwise I haven’t seen this level of instability personally, or if I did, it was my fault.

6

I’ve seen these crashes some time ago while there were unsafe pointers in my codebase. Buffer overflow was causing all kinds of strange and random errors (often in unity packages, confusingly enough). Migrated to safe containers since.
I suggest scanning your codebase for those, disabled safety attributes too.

1 Like
7

Ya I’ve hit this before see U5.3 - ERROR: SymGetSymFromAddr64, GetLastError: 'Attempt to access invalid address.' .

If you are corrupting memory with incorrect usage of native memory, this is very often the error you will get.

8

It varies between developer but in general we get 1-2 crashes a day reimporting scripts or assets. Never been able to pin point it down to dots though as the stacks are all over the place from mono to asset importer to the above mentioned error.

It’s not great but kind of use to it by now.

2 Likes
9

Applying -debugallocator command line argument upon project / Unity start also helps to catch native leaks.

Although its not recommended to always run with it, as its way slower than default one, but its great to have enabled when working on native containers.

1 Like
10

Actually paying attention, I’d say every single crash recently has been from addressables and the localization package entering play mode

0x00007FFFF49E4ED9 (KERNELBASE) RaiseException
0x00007FFEFC496025 (mono-2.0-bdwgc) [c:\build\output\unity-technologies\mono\mono\utils\mono-log-common.c:142] mono_log_write_logfile
0x00007FFEFC484CC3 (mono-2.0-bdwgc) [c:\build\output\unity-technologies\mono\mono\eglib\goutput.c:141] monoeg_assertion_message
0x00007FFEFC4B8528 (mono-2.0-bdwgc) [c:\build\output\unity-technologies\mono\mono\metadata\class.c:6602] mono_class_from_mono_type
0x00007FFEFC57A33F (mono-2.0-bdwgc) [c:\build\output\unity-technologies\mono\mono\metadata\reflection.c:452] mono_type_get_object_checked
0x00007FFEFC57A2D3 (mono-2.0-bdwgc) [c:\build\output\unity-technologies\mono\mono\metadata\reflection.c:430] mono_type_get_object
0x00007FF678B2D911 (Unity) scripting_class_get_object
0x00007FF67A9794DD (Unity) AssetDatabase_CUSTOM_GetMainAssetTypeAtPath
0x000001D7E5EF80AE (Mono JIT Code) (wrapper managed-to-native) UnityEditor.AssetDatabase:GetMainAssetTypeAtPath (string)
0x000001D7E5EF7E93 (Mono JIT Code) [PackageCache\com.unity.addressables@1.18.11\Editor\Settings\AddressableAssetEntry.cs:191] UnityEditor.AddressableAssets.Settings.AddressableAssetEntry:get_MainAssetType ()
// .... about 50 lines of resource manager
0x000001D7E5EF10AB (Mono JIT Code) [PackageCache\com.unity.localization@1.0.0-pre.10\Editor\UI\GameViewLanguageMenu.cs:50] UnityEditor.Localization.UI.GameViewLanguageMenu:Show ()
0x000001D7E5A2AAE3 (Mono JIT Code) [\PackageCache\com.unity.localization@1.0.0-pre.10\Editor\UI\GameViewLanguageMenu.cs:88] UnityEditor.Localization.UI.GameViewLanguageMenu:playModeStateChanged (UnityEditor.PlayModeStateChange)
11

@slims

try turning off every single option to use Burst/Jobs in the Player Settings.

See two things: what performance changes you get, and if the game crashes in this same way going in and out of PlayMode.

Graphics Jobs, OFF, was the best way for me to reduce these exact same type of crashes, but am on a Mac, not PC.

12

I don’t see many options for Burst/Jobs in the Player Settings. I have that all turned off via the menu items at the top. Graphics Jobs has been off this entire time. Also worth noting I’m not using any explicit unsafe code, and I’ve got no leak reports from native collection usages.

Crashes still continuing with no end in sight. Ready to abandon Unity after thousands of hours at this point.

13

Interesting, but I don’t even use localization in my project yet. This might be a red herring?

14

Consolation first:

Sorry you’re experiencing this.

At a similar point in my game, I got to a situation where Unity was crashing every second or third time I exited or entered Play Mode, and destroying and deleting half of whatever was in the Scene currently open. My game is quite specific in terms of where things need to be in the levels in the Scenes, as it’s physics based. So this hurt, a lot, every single time.

I begged Unity to help me, including offering to let them remote into my computer, for 6+ months, as I couldn’t find what was causing this, despite everything else working pretty much fine. I triaged like a madman.

It was, I think, something to do with a combination of driver issues with how Unity accessed GPU memory in an unsafe way causing some of the crashes, and then MacOS just kicking it out, and something odd in the Audio system. Two serious bugs.

Then, one day, a new version, and it’s all gone. Haven’t seen that behaviour once since then.

I examined the release notes and found a couple of things that I think fixed it and aligned with what I was experiencing and guessed to be happening… but Unity never once gave me any indication they were seriously looking into what I was showing them, or that they even cared.

So I’m very sorry you’re going through this, and not able to get any insight into what’s happening.

There were other things in my life, at the time, that meant that I could just bash my head against this wall, over and over again, and break up my project into smaller parts and work on other things that didn’t need me to be using full scene files etc… but I couldn’t see an end in sight and was really working on getting prepped to port to something else.

So… I dunno… maybe I’m saying… have a long hard think about what it would take to port your game. I did, and the way I’ve built since then is much more optimal as a result, even though I’m still in Unity. I learnt a LOT from getting ready to port, and my game is better for that, even though in Unity, still…

… so getting ready to port was not a waste of energy, at all!

15

This looks like an unsafe memory access is causing the OS to crash Unity.

It might not be the OS that’s crashing Unity… it may also be Unity not finding something that it needed/wanted and crashing, too.

OS and Unity
One of the scariest things I found, when trying to triage Unity’s crashes, was just how much bad behaviour of Unity’s code MacOS was willing to put up with before it would terminate it. There were hundreds of issues the OS was noting about Unity in every few minutes of Play Mode. I don’t know if Windows provides similar insights into what apps are doing, but that might help you find what’s specifically being done right before the invalid memory access gets too much for the OS and/or Unity to bear.

HARDWARE:
If it’s not isolated to your machine, if you’re seeing this behaviour on other machines with this project, then forget about the hardware issue, it’s much, much less likely to be one of these, however… if this is unique to one machine, you might want to try different RAM, a different GPU and a different power supply. One at a time, if you can, swap out each of these. As on a desktop, these three things are the most likely to cause issues when the machine is getting pushed a little.

A gremlin in a power supply can be a nightmare. This is the easiest to check first. Get a good name brand one that’s rated a bit higher than the one in your current machine. RAM can go ever so slightly wonky and be fine for a while, but because Unity recreates the entire Scene when entering PlayMode, it puts a lot of pressure on RAM when going in and out of PlayMode. Which might explain why and when this is happening to you.

And GPUs… well, they’re just fickle, sometimes. Test this last.

16

One other thing, and this might sound weird…

Unplug EVERYTHING from USB and turn off Bluetooth, and even go so far as to uninstall any exotic drivers you might have for attached devices.

Then restart your machine, and do any Registry cleaning required to truly get rid of drivers.

Then see if this is still happening. There’s another bug somewhere that has something to do with the bridge these go through, and is prevalent on both Windows and Mac when using Unity. I forget when this was first surfaced, but I got hammered by this a while ago, for a while, and have been running Unity on a purpose specific machine since, that has nothing else plugged into it, and almost no other software on it other than necessary for game production.

This might sound weird, but back in the old days, we’d build specific workstations for specific software. And Unity has now become so brittle that it’s somewhat necessary to do this to remain sane with it.

17

This behaviour happens on both developer machines my team uses. My machine has an AMD threadripper 3960x/3080, the other machine is an i9/2080. Crashes are consistent across both machines.

To your other point about upgrading, maybe 2021.2 will support entities/dots again, but they told us to stay on this LTS version for the foreseeable future.

And the funny thing is the current iteration IS a port from the non DOTS implementation of the game we did from 2017-2019. But going back is impossible at this point. It’s a factorio-like and without burst there’s no way we can compete performance wise. So if I had to port, I’d probably have to move entirely to unreal and rewrite the game completely.

18

Great… sort of… that narrows it down to being a Unity bug.

This is a step forward, even if it doesn’t feel like it.

I’m aware of the issues with the stall-out of DOTS and versions. I was super [un]lucky that they fixed most of my issues in later versions of 2019.2, so am now using 2019.LTS, and they’ll have to pry that from my cold, dead hands. I’ve tried, several times, to go further, as I’m reliant on deltaTime, which is reported more accurately in 2020.2 onwards, but there’s other bugs in it that are showstoppers for me.

So I know your pain, is what I’m saying.

Get ready to port in terms of all the things that are easy to port. That means sounds, shaders, graphics, models, etc. Tidying them up and making them ready to move over won’t be a waste of time, you’ll get benefit from this even if you stay. And you’ll likely find all sorts of things you can do to improve performance whilst doing this. I did!

DOTS seems to be in a very long limbo. They’ve said the earliest they’ll be back onstream is the end of the year… which probably means the end of next year, if their past abilities with schedules are anything to go by.

Also, reach out to Jackson Dunstan, he might be able to help. He’s a GEM of a human. Very, very special levels of insight and understanding of Unity foibles.

19

Judging by the crash logs, other suggestions:

  • See if it crashes outside of editor (e.g. in build), run a stability test for a few hours;
  • Check for memory leaks / jobs / native containers;
  • Unplug tablet, or similar hardware.
  • Check if you have enough space on the hard drive for the pagefile;
  • If you haven’t reported crash to via bug reporting tool - I’d highly suggest you to do so, since UT may have other useful insights on what’s going on.
  • Entities package do not like when you run with a bunch of errors in ECBs, so I’d suggest fixing those as soon as possible.

I do not have crash issues, and I’m running 2020.3.15f1 with a bunch of bursted jobs as well.
Only times when Unity’s crashed for me was memory corruption (caused by my own stupidity and unsafe pointer accesses), and native container memory leaks.

1 Like
20

Well my advice is don’t try random shit start using a methodical approach. Like start logging to narrow down where it hits. It’s not random and there is a trigger.

It really shouldn’t take you more then a day or two to narrow this down to is it your code, an asset, a package, or the engine. Just using logging. Especially if it’s usually on start/stop that already narrows it down for you quite a bit.

The obvious is that most people aren’t crashing this much. So the next obvious thing is you are doing something outside the norm. That’s not a good/bad judgement it’s just the math of it. So start setting stuff to defaults, disabling stuff you don’t really need. Like baked lighting would be an obvious thing to disable. Or going back to default graphics settings on the low/medium end. It’s all just a process of elimination and the goal is to find the biggest bang for buck, what removes the largest surface area with the least amount of work.

2 Likes