Burst corruption

I’ve been getting a lot of burst cache corruption again. Current versions
Unity 2022.2.0f1
Burst 1.8.2

They manifest like this

NullReferenceException: System.NullReferenceException: Object reference not set to an instance of an object
This Exception was thrown from a function compiled with Burst, which has limited exception support.
0x00007ff7026706fd (Unity) burst_signal_handler
0x00007ff809428b4c (ntdll) RtlDeleteAce
0x00007ff8094012c6 (ntdll) RtlRaiseException
0x00007ff809450f4e (ntdll) KiUserExceptionDispatcher
0x00007fff921e84cf (9c33427477f2fcf9f52aa87bcd77068) Unity.Physics.Systems.CreateJacobiansSystem::Unity.Physics.Systems.CreateJacobiansSystem.OnUpdate (at I:/Documents/BovineLabs/com.bovinelabs.shattered/Library/PackageCache/com.unity.physics@1.0.0-pre.15/Unity.Physics/ECS/Base/Systems/UnityPhysicsSimulationSystems.cs:271)

But they are popping up everywhere, including code gen files

NullReferenceException: System.NullReferenceException: Object reference not set to an instance of an object
This Exception was thrown from a function compiled with Burst, which has limited exception support.
0x00007ff7026706fd (Unity) burst_signal_handler
0x00007ff809428b4c (ntdll) RtlDeleteAce
0x00007ff8094012c6 (ntdll) RtlRaiseException
0x00007ff809450f4e (ntdll) KiUserExceptionDispatcher
0x00007fff92157e4b (bc02d18ffb1d97782999fc13c0444f5) BovineLabs.Game.SubScenes.LoadWithBoundingVolumeSystem::BovineLabs.Game.SubScenes.LoadWithBoundingVolumeSystem.OnUpdate (at I:/Documents/BovineLabs/com.bovinelabs.shattered/Temp/GeneratedCode/BovineLabs.Game/LoadWithBoundingVolumeSystem__System_502371246.g.cs:33)

This eventually leads to an editor crash after a lot of spam

I can fix it by

  • turning off burst (obviously not ideal)
  • closing editor, deleting burst cache, opening editor
  • turning safety off (and optionally on again) to force a burst recompile
1 Like

Hi @tertle , do you have a consistent way of reproducing this? Or have noticed any specific circumstances that trigger this?

I seem to reproduce it quite frequently in one of my projects and I think it’s somewhat related to codegen (maybe even netcode codegen specifically) and recompiling during burst compiling this.

Sorry I was meant to follow this up more since this post but I have a massive milestone due tomorrow so I haven’t had time. I’ll try find a somewhat consistent repo this weekend in project.

No repo yet, but I took note of exactly what happened on the most recent one

8646618--1163130--upload_2022-12-9_7-39-36.png

On a recompile I got an infinite wait on entering play mode, force breaking inside unity shows it’s waiting on an native AssetDatabase operation so I can’t investigate further.

Force closing unity at this point and restarting produces a corrupted burst cache and requires a second unity restart to fix.

@MarcoPersson ok so the most common way I get this is

  • after a script change, hit play while unity is still recompiling
  • sometimes (reasonably often for me) it causes an editor lockup during subscene loading/baking
  • force close editor via task manager
  • start project again

Burst is now corrupted

Hooking up a debugging during the lockup it seems stuck on

AssetDatabasExperimental.ProduceArtifact()

So yeah, not sure how much of this is on burst team vs entity team vs asset team since I think a lockup in assetdatabase / baking

I’m encountering a similar problem. Editor locks up. Cursor is invisible, no loading bar taking focus. Attaching a debugger and I see a thread with the same callstack as this (AssetDatabaseExperimental.ProduceArtifacts).

I’m not seeing any burst errors after an editor restart. But I am seeing this one, although it doesn’t seem correlated with the editor freezing.

From the stack-trace you posted, that part on its own at least doesn’t seem to be directly related to Burst.
I’d recommend reporting the freeze bug so my AssetDatabase colleagues can have a look at it.
If there’s still a Burst bug hiding there I’d gladly have a look at it if I can repro it

Do note that people are starting to go on holidays now, so you might not see much movement until after new-years though:)

1 Like

So while the error I posted above is definitely an issue, I think there is more going on here.
I’m finding my function pointers being scrambled on domain reload

DivideByZeroException: System.DivideByZeroException: Attempted to divide by zero
This Exception was thrown from a function compiled with Burst, which has limited exception support.
...
Shattered.Game.States.OptionsStateSystem.__codegen__OnCreate(System.IntPtr self, System.IntPtr state) -> void_aef4d46f9ec6d18a03913cfa3cd76414 from Shattered.Game, Version=0.0.0.0, Culture=neutral, PublicKeyToken=null (at I:/Documents/BovineLabs/com.bovinelabs.shattered/Library/PackageCache/com.unity.burst@1.8.2/.Runtime/unknown/unknown:0)
object:wrapper_native_000001B790D8A8C0 (intptr,intptr)
Unity.Entities.SystemBaseRegistry:ForwardToManaged (intptr,Unity.Entities.SystemState*,void*) (at I:/Documents/BovineLabs/com.bovinelabs.shattered/Library/PackageCache/com.unity.entities@1.0.0-pre.15/Unity.Entities/SystemBaseRegistry.cs:363)
Unity.Entities.SystemBaseRegistry:CallForwardingFunction (Unity.Entities.SystemState*,int) (at I:/Documents/BovineLabs/com.bovinelabs.shattered/Library/PackageCache/com.unity.entities@1.0.0-pre.15/Unity.Entities/SystemBaseRegistry.cs:311)
Unity.Entities.SystemBaseRegistry:CallOnDestroy (Unity.Entities.SystemState*) (at I:/Documents/BovineLabs/com.bovinelabs.shattered/Library/PackageCache/com.unity.entities@1.0.0-pre.15/Unity.Entities/SystemBaseRegistry.cs:381)
Unity.Entities.World:smile:estroyAllSystemsAndLogException () (at I:/Documents/BovineLabs/com.bovinelabs.shattered/Library/PackageCache/com.unity.entities@1.0.0-pre.15/Unity.Entities/World.cs:1011)
Unity.Entities.World:smile:ispose () (at I:/Documents/BovineLabs/com.bovinelabs.shattered/Library/PackageCache/com.unity.entities@1.0.0-pre.15/Unity.Entities/World.cs:306)
Unity.Entities.World:smile:isposeAllWorlds () (at I:/Documents/BovineLabs/com.bovinelabs.shattered/Library/PackageCache/com.unity.entities@1.0.0-pre.15/Unity.Entities/World.cs:332)
Unity.Entities.DefaultWorldInitialization:smile:omainUnloadOrPlayModeChangeShutdown () (at I:/Documents/BovineLabs/com.bovinelabs.shattered/Library/PackageCache/com.unity.entities@1.0.0-pre.15/Unity.Entities/DefaultWorldInitialization.cs:101)

OnCreate in a system somehow got called during on a domain reload destroying the world.
It should have been calling OnDestroy but invoked OnCreate - it’s like all my burst function pointers are getting muddled on domain reload.

-edit-

even weirder, is this gets called even if I’ve never entered play mode to create the system and I’ve verified it isn’t in the editor world. Thoroughly confused what is going on.

OnCreate in a system somehow got called during on a domain reload destroying the world.
It should have been calling OnDestroy but invoked OnCreate - it’s like all my burst function pointers are getting muddled on domain reload.

Yes, this is indeed caused by a bug in Burst - it took us a while to track down - quite a “fun” one :slight_smile: It was exactly what you said - function pointers got mixed up after a domain reload. The fix will be in the next version of Burst.

5 Likes

Thank you for finding this. It has been a thorn in my side, completely demotivating me. It’s not really feasible for me to leave burst enabled at the moment due to the frequency of crashes in my project, so this fix can’t come soon enough.

So I believe there is still a (very rare) corruption that is still possible

What you’re seeing here is a bursted OnUpdate in a system that only exists at runtime

BovineLabs.Netcode.States.Client.ClientJoinGameStateSystem.__codegen__OnUpdate (System.IntPtr self, System.IntPtr state) <0x24e762387d0 + 0x0005a> in <53584b9e4565462ea4644f8b6cf28928>:0

Being called from a baker

Unity.Entities.BakingUtility:postprocessBake(World, BakingSettings, BakingSystem) (at ./Library/PackageCache/com.unity.entities@e98ed4d1fd/Unity.Entities.Hybrid/Baking/BakingUtility.cs:141)

These are the filters for the ClientJoinGameStateSystem, clearly no baking flag

[UpdateInGroup(typeof(SimulationSystemGroup), OrderFirst = true)]
[WorldSystemFilter(
WorldSystemFilterFlags.ClientSimulation | WorldSystemFilterFlags.ThinClientSimulation,
WorldSystemFilterFlags.ClientSimulation | WorldSystemFilterFlags.ThinClientSimulation)]

Unfortunately I have no repo

Potentially related to

Which I see slightly more often

Unity 2022.3.15f1 - seems still not resolved the burst bug?
Note: I don’t use any jobs nor Burst Compile in my code.

I have the following settings:
Project Settings>Burst AOT Settings>Enable Burst Compilation DESELECTED
Project Settings>Editor>Enter Play Mode Settings>Enter Play Mode Options ON
Project Settings>Editor>Enter Play Mode Settings>Reload Domain ON
Jobs>Burst>Safety Checks ON
Jobs>Burst>Enable Compilation DESELECTED

If Project Settings>Burst AOT Settings>Enable Burst Compilation is SELECTED don’t see the bugs anymore (CORRECTION UPDATE: Still occurs regardless of whether Selected or not), so can I assume that even if not using any Burst Compile or Jobs in my code I still need to have Enable Burst Compilation SELECTED?

(Below,errors when Enable Burst Compilation DESELECTED)

InvalidCastException: Specified cast is not valid.
(wrapper castclass) System.Object.__castclass_with_cache(object,intptr,intptr)
System.Runtime.InteropServices.Marshal.GetDelegateForFunctionPointer[TDelegate] (System.IntPtr ptr) (at <7ec8e29954a6455daa48484a381ec418>:0)
Unity.Burst.FunctionPointer`1[T].get_Invoke () (at ./Library/PackageCache/com.unity.burst@1.8.11/Runtime/FunctionPointer.cs:68)
Unity.Entities.Content.RuntimeContentManager.ProcessQueuedCommands$BurstManaged () (at ./Library/PackageCache/com.unity.entities@1.0.11/Unity.Entities/Content/RuntimeContentManager.cs:706)
Unity.Entities.Content.RuntimeContentManager+ProcessQueuedCommands_00001AC3$BurstDirectCall.Invoke () (at <464ba5a980eb4cf189a68ccd0ba6ad41>:0)
Unity.Entities.Content.RuntimeContentManager.ProcessQueuedCommands () (at ./Library/PackageCache/com.unity.entities@1.0.11/Unity.Entities/Content/RuntimeContentManager.cs:675)
Unity.Entities.Content.RuntimeContentSystem.Update () (at ./Library/PackageCache/com.unity.entities@1.0.11/Unity.Entities/Content/RuntimeContentSystem.cs:94)
UnityEditor.EditorApplication.Internal_CallUpdateFunctions () (at <64861216782042de93237b5c061bff1b>:0)

Hi @creativelefty ,Thanks for sharing this is happening to you.

That looks like odd behavior indeed. Is it possible for you to make an official bug report regarding this behavior, and include your project that recreates it? (Just in case you haven’t made official Unity bug reports before here’s how one goes about it)
That way we can take a proper look at it!

1 Like

Encountering this exact issue, any updates?

Hi @alexchisholm343

Thank you for writing!

If I remember correctly, I don’t believe we received a bug-report with details needed to investigate this bug much further (since its reappearance). Could you possibly please make an official bug-report for us (including the project that reproduces it) so we can take a proper look at it? :slight_smile:

We’d love to squash this bug, especially as now several of you have mentioned it happening to you and disrupting your workflow.

Thanks in advance!

PS: Just out of curiosity, which version of Burst and Unity are you on? And if you’re not on latest (Burst version 1.8.15) does it still happen if you upgrade to latest?

Yes, I am seeing this issue as well. Came here to see if it had been resolved, looks like its still happening on Burst v1.8.19.

I’m able to reproduce the issue by just performing domain reloads, particularly by editing a code file (doing some small change like adding a comment or some whitespace), then saving it. With each domain reload, there is a chance that on playing the scene, a null reference exception error occurs (that really shouldnt happen). Other times, it works as expected.

I have a Github repo to reproduce the issue in, but it is a repo used for my personal project. It has private access. If you give me an email that I can use to add you to it to access it, I can go ahead and give you an environment to reproduce it in. I dont reallllly want to just make the project completely public :slight_smile:

What’s weird though is that I never saw this issue before in my burst compiled code when I was using SSE intrinsics. As soon as I started trying to implement code using AVX/AVX2, I saw the issue appear :thinking:

My CPU is Ryzen 7 7800X3D, so I dont think it is some hardware incompatibility.

Hi @wshope !

That would be great :slight_smile: I’ll send you a DM.

I just realized (and I double checked internally with our teams to be absolutely certain) that if you make an official bug report via Help → Report a Bug, that will automatically include the project you have opened and send it to us (with the repro-steps you’ve included). So you don’t have to make your repo public or add us to your private repo :slight_smile:

Sent you a DM with sharing this in case you’d miss it in the thread, and also sharing it here in case anyone else is wondering!

So if you can could you please make a report via Help → Report a Bug? Then we’ll take a look as soon as possible

Sure, I’ll go ahead and cut a help ticket for it.

Edit: I didnt realize you could report bugs directly in the Unity editor. I was trying to find the Help link in the forums here :man_facepalming:

I went ahead and reported the bug. Thanks for your help!