EditorFreeze - BurstCompiler.CompileFunctionPointer (synchronous, 1.7.0)

OndrejP · May 11, 2022, 12:40pm

I’ve upgraded project to Unity 2022.1.0f1, which auto-upgraded Burst package from 1.6.5 to 1.7.0.

When I enable SynchronousCompilation and try to compile any burst function with BurstCompiler.CompileFunctionPointer Editor freezes completely (10+ minutes waiting, then kill). There’s no error in the log file.

When it actually happens?

when called from OnGUI in EditorWindow
when called from Awake in script on GameObject in scene, happens when entering Play Mode

When I upgrade to 1.7.1 or 1.8.0-pre.1, same issue.
When I downgrade to 1.6.5, everything works is fine.
So I’ll be staying on 1.6.5 for now.

Why Synchronous compilation?

I have some benchmarks comparing performance of Burst to .NET
Sometimes I’m running profiler and want to make sure Burst is used

Why CompileFunctionPointer and not jobs?

simplicity of benchmarks
in few other cases it’s faster: whole execution takes 0.005ms, job overhead as measured in 2021 was ~0.015ms

@xoofx Any idea what might be causing this?

OndrejP · May 12, 2022, 7:29am

I’m trying to create small repro project, but when I copy just the scripts with BurstCompile and all their dependencies it works. So it seems something else is influencing that…does anyone have any idea what might be an issue?

When I attach debugger and pause I can’t see the Main thread, but I see that all 5 Burst compiler threads are waiting for work (ConcurrentQueue.Take) - I have six core CPU, so I’m guessing 5 threads is all of them.

The control flow enters BurstCompiler.CompileFunctionPointer, but never comes out (log calls before/after).

OndrejP · May 12, 2022, 7:55am

When I step-in with debugger, I can see that the freeze happens in Unity.Burst.LowLevel.BurstCompilerService.CompileAsyncDelegateMethod
(external method implemented as free function in Unity Engine)

That’s deepest I can get. There’s screenshot from Thread window showing that compiler workers are doing nothing.

Native debugger screenshot, main thread is waiting for something:

sheredom · May 12, 2022, 10:31am

Could you run Unity with an environment variable set for us? If you set UNITY_BURST_DEBUG=1, then Burst should generate some extra logs in the /Logs folder that’ll help us try and work out what this is. My guess is that we’re not compiling a job/function-pointer that you are synchronously relying on being there, and the main thread hangs. But this will help us confirm it.

OndrejP · May 12, 2022, 11:16am

No problem, logs attached

What I did:

Started UnityEditor (I’ve noticed message about burst logging in console)
Pressed Play
Editor froze
Waited couple minutes
Killed Editor
Grabbed log files

Burst tries to compile this method and freezes: SubTriggerBox.RaycastBatch

This is probably what you’re looking for:

2022-05-12 13:08:45.145 Handling request for `e9dea85fd1367d0216e33aa7e71b6c33` `SubTriggerBox, Volcanoids, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null::RaycastBatch(SubTriggerRaycast*, Volcanoids, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null|System.Int32, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)` with options `--platform=Windows --backend=burst-llvm-12 --global-safety-checks-setting=On --dump=Function --float-precision=Standard --enable-synchronous-compilation --log-timings `
2022-05-12 13:08:45.145 Storing function pointer request
2022-05-12 13:08:45.146 Waiting synchronously for `e9dea85fd1367d0216e33aa7e71b6c33` `SubTriggerBox, Volcanoids, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null::RaycastBatch(SubTriggerRaycast*, Volcanoids, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null|System.Int32, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)`

…and that’s the end of the burst-client.log

8122064–1052798–BurstLogs.zip (21 KB)
8122064–1052801–EditorLog.zip (9.11 KB)

OndrejP · May 12, 2022, 11:25am

Oh and there’s bunch of these errors, seems like Burst has some problems when trying to find some methods in a assembly which contains unsafe code. Maybe it can’t handle when there’s TypedReference or other weird unsafe feature?

I mean I’m not expecting Burst to compile anything with TypedReference, but I’m expecting it not to crash when it compiles a method which is in the same assembly as other method which uses TypedReference.

I’ll try to comment out certain parts of the code to confirm it.

2022-05-12 13:18:20.482 Executing job Find Methods (Samples.CosmeticDevice)
2022-05-12 13:18:20.484 Using existing .bhc file for c:\Data\Projects\Volcanoids\Library\ScriptAssemblies\Samples.CosmeticDevice.dll with timestamp 5/12/2022 7:22:29 AM and GUID fb6c87b6-5107-4a57-888a-9b4ab88c6fcb
2022-05-12 13:18:20.536 No existing .bhc file for c:\Data\Projects\Volcanoids\Library\ScriptAssemblies\Rock.Base.dll
2022-05-12 13:18:20.574 Exception while executing job: System.AggregateException: One or more errors occurred. (Unimplemented Instruction RefAnyVal) ---> System.NotImplementedException: Unimplemented Instruction RefAnyVal
                          at Burst.Compiler.IL.Hashing.CacheBuilder.ILHasher+InstructionHasher.DecodeInstruction (System.Reflection.Metadata.BlobReader& reader) [0x00509] in <660bf86525254f05bf0505fe2843d6a3>:0
                          at Burst.Compiler.IL.Hashing.CacheBuilder.ILHasher.HashMethodDefinition (System.Reflection.Metadata.MethodDefinitionHandle methodDefinitionHandle, Burst.Compiler.IL.Hashing.CacheBuilder.Util.CachingMetadataReader metadataReader, System.Reflection.PortableExecutable.PEReader peReader, Burst.Compiler.IL.DebugInfo.PortablePdbInstance pdbInstance, Burst.Compiler.IL.Hashing.ILSignatureProvider signatureProvider, Burst.Compiler.IL.Hashing.Storage.BlobCollection blobCollection) [0x00086] in <660bf86525254f05bf0505fe2843d6a3>:0
                          at Burst.Compiler.IL.Hashing.CacheBuilder.ILHasher+<>c__DisplayClass12_0.<HashImpl>b__4 (System.Reflection.Metadata.MethodDefinitionHandle methodDefinitionHandle, System.Threading.Tasks.ParallelLoopState loopState, System.Int64 i) [0x00000] in <660bf86525254f05bf0505fe2843d6a3>:0
                          at System.Threading.Tasks.Parallel+<>c__DisplayClass44_0`2[TSource,TLocal].<PartitionerForEachWorker>b__1 (System.Collections.IEnumerator& partitionState, System.Int32 timeout, System.Boolean& replicationDelegateYieldedBeforeCompletion) [0x00132] in <f4694e4600d74e03a4e35def8b97021c>:0
                        --- End of stack trace from previous location where exception was thrown ---
                       
                          at System.Threading.Tasks.Parallel+<>c__DisplayClass44_0`2[TSource,TLocal].<PartitionerForEachWorker>b__1 (System.Collections.IEnumerator& partitionState, System.Int32 timeout, System.Boolean& replicationDelegateYieldedBeforeCompletion) [0x0025f] in <f4694e4600d74e03a4e35def8b97021c>:0
                          at System.Threading.Tasks.TaskReplicator+Replica`1[TState].ExecuteAction (System.Boolean& yieldedBeforeCompletion) [0x00000] in <f4694e4600d74e03a4e35def8b97021c>:0
                          at System.Threading.Tasks.TaskReplicator+Replica.Execute () [0x00023] in <f4694e4600d74e03a4e35def8b97021c>:0
                           --- End of inner exception stack trace ---
                          at System.Threading.Tasks.TaskReplicator.Run[TState] (System.Threading.Tasks.TaskReplicator+ReplicatableUserAction`1[TState] action, System.Threading.Tasks.ParallelOptions options, System.Boolean stopOnFirstFailure) [0x0005b] in <f4694e4600d74e03a4e35def8b97021c>:0
                          at System.Threading.Tasks.Parallel.PartitionerForEachWorker[TSource,TLocal] (System.Collections.Concurrent.Partitioner`1[TSource] source, System.Threading.Tasks.ParallelOptions parallelOptions, System.Action`1[T] simpleBody, System.Action`2[T1,T2] bodyWithState, System.Action`3[T1,T2,T3] bodyWithStateAndIndex, System.Func`4[T1,T2,T3,TResult] bodyWithStateAndLocal, System.Func`5[T1,T2,T3,T4,TResult] bodyWithEverything, System.Func`1[TResult] localInit, System.Action`1[T] localFinally) [0x0017c] in <f4694e4600d74e03a4e35def8b97021c>:0
                        --- End of stack trace from previous location where exception was thrown ---
                       
                          at System.Threading.Tasks.Parallel.ThrowSingleCancellationExceptionOrOtherException (System.Collections.ICollection exceptions, System.Threading.CancellationToken cancelToken, System.Exception otherException) [0x0000b] in <f4694e4600d74e03a4e35def8b97021c>:0
                          at System.Threading.Tasks.Parallel.PartitionerForEachWorker[TSource,TLocal] (System.Collections.Concurrent.Partitioner`1[TSource] source, System.Threading.Tasks.ParallelOptions parallelOptions, System.Action`1[T] simpleBody, System.Action`2[T1,T2] bodyWithState, System.Action`3[T1,T2,T3] bodyWithStateAndIndex, System.Func`4[T1,T2,T3,TResult] bodyWithStateAndLocal, System.Func`5[T1,T2,T3,T4,TResult] bodyWithEverything, System.Func`1[TResult] localInit, System.Action`1[T] localFinally) [0x001db] in <f4694e4600d74e03a4e35def8b97021c>:0
                          at System.Threading.Tasks.Parallel.ForEachWorker[TSource,TLocal] (System.Collections.Generic.IEnumerable`1[T] source, System.Threading.Tasks.ParallelOptions parallelOptions, System.Action`1[T] body, System.Action`2[T1,T2] bodyWithState, System.Action`3[T1,T2,T3] bodyWithStateAndIndex, System.Func`4[T1,T2,T3,TResult] bodyWithStateAndLocal, System.Func`5[T1,T2,T3,T4,TResult] bodyWithEverything, System.Func`1[TResult] localInit, System.Action`1[T] localFinally) [0x00050] in <f4694e4600d74e03a4e35def8b97021c>:0
                          at System.Threading.Tasks.Parallel.ForEach[TSource] (System.Collections.Generic.IEnumerable`1[T] source, System.Threading.Tasks.ParallelOptions parallelOptions, System.Action`3[T1,T2,T3] body) [0x0002a] in <f4694e4600d74e03a4e35def8b97021c>:0
                          at Burst.Compiler.IL.Hashing.CacheBuilder.ILHasher.ParallelForEach[T] (System.Collections.Generic.IEnumerable`1[T] elements, System.Action`3[T1,T2,T3] callback) [0x00018] in <660bf86525254f05bf0505fe2843d6a3>:0
                          at Burst.Compiler.IL.Hashing.CacheBuilder.ILHasher.HashImpl () [0x001c2] in <660bf86525254f05bf0505fe2843d6a3>:0
                          at Burst.Compiler.IL.Hashing.CacheBuilder.ILHasher.HashAssembly (System.String filePath) [0x00007] in <660bf86525254f05bf0505fe2843d6a3>:0
                          at Burst.Compiler.IL.Hashing.CacheRuntime.HashCacheLoader.CreateHashingResult (Mono.Cecil.AssemblyNameReference assemblyNameReference, Burst.Compiler.IL.AssemblyLoader assemblyLoader, Burst.Compiler.IL.Helpers.DebugLogWriter debugLogWriter) [0x00160] in <660bf86525254f05bf0505fe2843d6a3>:0
                          at Burst.Compiler.IL.Hashing.CacheRuntime.HashCacheAssemblyStore.GetAssemblyState (System.String assemblyName, Burst.Compiler.IL.AssemblyLoader assemblyLoader) [0x000a7] in <660bf86525254f05bf0505fe2843d6a3>:0
                          at Burst.Compiler.IL.Server.EntryPointMethodFinder.LoadAllAssemblyReferences (Mono.Cecil.AssemblyDefinition asmDef, System.Collections.Generic.Dictionary`2[TKey,TValue] visited, System.Collections.Generic.Dictionary`2[TKey,TValue] jobProducers, Burst.Compiler.IL.Hashing.CacheRuntime.HashCacheAssemblyStore assemblyStore, Burst.Compiler.IL.AssemblyLoader assemblyLoader) [0x00020] in <660bf86525254f05bf0505fe2843d6a3>:0
                          at Burst.Compiler.IL.Server.EntryPointMethodFinder.LoadAllAssemblyReferences (Mono.Cecil.AssemblyDefinition asmDef, System.Collections.Generic.Dictionary`2[TKey,TValue] visited, System.Collections.Generic.Dictionary`2[TKey,TValue] jobProducers, Burst.Compiler.IL.Hashing.CacheRuntime.HashCacheAssemblyStore assemblyStore, Burst.Compiler.IL.AssemblyLoader assemblyLoader) [0x001de] in <660bf86525254f05bf0505fe2843d6a3>:0
                          at Burst.Compiler.IL.Server.EntryPointMethodFinder.LoadAllAssemblyReferences (Mono.Cecil.AssemblyDefinition asmDef, System.Collections.Generic.Dictionary`2[TKey,TValue] visited, System.Collections.Generic.Dictionary`2[TKey,TValue] jobProducers, Burst.Compiler.IL.Hashing.CacheRuntime.HashCacheAssemblyStore assemblyStore, Burst.Compiler.IL.AssemblyLoader assemblyLoader) [0x001de] in <660bf86525254f05bf0505fe2843d6a3>:0
                          at Burst.Compiler.IL.Server.EntryPointMethodFinder.FindEntryPoints (System.String[] rootAssemblyNames, Burst.Compiler.IL.Hashing.CacheRuntime.HashCacheAssemblyStore assemblyStore, Burst.Compiler.IL.AssemblyLoader assemblyLoader, Burst.Compiler.IL.NativeCompilerOptions options, Burst.Compiler.IL.Server.ProfileDelegate profileCallback, Burst.Compiler.IL.Helpers.DebugLogWriter debugWriter) [0x00062] in <660bf86525254f05bf0505fe2843d6a3>:0
                          at Burst.Compiler.IL.Server.FindMethodsJob.Execute (Burst.Compiler.IL.Server.CompilerServerJobExecutionContext context) [0x0000c] in <660bf86525254f05bf0505fe2843d6a3>:0
                          at Burst.Compiler.IL.Server.CompilerThread.CompilerThreadLoop () [0x00103] in <660bf86525254f05bf0505fe2843d6a3>:0
                        ---> (Inner Exception #0) System.NotImplementedException: Unimplemented Instruction RefAnyVal
                          at Burst.Compiler.IL.Hashing.CacheBuilder.ILHasher+InstructionHasher.DecodeInstruction (System.Reflection.Metadata.BlobReader& reader) [0x00509] in <660bf86525254f05bf0505fe2843d6a3>:0
                          at Burst.Compiler.IL.Hashing.CacheBuilder.ILHasher.HashMethodDefinition (System.Reflection.Metadata.MethodDefinitionHandle methodDefinitionHandle, Burst.Compiler.IL.Hashing.CacheBuilder.Util.CachingMetadataReader metadataReader, System.Reflection.PortableExecutable.PEReader peReader, Burst.Compiler.IL.DebugInfo.PortablePdbInstance pdbInstance, Burst.Compiler.IL.Hashing.ILSignatureProvider signatureProvider, Burst.Compiler.IL.Hashing.Storage.BlobCollection blobCollection) [0x00086] in <660bf86525254f05bf0505fe2843d6a3>:0
                          at Burst.Compiler.IL.Hashing.CacheBuilder.ILHasher+<>c__DisplayClass12_0.<HashImpl>b__4 (System.Reflection.Metadata.MethodDefinitionHandle methodDefinitionHandle, System.Threading.Tasks.ParallelLoopState loopState, System.Int64 i) [0x00000] in <660bf86525254f05bf0505fe2843d6a3>:0
                          at System.Threading.Tasks.Parallel+<>c__DisplayClass44_0`2[TSource,TLocal].<PartitionerForEachWorker>b__1 (System.Collections.IEnumerator& partitionState, System.Int32 timeout, System.Boolean& replicationDelegateYieldedBeforeCompletion) [0x00132] in <f4694e4600d74e03a4e35def8b97021c>:0
                        --- End of stack trace from previous location where exception was thrown ---
                       
                          at System.Threading.Tasks.Parallel+<>c__DisplayClass44_0`2[TSource,TLocal].<PartitionerForEachWorker>b__1 (System.Collections.IEnumerator& partitionState, System.Int32 timeout, System.Boolean& replicationDelegateYieldedBeforeCompletion) [0x0025f] in <f4694e4600d74e03a4e35def8b97021c>:0
                          at System.Threading.Tasks.TaskReplicator+Replica`1[TState].ExecuteAction (System.Boolean& yieldedBeforeCompletion) [0x00000] in <f4694e4600d74e03a4e35def8b97021c>:0
                          at System.Threading.Tasks.TaskReplicator+Replica.Execute () [0x00023] in <f4694e4600d74e03a4e35def8b97021c>:0 <---

sheredom · May 12, 2022, 11:28am

Can you do my a favour and delete /Library/BurstCache, and then see if the error still happens?

OndrejP · May 12, 2022, 11:33am

OK, I found it, when I modify this method like this, it works.

public static TOut Reinterpret<TIn, TOut>(TIn value)
{
    //return __refvalue(__makeref(value), TOut);
    throw new NotImplementedException();
}

I’ll delete cache and see if that works as different workaround.

OndrejP · May 12, 2022, 11:42am

Deleting cache does not help.

Since I’ve found what causes the issue, I’m okay with rewriting the method, there’s no need to use TypedReference, it can be rewritten with UnsafeUtility

I’m not sure whether Unity officially supports TypedReferences or not, but Burst should IMO at least ignore methods which use them and not crash. Especially when it’s not needed at all for the code Burst is compiling / executing.

Thank you for help @sheredom , I wouldn’t be able to figure it out without you.

R2-RT · May 12, 2022, 6:28pm

@OndrejP According to this table, TypedReference is not supported only by IL2CPP, so I guess it should work in mono Unity - Manual: Scripting restrictions

Btw, its nice to learn something new, although it is hard to find any resources/documentation on TypedReferences. What is your use case of them?

OndrejP · May 12, 2022, 8:10pm

Thanks, I had no idea TypedReference is mentioned there.

I used them mainly for casting generic type to concrete type in generic method/class.
Imagine you have generic method DoSomething<T> and you know that T is bool.
There’s no way in C# to cast variable of type T to bool (without GC garbage or at least unsafe code).

So it was used in few rare cases when I wanted to branch in generic method for some special types.
Example: if (type == typeof(bool)) return Utils.Reinterpret<bool, T>(true);

As I understand TypedReference, if I made error in the code and would accidentally cast variable to different type, it would throw exception.

Here’s few other use cases:
https://stackoverflow.com/questions/4764573/why-is-typedreference-behind-the-scenes-its-so-fast-and-safe-almost-magical

sheredom · May 13, 2022, 8:18am

Hand on heart - didn’t even know these TypedReference things existed. I’ll try and get repro’s for this fail though - we definitely should at least error gracefully if this isn’t supportable by Burst!

sheredom · May 13, 2022, 11:04am

Ok I’ve found two bugs in the compiler that could cause us to crash with this - fixed both of them locally. Will go out in a new Burst version at some point.

Thanks for your help to make Burst better!