Help diagnosing a Linux editor performance regression in 2022 LTS

Unity Engine
, , ,
1

I’m experiencing a problem with a sizable game project in the Linux editor after updating to 2022 LTS. Both 2022.3.39 and .40 are having the same issue for me. The Linux editor was fine on 2019, 2020 and 2021 versions.

The visible symptoms are, after changing some code and refreshing the asset database in Unity, the editor UI freezes (no visible progress bar or anything useful) for a long time, about two minutes. After the freeze ends, the progress bar shows up, indicating compilation and domain reload, then I can enter play mode. Iteration speed generally hasn’t been amazing with Unity, but of course a two-minute freeze after any change means it’s practically impossible to get real work done.

I’ve been trying to diagnose or work around this with no success so far. Here’s what I’ve found:

The problem doesn’t seem to be with compilation times or domain reload. Once the progress bar for those operations pops up, it completes in a time comparable to 2021.

It’s also not the asset database refresh. I’ve tried with and without auto-refresh, having auto-refresh does trigger the freeze immediately after changing, otherwise it triggers after a manual refresh, but according to the logs, the refresh itself is fast.

Editor.log doesn’t shed much light on what’s going on. Here’s what an occurrence of the problem looks like in the log

[ScriptCompilation] Requested script compilation because: Assetdatabase observed changes in script compilation related files
AssetDatabase: script compilation time: 0.000793s
Start importing Assets/GameAssets/Scripts/Rendering/Borders/BorderComponent.cs using Guid(612a90334a8df1e4885e674b61e9cec4) Importer(-1,00000000000000000000000000000000) 
(Unity:3721): GLib-GIO-CRITICAL **: 16:28:03.143: g_dbus_proxy_call_sync_internal: assertion 'G_IS_DBUS_PROXY (proxy)' failed
[PhysX] Initialized MultithreadedTaskDispatcher with 24 workers.

... a few artifact import messages...
[Licensing::Client] Successfully resolved entitlement details
Refreshing native plugins compatible for Editor in 9.39 ms, found 6 plugins.
Preloading 1 native plugins for Editor in 0.09 ms.
Asset Pipeline Refresh (id=3fc24b1fa01e179039a4cae890f66a36): Total: 0.831 seconds - Initiated by RefreshV2(NoUpdateAssetOptions)
	Summary:
		Imports: total=22 (actual=22, local cache=0, cache server=0)
		Asset DB Process Time: managed=2 ms, native=434 ms
		Asset DB Callback time: managed=66 ms, native=326 ms
		Scripting: domain reloads=0, domain reload time=0 ms, compile time=1 ms, other=0 ms
		Project Asset Count: scripts=8725, non-scripts=13120
		Asset File Changes: new=0, changed=22, moved=0, deleted=0
		Scan Filter Count: 1
	InvokeBeforeRefreshCallbacks: 0.000ms
	ApplyChangesToAssetFolders: 0.115ms
	Scan: 195.518ms
	OnSourceAssetsModified: 1.717ms
	GetAllGuidsForCategorization: 2.021ms
	CategorizeAssets: 24.611ms
	ImportOutOfDateAssets: 189.930ms (-2.900ms without children)
		ImportManagerImport: 182.335ms (18.557ms without children)
			ImportInProcess: 163.607ms
			UpdateCategorizedAssets: 0.171ms
		CompileScripts: 0.803ms
		ReloadNativeAssets: 0.345ms
		UnloadImportedAssets: 0.755ms
		ReloadImportedAssets: 3.233ms
		EnsureUptoDateAssetsAreRegisteredWithGuidPM: 1.952ms
		InitializingProgressBar: 0.003ms
		PostProcessAllAssetNotificationsAddChangedAssets: 0.001ms
		OnDemandSchedulerStart: 3.402ms
	PostProcessAllAssets: 366.816ms
	Hotreload: 24.607ms
	GatherAllCurrentPrimaryArtifactRevisions: 0.748ms
	UnloadStreamsBegin: 0.501ms
	PersistCurrentRevisions: 1.026ms
	UnloadStreamsEnd: 0.210ms
	Untracked: 22.985ms

It seems to correctly identify the changed files, and the timing in the log indicates no problems. ImportOutOfDateAssets + PostProcessAllAssets is the bulk of the time, and that’s less than a second. After the above log output, the freeze occurs. Then there’s no output until the editor unfreezes, at which point I see ILPP and Tundra output as follows (just censoring some file names):

e[40me[32minfoe[39me[22me[49m: Microsoft.AspNetCore.Hosting.Diagnostics[1]
      Request starting HTTP/2 POST http://ilpp/UnityILPP.PostProcessing/Ping application/grpc -
e[40me[32minfoe[39me[22me[49m: Microsoft.AspNetCore.Routing.EndpointMiddleware[0]
      Executing endpoint 'gRPC - /UnityILPP.PostProcessing/Ping'
e[40me[32minfoe[39me[22me[49m: Microsoft.AspNetCore.Routing.EndpointMiddleware[1]
      Executed endpoint 'gRPC - /UnityILPP.PostProcessing/Ping'
e[40me[32minfoe[39me[22me[49m: Microsoft.AspNetCore.Hosting.Diagnostics[2]
      Request finished HTTP/2 POST http://ilpp/UnityILPP.PostProcessing/Ping application/grpc - - 200 - application/grpc 0.1118ms
Starting: SOMEPATH/2022.3.40f1/Editor/Data/bee_backend --dont-print-to-structured-log --ipc --defer-dag-verification --dagfile="Library/Bee/2400b0aEDbg.dag" --continue-on-failure --profile="Library/Bee/backend1.traceevents" ScriptAssemblies
WorkingDir: MYPROJECTPATH
[Licensing::Client] Successfully resolved entitlement details
e[40me[32minfoe[39me[22me[49m: Microsoft.AspNetCore.Hosting.Diagnostics[1]
      Request starting HTTP/2 POST http://ilpp/UnityILPP.PostProcessing/ConfigurePostProcessors application/grpc -
e[40me[32minfoe[39me[22me[49m: Microsoft.AspNetCore.Routing.EndpointMiddleware[0]
      Executing endpoint 'gRPC - /UnityILPP.PostProcessing/ConfigurePostProcessors'
e[40me[32minfoe[39me[22me[49m: Unity.ILPP.Runner.PostProcessingAssemblyLoadContext[0]
      Current configuration matches the requested one, skipping
e[40me[32minfoe[39me[22me[49m: Microsoft.AspNetCore.Routing.EndpointMiddleware[1]
      Executed endpoint 'gRPC - /UnityILPP.PostProcessing/ConfigurePostProcessors'
e[40me[32minfoe[39me[22me[49m: Microsoft.AspNetCore.Hosting.Diagnostics[2]
      Request finished HTTP/2 POST http://ilpp/UnityILPP.PostProcessing/ConfigurePostProcessors application/grpc - - 200 - application/grpc 0.2139ms
e[40me[32minfoe[39me[22me[49m: Microsoft.AspNetCore.Hosting.Diagnostics[1]
      Request starting HTTP/2 POST http://ilpp/UnityILPP.PostProcessing/PostProcessAssembly application/grpc -
e[40me[32minfoe[39me[22me[49m: Microsoft.AspNetCore.Routing.EndpointMiddleware[0]
      Executing endpoint 'gRPC - /UnityILPP.PostProcessing/PostProcessAssembly'
e[40me[32minfoe[39me[22me[49m: Unity.ILPP.Runner.PostProcessingPipeline[0]
      Processing assembly Library/Bee/artifacts/2400b0aEDbg.dag/ThirdParty.dll, with 121 defines and 237 references
e[40me[32minfoe[39me[22me[49m: Unity.ILPP.Runner.PostProcessingPipeline[0]
      processors: zzzUnity.Burst.CodeGen.BurstILPostProcessor
e[40me[32minfoe[39me[22me[49m: Unity.ILPP.Runner.PostProcessingPipeline[0]
      running zzzUnity.Burst.CodeGen.BurstILPostProcessor
e[40me[32minfoe[39me[22me[49m: Microsoft.AspNetCore.Routing.EndpointMiddleware[1]
      Executed endpoint 'gRPC - /UnityILPP.PostProcessing/PostProcessAssembly'
e[40me[32minfoe[39me[22me[49m: Microsoft.AspNetCore.Hosting.Diagnostics[2]
      Request finished HTTP/2 POST http://ilpp/UnityILPP.PostProcessing/PostProcessAssembly application/grpc - - 200 - application/grpc 2.6818ms
e[40me[32minfoe[39me[22me[49m: Microsoft.AspNetCore.Hosting.Diagnostics[1]
      Request starting HTTP/2 POST http://ilpp/UnityILPP.PostProcessing/PostProcessAssembly application/grpc -
e[40me[32minfoe[39me[22me[49m: Microsoft.AspNetCore.Routing.EndpointMiddleware[0]
      Executing endpoint 'gRPC - /UnityILPP.PostProcessing/PostProcessAssembly'
e[40me[32minfoe[39me[22me[49m: Unity.ILPP.Runner.PostProcessingPipeline[0]
      Processing assembly Library/Bee/artifacts/2400b0aEDbg.dag/SESSAOEditor.dll, with 122 defines and 241 references
e[40me[32minfoe[39me[22me[49m: Unity.ILPP.Runner.PostProcessingPipeline[0]
      processors: zzzUnity.Burst.CodeGen.BurstILPostProcessor
e[40me[32minfoe[39me[22me[49m: Unity.ILPP.Runner.PostProcessingPipeline[0]
      running zzzUnity.Burst.CodeGen.BurstILPostProcessor
e[40me[32minfoe[39me[22me[49m: Microsoft.AspNetCore.Routing.EndpointMiddleware[1]
      Executed endpoint 'gRPC - /UnityILPP.PostProcessing/PostProcessAssembly'
e[40me[32minfoe[39me[22me[49m: Microsoft.AspNetCore.Hosting.Diagnostics[2]
      Request finished HTTP/2 POST http://ilpp/UnityILPP.PostProcessing/PostProcessAssembly application/grpc - - 200 - application/grpc 0.9146ms
e[40me[32minfoe[39me[22me[49m: Microsoft.AspNetCore.Hosting.Diagnostics[1]
      Request starting HTTP/2 POST http://ilpp/UnityILPP.PostProcessing/PostProcessAssembly application/grpc -
e[40me[32minfoe[39me[22me[49m: Microsoft.AspNetCore.Routing.EndpointMiddleware[0]
      Executing endpoint 'gRPC - /UnityILPP.PostProcessing/PostProcessAssembly'
e[40me[32minfoe[39me[22me[49m: Unity.ILPP.Runner.PostProcessingPipeline[0]
      Processing assembly Library/Bee/artifacts/2400b0aEDbg.dag/AmplifyColorEditorUnity.dll, with 122 defines and 241 references
e[40me[32minfoe[39me[22me[49m: Unity.ILPP.Runner.PostProcessingPipeline[0]
      processors: zzzUnity.Burst.CodeGen.BurstILPostProcessor
e[40me[32minfoe[39me[22me[49m: Unity.ILPP.Runner.PostProcessingPipeline[0]
      running zzzUnity.Burst.CodeGen.BurstILPostProcessor
e[40me[32minfoe[39me[22me[49m: Microsoft.AspNetCore.Routing.EndpointMiddleware[1]
      Executed endpoint 'gRPC - /UnityILPP.PostProcessing/PostProcessAssembly'
e[40me[32minfoe[39me[22me[49m: Microsoft.AspNetCore.Hosting.Diagnostics[2]
      Request finished HTTP/2 POST http://ilpp/UnityILPP.PostProcessing/PostProcessAssembly application/grpc - - 200 - application/grpc 0.9204ms
e[40me[32minfoe[39me[22me[49m: Microsoft.AspNetCore.Hosting.Diagnostics[1]
      Request starting HTTP/2 POST http://ilpp/UnityILPP.PostProcessing/PostProcessAssembly application/grpc -
e[40me[32minfoe[39me[22me[49m: Microsoft.AspNetCore.Routing.EndpointMiddleware[0]
      Executing endpoint 'gRPC - /UnityILPP.PostProcessing/PostProcessAssembly'
e[40me[32minfoe[39me[22me[49m: Unity.ILPP.Runner.PostProcessingPipeline[0]
      Processing assembly Library/Bee/artifacts/2400b0aEDbg.dag/AmplifyImpostorsEditor.dll, with 122 defines and 242 references
e[40me[32minfoe[39me[22me[49m: Unity.ILPP.Runner.PostProcessingPipeline[0]
      processors: zzzUnity.Burst.CodeGen.BurstILPostProcessor
e[40me[32minfoe[39me[22me[49m: Unity.ILPP.Runner.PostProcessingPipeline[0]
      running zzzUnity.Burst.CodeGen.BurstILPostProcessor
e[40me[32minfoe[39me[22me[49m: Microsoft.AspNetCore.Routing.EndpointMiddleware[1]
      Executed endpoint 'gRPC - /UnityILPP.PostProcessing/PostProcessAssembly'
e[40me[32minfoe[39me[22me[49m: Microsoft.AspNetCore.Hosting.Diagnostics[2]
      Request finished HTTP/2 POST http://ilpp/UnityILPP.PostProcessing/PostProcessAssembly application/grpc - - 200 - application/grpc 0.9275ms
e[40me[32minfoe[39me[22me[49m: Microsoft.AspNetCore.Hosting.Diagnostics[1]
      Request starting HTTP/2 POST http://ilpp/UnityILPP.PostProcessing/PostProcessAssembly application/grpc -
e[40me[32minfoe[39me[22me[49m: Microsoft.AspNetCore.Routing.EndpointMiddleware[0]
      Executing endpoint 'gRPC - /UnityILPP.PostProcessing/PostProcessAssembly'
e[40me[32minfoe[39me[22me[49m: Unity.ILPP.Runner.PostProcessingPipeline[0]
      Processing assembly Library/Bee/artifacts/2400b0aEDbg.dag/GameName.GameCore.dll, with 121 defines and 237 references
e[40me[32minfoe[39me[22me[49m: Unity.ILPP.Runner.PostProcessingPipeline[0]
      processors: zzzUnity.Burst.CodeGen.BurstILPostProcessor
e[40me[32minfoe[39me[22me[49m: Unity.ILPP.Runner.PostProcessingPipeline[0]
      running zzzUnity.Burst.CodeGen.BurstILPostProcessor
e[40me[32minfoe[39me[22me[49m: Microsoft.AspNetCore.Routing.EndpointMiddleware[1]
      Executed endpoint 'gRPC - /UnityILPP.PostProcessing/PostProcessAssembly'
e[40me[32minfoe[39me[22me[49m: Microsoft.AspNetCore.Hosting.Diagnostics[2]
      Request finished HTTP/2 POST http://ilpp/UnityILPP.PostProcessing/PostProcessAssembly application/grpc - - 200 - application/grpc 11.1562ms
e[40me[32minfoe[39me[22me[49m: Microsoft.AspNetCore.Hosting.Diagnostics[1]
      Request starting HTTP/2 POST http://ilpp/UnityILPP.PostProcessing/PostProcessAssembly application/grpc -
e[40me[32minfoe[39me[22me[49m: Microsoft.AspNetCore.Routing.EndpointMiddleware[0]
      Executing endpoint 'gRPC - /UnityILPP.PostProcessing/PostProcessAssembly'
e[40me[32minfoe[39me[22me[49m: Unity.ILPP.Runner.PostProcessingPipeline[0]
      Processing assembly Library/Bee/artifacts/2400b0aEDbg.dag/GameName.CampaignAssembly.dll, with 121 defines and 238 references
e[40me[32minfoe[39me[22me[49m: Unity.ILPP.Runner.PostProcessingPipeline[0]
      processors: zzzUnity.Burst.CodeGen.BurstILPostProcessor
e[40me[32minfoe[39me[22me[49m: Unity.ILPP.Runner.PostProcessingPipeline[0]
      running zzzUnity.Burst.CodeGen.BurstILPostProcessor
e[40me[32minfoe[39me[22me[49m: Microsoft.AspNetCore.Routing.EndpointMiddleware[1]
      Executed endpoint 'gRPC - /UnityILPP.PostProcessing/PostProcessAssembly'
e[40me[32minfoe[39me[22me[49m: Microsoft.AspNetCore.Hosting.Diagnostics[2]
      Request finished HTTP/2 POST http://ilpp/UnityILPP.PostProcessing/PostProcessAssembly application/grpc - - 200 - application/grpc 0.6919ms
e[40me[32minfoe[39me[22me[49m: Microsoft.AspNetCore.Hosting.Diagnostics[1]
      Request starting HTTP/2 POST http://ilpp/UnityILPP.PostProcessing/PostProcessAssembly application/grpc -
e[40me[32minfoe[39me[22me[49m: Microsoft.AspNetCore.Routing.EndpointMiddleware[0]
      Executing endpoint 'gRPC - /UnityILPP.PostProcessing/PostProcessAssembly'
e[40me[32minfoe[39me[22me[49m: Unity.ILPP.Runner.PostProcessingPipeline[0]
      Processing assembly Library/Bee/artifacts/2400b0aEDbg.dag/Assembly-CSharp-firstpass.dll, with 121 defines and 295 references
e[40me[32minfoe[39me[22me[49m: Unity.ILPP.Runner.PostProcessingPipeline[0]
      processors: zzzUnity.Burst.CodeGen.BurstILPostProcessor
e[40me[32minfoe[39me[22me[49m: Unity.ILPP.Runner.PostProcessingPipeline[0]
      running zzzUnity.Burst.CodeGen.BurstILPostProcessor
e[40me[32minfoe[39me[22me[49m: Microsoft.AspNetCore.Routing.EndpointMiddleware[1]
      Executed endpoint 'gRPC - /UnityILPP.PostProcessing/PostProcessAssembly'
e[40me[32minfoe[39me[22me[49m: Microsoft.AspNetCore.Hosting.Diagnostics[2]
      Request finished HTTP/2 POST http://ilpp/UnityILPP.PostProcessing/PostProcessAssembly application/grpc - - 200 - application/grpc 8.5502ms
e[40me[32minfoe[39me[22me[49m: Microsoft.AspNetCore.Hosting.Diagnostics[1]
      Request starting HTTP/2 POST http://ilpp/UnityILPP.PostProcessing/PostProcessAssembly application/grpc -
e[40me[32minfoe[39me[22me[49m: Microsoft.AspNetCore.Routing.EndpointMiddleware[0]
      Executing endpoint 'gRPC - /UnityILPP.PostProcessing/PostProcessAssembly'
e[40me[32minfoe[39me[22me[49m: Unity.ILPP.Runner.PostProcessingPipeline[0]
      Processing assembly Library/Bee/artifacts/2400b0aEDbg.dag/Assembly-CSharp.dll, with 121 defines and 296 references
e[40me[32minfoe[39me[22me[49m: Unity.ILPP.Runner.PostProcessingPipeline[0]
      processors: zzzUnity.Burst.CodeGen.BurstILPostProcessor
e[40me[32minfoe[39me[22me[49m: Unity.ILPP.Runner.PostProcessingPipeline[0]
      running zzzUnity.Burst.CodeGen.BurstILPostProcessor
e[40me[32minfoe[39me[22me[49m: Microsoft.AspNetCore.Routing.EndpointMiddleware[1]
      Executed endpoint 'gRPC - /UnityILPP.PostProcessing/PostProcessAssembly'
e[40me[32minfoe[39me[22me[49m: Microsoft.AspNetCore.Hosting.Diagnostics[2]
      Request finished HTTP/2 POST http://ilpp/UnityILPP.PostProcessing/PostProcessAssembly application/grpc - - 200 - application/grpc 309.6288ms
e[40me[32minfoe[39me[22me[49m: Microsoft.AspNetCore.Hosting.Diagnostics[1]
      Request starting HTTP/2 POST http://ilpp/UnityILPP.PostProcessing/PostProcessAssembly application/grpc -
e[40me[32minfoe[39me[22me[49m: Microsoft.AspNetCore.Routing.EndpointMiddleware[0]
      Executing endpoint 'gRPC - /UnityILPP.PostProcessing/PostProcessAssembly'
e[40me[32minfoe[39me[22me[49m: Unity.ILPP.Runner.PostProcessingPipeline[0]
      Processing assembly Library/Bee/artifacts/2400b0aEDbg.dag/Assembly-CSharp-Editor.dll, with 122 defines and 308 references
e[40me[32minfoe[39me[22me[49m: Unity.ILPP.Runner.PostProcessingPipeline[0]
      processors: zzzUnity.Burst.CodeGen.BurstILPostProcessor
e[40me[32minfoe[39me[22me[49m: Unity.ILPP.Runner.PostProcessingPipeline[0]
      running zzzUnity.Burst.CodeGen.BurstILPostProcessor
e[40me[32minfoe[39me[22me[49m: Microsoft.AspNetCore.Routing.EndpointMiddleware[1]
      Executed endpoint 'gRPC - /UnityILPP.PostProcessing/PostProcessAssembly'
e[40me[32minfoe[39me[22me[49m: Microsoft.AspNetCore.Hosting.Diagnostics[2]
      Request finished HTTP/2 POST http://ilpp/UnityILPP.PostProcessing/PostProcessAssembly application/grpc - - 200 - application/grpc 51.9650ms
ExitCode: 0 Duration: 8s190ms
[720/997    1s] ILPP-Configuration Library/ilpp-configuration.nevergeneratedoutput
[961/997    1s] Csc Library/Bee/artifacts/2400b0aEDbg.dag/ThirdParty.dll (+2 others)
[962/997    0s] ILPostProcess Library/Bee/artifacts/2400b0aEDbg.dag/post-processed/ThirdParty.dll (+pdb)
Processing assembly Library/Bee/artifacts/2400b0aEDbg.dag/ThirdParty.dll, with 121 defines and 237 references
processors: zzzUnity.Burst.CodeGen.BurstILPostProcessor
running zzzUnity.Burst.CodeGen.BurstILPostProcessor
[963/997    0s] CopyFiles Library/ScriptAssemblies/ThirdParty.pdb
[964/997    0s] CopyFiles Library/ScriptAssemblies/ThirdParty.dll
[965/997    0s] Csc Library/Bee/artifacts/2400b0aEDbg.dag/SESSAOEditor.dll (+2 others)
[966/997    0s] ILPostProcess Library/Bee/artifacts/2400b0aEDbg.dag/post-processed/SESSAOEditor.dll (+pdb)
Processing assembly Library/Bee/artifacts/2400b0aEDbg.dag/SESSAOEditor.dll, with 122 defines and 241 references
processors: zzzUnity.Burst.CodeGen.BurstILPostProcessor
running zzzUnity.Burst.CodeGen.BurstILPostProcessor
[967/997    0s] CopyFiles Library/ScriptAssemblies/SESSAOEditor.dll
[968/997    0s] CopyFiles Library/ScriptAssemblies/SESSAOEditor.pdb
[969/997    0s] Csc Library/Bee/artifacts/2400b0aEDbg.dag/AmplifyColorEditorUnity.dll (+2 others)
[970/997    0s] ILPostProcess Library/Bee/artifacts/2400b0aEDbg.dag/post-processed/AmplifyColorEditorUnity.dll (+pdb)
Processing assembly Library/Bee/artifacts/2400b0aEDbg.dag/AmplifyColorEditorUnity.dll, with 122 defines and 241 references
processors: zzzUnity.Burst.CodeGen.BurstILPostProcessor
running zzzUnity.Burst.CodeGen.BurstILPostProcessor
[971/997    0s] CopyFiles Library/ScriptAssemblies/AmplifyColorEditorUnity.pdb
[972/997    0s] CopyFiles Library/ScriptAssemblies/AmplifyColorEditorUnity.dll
[973/997    0s] Csc Library/Bee/artifacts/2400b0aEDbg.dag/AmplifyImpostorsEditor.dll (+2 others)
[974/997    0s] ILPostProcess Library/Bee/artifacts/2400b0aEDbg.dag/post-processed/AmplifyImpostorsEditor.dll (+pdb)
Processing assembly Library/Bee/artifacts/2400b0aEDbg.dag/AmplifyImpostorsEditor.dll, with 122 defines and 242 references
processors: zzzUnity.Burst.CodeGen.BurstILPostProcessor
running zzzUnity.Burst.CodeGen.BurstILPostProcessor
[975/997    0s] CopyFiles Library/ScriptAssemblies/AmplifyImpostorsEditor.pdb
[976/997    0s] CopyFiles Library/ScriptAssemblies/AmplifyImpostorsEditor.dll
[977/997    5s] Csc Library/Bee/artifacts/2400b0aEDbg.dag/GameName.GameCore.dll (+2 others)
[978/997    0s] ILPostProcess Library/Bee/artifacts/2400b0aEDbg.dag/post-processed/GameName.GameCore.dll (+pdb)
Processing assembly Library/Bee/artifacts/2400b0aEDbg.dag/GameName.GameCore.dll, with 121 defines and 237 references
processors: zzzUnity.Burst.CodeGen.BurstILPostProcessor
running zzzUnity.Burst.CodeGen.BurstILPostProcessor
[979/997    0s] CopyFiles Library/ScriptAssemblies/GameName.GameCore.pdb
[980/997    0s] CopyFiles Library/ScriptAssemblies/GameName.GameCore.dll
[981/997    0s] Csc Library/Bee/artifacts/2400b0aEDbg.dag/GameName.CampaignAssembly.dll (+2 others)
[982/997    0s] ILPostProcess Library/Bee/artifacts/2400b0aEDbg.dag/post-processed/GameName.CampaignAssembly.dll (+pdb)
Processing assembly Library/Bee/artifacts/2400b0aEDbg.dag/GameName.CampaignAssembly.dll, with 121 defines and 238 references
processors: zzzUnity.Burst.CodeGen.BurstILPostProcessor
running zzzUnity.Burst.CodeGen.BurstILPostProcessor
[983/997    0s] CopyFiles Library/ScriptAssemblies/GameName.CampaignAssembly.pdb
[984/997    0s] CopyFiles Library/ScriptAssemblies/GameName.CampaignAssembly.dll
[985/997    0s] Csc Library/Bee/artifacts/2400b0aEDbg.dag/Assembly-CSharp-firstpass.dll (+2 others)
[986/997    0s] ILPostProcess Library/Bee/artifacts/2400b0aEDbg.dag/post-processed/Assembly-CSharp-firstpass.dll (+pdb)
Processing assembly Library/Bee/artifacts/2400b0aEDbg.dag/Assembly-CSharp-firstpass.dll, with 121 defines and 295 references
processors: zzzUnity.Burst.CodeGen.BurstILPostProcessor
running zzzUnity.Burst.CodeGen.BurstILPostProcessor
[987/997    0s] CopyFiles Library/ScriptAssemblies/Assembly-CSharp-firstpass.pdb
[988/997    0s] CopyFiles Library/ScriptAssemblies/Assembly-CSharp-firstpass.dll
[989/997    2s] Csc Library/Bee/artifacts/2400b0aEDbg.dag/Assembly-CSharp.dll (+2 others)
[990/997    0s] ILPostProcess Library/Bee/artifacts/2400b0aEDbg.dag/post-processed/Assembly-CSharp.dll (+pdb)
Processing assembly Library/Bee/artifacts/2400b0aEDbg.dag/Assembly-CSharp.dll, with 121 defines and 296 references
processors: zzzUnity.Burst.CodeGen.BurstILPostProcessor
running zzzUnity.Burst.CodeGen.BurstILPostProcessor
[991/997    0s] CopyFiles Library/ScriptAssemblies/Assembly-CSharp.pdb
[992/997    0s] Csc Library/Bee/artifacts/2400b0aEDbg.dag/Assembly-CSharp-Editor.dll (+2 others)
[993/997    0s] CopyFiles Library/ScriptAssemblies/Assembly-CSharp.dll
[994/997    0s] ILPostProcess Library/Bee/artifacts/2400b0aEDbg.dag/post-processed/Assembly-CSharp-Editor.dll (+pdb)
Processing assembly Library/Bee/artifacts/2400b0aEDbg.dag/Assembly-CSharp-Editor.dll, with 122 defines and 308 references
processors: zzzUnity.Burst.CodeGen.BurstILPostProcessor
running zzzUnity.Burst.CodeGen.BurstILPostProcessor
[995/997    0s] CopyFiles Library/ScriptAssemblies/Assembly-CSharp-Editor.pdb
[996/997    0s] CopyFiles Library/ScriptAssemblies/Assembly-CSharp-Editor.dll
*** Tundra build success (7.90 seconds), 37 items updated, 997 evaluated
Total cache size 79704068
Total cache size after purge 79704068
Reloading assemblies after finishing script compilation.

The 8 seconds Tundra reports for the build seems accurate, too. I don’t see anything wrong in the log, so I think the problem is with something that doesn’t show up there.

I find this bit suspicious:

[720/997    1s] ILPP-Configuration Library/ilpp-configuration.nevergeneratedoutput
[961/997    1s] Csc Library/Bee/artifacts/2400b0aEDbg.dag/ThirdParty.dll (+2 others)
[962/997    0s] ILPostProcess Library/Bee/artifacts/2400b0aEDbg.dag/post-processed/ThirdParty.dll (+pdb)
Processing assembly Library/Bee/artifacts/2400b0aEDbg.dag/ThirdParty.dll, with 121 defines and 237 references
processors: zzzUnity.Burst.CodeGen.BurstILPostProcessor

If the 997 is some kind of task counter, I have no output from the vast majority of tasks. It seems like the freeze could be with whatever they’re doing, or not doing. It’s probably not a bee_backend issue as I don’t see it in the process list during the freeze.

Is there a way for me to enable more editor logging or otherwise diagnose what it’s doing during the freeze?

2

As usual, posting about a problem leads to some new ideas on what to do.

I attached strace to Unity processes once the editor froze and was greeted with a suspect process running deluge of close() calls returning EBADF.

close(476489147)                        = -1 EBADF (Bad file descriptor)
close(476489148)                        = -1 EBADF (Bad file descriptor)
close(476489149)                        = -1 EBADF (Bad file descriptor)
close(476489150)                        = -1 EBADF (Bad file descriptor)
close(476489151)                        = -1 EBADF (Bad file descriptor)
close(476489152)                        = -1 EBADF (Bad file descriptor)
close(476489153)                        = -1 EBADF (Bad file descriptor)
close(476489154)                        = -1 EBADF (Bad file descriptor)
close(476489155)                        = -1 EBADF (Bad file descriptor)
close(476489156)                        = -1 EBADF (Bad file descriptor)
close(476489157)                        = -1 EBADF (Bad file descriptor)
close(476489158)                        = -1 EBADF (Bad file descriptor)
close(476489159)                        = -1 EBADF (Bad file descriptor)
close(476489160)                        = -1 EBADF (Bad file descriptor)
close(476489161)                        = -1 EBADF (Bad file descriptor)

The editor appears to be stuck in a loop where it tries to close non-existent file descriptors, and it continues until it runs out of ints. The questionable process does have a pipe open, the other end of which is another Unity process, I believe the main one.

Forcefully killing the process produces this in Editor.log

Connectivity with IL Post Processor runner cannot be established, restarting ILPP runner: info: Microsoft.Hosting.Lifetime[0]

I also notice there’s a process in the same close() loop when initially opening the project in Unity, the progress bar is then at “Initial Asset Database Refresh”. Downgrading the Burst package on a hunch doesn’t change anything, neither does disabling stubbing out the editor’s Unity.ILPP.Runner binary so I’m not sure if it actually has anything to do with ILPP.

EDIT: This is actually independent of the project. I get the same behavior with a small game project, as well as a newly created test project.

3

Well, I found what was causing this. After refreshing the asset database, it seems that Unity attempts to close fds up to the limit of files per process. Sure enough, I’d configured this machine to have no hard limit so while the soft limit (ulimit -f) was 4096, the hard one (ulimit -f -H) was unlimited, so the 2022 Editor gladly boosted its soft limit to the hard one and would then spend a couple minutes in the close loop after any database refresh.

So as a workaround, setting the hard limit to 4096 did the trick. strace confirms that the problematic process still calls close() on every fd < 4096 and therefore still wastes calls but that’s pretty instant. But 2022.3 LTS appears unusable if there’s no hard open file limit, possibly due to a fragment of code that closes up to getrlimit() files instead of keeping track of used fds.

2 Likes