Hey,
I am seeing batchmode builds hang, but only over SSH, and only for some platforms (it seems IL2CPP related), I do not see any issues when using Mono for example.
If I RDP into the VM (Windows Server 2022) and run the same build script using cmd/powershell, it builds successfully. But when triggered via SSH, it hangs right near the end.
Env vars are the same, the user is the same, I have checked all the basic stuff.
Other weird stuff, this used to work, but one day it just started happening (windows updates are off, nothing noteworthy in the codebase…)
And, even weirder, I was also seeing various issues relating to bee_backend.exe failing but after forcing the Windows Defender off, those specific errors appear to have gone away (I’m not entirely convinced though, it could all be related just manifesting in different ways and the Windows Defender change was a fluke). Here is the error that I was seeing that has now vanished:
Starting: C:\Program Files\Unity\Hub\Editor\2021.3.39f1\Editor\Data\bee_backend.exe --profile=“Library/Bee/backend_profiler0.traceevents” --stdin-canary --dagfile=“Library/Bee/Player9729a761.dag” --continue-on-failure Player
WorkingDir: C:/Users/xxx/j/workspace/Steam_xxx
ExitCode: 4 Duration: 0s25ms
[ ] Require frontend run. Library/Bee/Player9729a761.dag couldn’t be loaded
Starting: C:\Program Files\Unity\Hub\Editor\2021.3.39f1\Editor\Data\Tools\netcorerun\netcorerun.exe “C:\Program Files\Unity\Hub\Editor\2021.3.39f1\Editor\Data\PlaybackEngines\WindowsStandaloneSupport\WinPlayerBuildProgram.exe” “C:/Program Files/Unity/Hub/Editor/2021.3.39f1/Editor/Data/PlaybackEngines/WindowsStandaloneSupport/Bee;C:/Program Files/Unity/Hub/Editor/2021.3.39f1/Editor/Data/Tools/BuildPipeline” “Library/Bee/Player9729a761.dag.json” “Library/Bee/Player9729a761-inputdata.json” “Library/Bee/buildprogram0.traceevents”
WorkingDir: C:/Users/xxx/j/workspace/Steam_xxx
ExitCode: -2146232000 Duration: 0s411ms
Internal build system error. BuildProgram exited with code -2146232000.
System.Xml.XmlException: Data at the root level is invalid. Line 1, position 1.
at System.Xml.XmlTextReaderImpl.Throw(Exception e)
at System.Xml.XmlTextReaderImpl.Throw(String res, String arg)
at System.Xml.XmlTextReaderImpl.ParseRootLevelWhitespace()
at System.Xml.XmlTextReaderImpl.ParseDocumentContent()
at System.Xml.XmlTextReaderImpl.Read()
at System.Xml.Linq.XDocument.Load(XmlReader reader, LoadOptions options)
In there it says Require frontend run I don’t know exactly what that means but it seems like it is talking about the Editor frontend? But this is batchmode? Ah I don’t know.
The hang is pretty consistent now. It hangs right near the end. The last line that is logged is (Filename: C:/build/output/unity/unity/PlatformDependent/WinPlayer/Extensions/Managed/WinPlayerPostProcessor.cs Line: 107)
Interestingly I don’t even see that log in a successful build. I also see these logs in the build that hangs which are not present in a successful build:
*** Additional run caused by: contents change of C:/Users/xxx/j/workspace/Steam_xxx/Library/Bee/artifacts/WinPlayerBuildProgram/il2cppOutput/cpp
STDERR:
[W] FindFirstFile() failed: C:\Users\xxx\j\workspace\Steam_xxx\Library\Bee\artifacts\WinPlayerBuildProgram\il2cppOutput\cpp\Symbols
Starting: C:\Program Files\Unity\Hub\Editor\2021.3.39f1\Editor\Data\Tools\netcorerun\netcorerun.exe “C:\Program Files\Unity\Hub\Editor\2021.3.39f1\Editor\Data\PlaybackEngines\WindowsStandaloneSupport\WinPlayerBuildProgram.exe” “C:/Program Files/Unity/Hub/Editor/2021.3.39f1/Editor/Data/PlaybackEngines/WindowsStandaloneSupport/Bee;C:/Program Files/Unity/Hub/Editor/2021.3.39f1/Editor/Data/Tools/BuildPipeline” “Library/Bee/Player9729a761.dag.json” “Library/Bee/Player9729a761-inputdata.json” “Library/Bee/buildprogram1.traceevents”
That very last line is interesting, it is running netcorerun.exe and passing in WinPlayerBuildProgram.exe but that doesn’t happen on a successful build… WHY? and why only when over SSH?
If anyone can help me unravel this mystery, I will be eternally greatful.
Thanks!