2019.4.18 - CLI build fail (Editor works) - Emscripten [Error 6] The handle is invalid

Unity Engine
1

Hello,

I hope I did not miss it in search

We have a problem with building our upcoming games (2 of them) using Command Line Interface. When using the very same build command from editor everything works fine

We tried it on 2 different PCs with 2019.4.16 and 2019.4.18 and they receive the same error. This one is from TeamCity build chain, but it looks the same (sans paths) on my local machine. Second game results in exactly identical issue and again builds just fine from editor itself. Android Builds also build just fine from CLI, it’s just WebGL that is causing headache. New/empty WebGL projects build fine. So it’s not an issue with machines or Unity installation itself

I am removing parts of the pathnames for clarity

[UNITY] → Unity Install Location \ WebGLSupport\BuildTools

— code follows (super long lines, was breaking code tags) —

Filename: “[UNITY]\Emscripten_Win\python\2.7.5.3_64bit\python.exe”
Arguments: -E “[UNITY]\Emscripten\emcc”
@“D:\AgentUnity1\work\37c533f1543ab8a2\Assets..\Temp\emcc_arguments.resp”
index: -1
Failed running “[UNITY]\Emscripten_Win\python\2.7.5.3_64bit\python.exe” -E
“C[UNITY]\Emscripten\emcc”
@“D:\AgentUnity1\work\37c533f1543ab8a2\Assets..\Temp\emcc_arguments.resp”

stdout:
stderr:Traceback (most recent call last): File “[UNITY]\Emscripten\emcc.py”, line 3063, in sys.exit(run()) File “[UNITY]\Emscripten\emcc.py”, line 1632, in run extra_files_to_link += system_libs.calculate([f for , f in sorted(temp_files)] + extra_files_to_link, in_temp, stdout=None, stderr_=None, forced=forced_stdlibs) File “[UNITY]\Emscripten\tools\system_libs.py”, line 520, in calculate symbolses = shared.Building.parallel_llvm_nm(list(map(os.path.abspath, temp_files)))
File “[UNITY]\Emscripten\tools\shared.py”, line 1844, in parallel_llvm_nm object_contents = pool.map(g_llvm_nm_uncached, files)
File “[UNITY]\Emscripten_Win\python\2.7.5.3_64bit\lib\multiprocessing\pool.py”, line 250, in map return self.map_async(func, iterable, chunksize).get()
File “[UNITY]\Emscripten_Win\python\2.7.5.3_64bit\lib\multiprocessing\pool.py”, line 554, in get raise self._valueWindowsError: [Error 6] The handle is invalid

Overall: [Error 6] The handle is invalid

The command used was rather simple, just project path, execute method and the standard “-quit -nographics -batchmode -buildTarget WebGL”

We spent quite some time on Google looking for some suggestions, but things we found (like removing jslibs from project) did not help

Full log is >8 MB and contains things that we cant show due to NDA, but here’s a redacted part starting with Emscripten linking process:
https://imdev.pl/share/6c84886a-8ae3-491d-b8ad-ea23c8f26710/buildlog_webgl_dev.txt

We will be very helpful for any help here

2

One thing to try to diagnose is to set global environment variables EMCC_CORES=1 and/or EMCC_DEBUG=1 , then rerun. First one will disable Emscripten multithreading, second one will add more debug printing to Emscripten.

3

@jukka_j
I added those but same error unfortunately :frowning: And no extra debug info. Unless Emscripten keeps it logs somewhere else that is

2019.4.18f1 - cutting full folder names

Filename: “BuildTools\Emscripten_Win\python\2.7.5.3_64bit\python.exe”
Arguments: -E “BuildTools\Emscripten\emcc” @“D:\Work\PROJECT\Assets..\Temp\emcc_arguments.resp”
index: -1
Failed running “\BuildTools\Emscripten_Win\python\2.7.5.3_64bit\python.exe” -E “\BuildTools\Emscripten\emcc” @“D:\Work\PROJECT\Assets..\Temp\emcc_arguments.resp”

Emscripten_Win\python\2.7.5.3_64bit\lib\multiprocessing\pool.py", line 554, in get raise self._valueWindowsError: [Error 6] The handle is invalid

Im not sure why its calling multiprocessing part of library (or maybe it doesnt matter?) but the line 554 relates to error raising from:

Class whose instances are returned by Pool.apply_async()

class ApplyResult(object):

PS: it works just fine in 2020.2 but we have significant in-game issues and bugs there, so cant upgrade easily

4

Sorry to hear. If you are able to reduce the failure down to a small test case, which fails in 2019.4.18f1 but works in 2020.2, you can try reporting that as a bug. It sounds like it is an issue that has been fixed, but the fix has not made its way back to 2019.4.18f1. Maybe a bug report can help QA bisect down where the particular fix happened.

5

Thanks @jukka_j

Unfortunately not much I can do. I spent the whole day yesterday just binary-search-building through commits history and it led me absolutely nowhere

I did get one commit to build and it was from right before our 2019.3->2019.4 transition. No further commits built properly, so I went back to the one that worked and… It failed with Handle Error 6. Restarted my PC, re-downloaded that commit, cleaned up library and everything. Will not build ever again

We are at a complete 100% loss, been looking at it since early December and tried basically every single thing…

6

Searching the web, there are similar odd python multiprocessing issues reported, e.g. here: windows - WinError6 The Handle is Invalid Python 3+ multiprocessing - Stack Overflow . The Emscripten project ( GitHub - emscripten-core/emscripten: Emscripten: An LLVM-to-WebAssembly Compiler ) uses Python multiprocessing for parallel task spawning.

However I am surprised that the issue would occur even when EMCC_CORES=1 is set. Emcripten should disable multiprocessing with EMCC_CORES=1 environment variable is set. Can you try double-checking that option? (e.g. set it as a global env. var for all users, and restart the computer for good measure)

It sounds like there is some nondeterminism in play that is causing a flaw in python-Emscripten interaction.

7

Thanks @jukka_j

I tried multiple options and collected just the error logs in the linked zip file

Before that:

  • reinstalled Unity, 2019.4.20f1
  • deleted project + downloaded and imported from scratch

There are:

  • baseline - with no system environment variables

  • just EM_POPEN_WORKAROUND=1 - which was suggested in some places

  • EM_POPEN_WORKAROUND=1 + EMCC_CORES=1

  • EM_POPEN_WORKAROUND=1 + EMCC_CORES=1 + EMCC_DEBUG=1

  • EMCC_CORES=1

  • Adding DEBUG caused a weird “PYTHON not defined” error so I added python.exe location from Unity Install to environment variables but this changed nothing so I skipped this log

I restarted Windows before each and every test

Handle Error 6… always

https://imdev.pl/share/a51d56ff-8bb9-4d34-a3fc-8a20710056c2/ErrorLogs.zip

8

Other threads on the web also mention about python multiprocessing having this bug:

In 2013 I seem to have observed that having 32-bit python and 64-bit LLVM installed can cause a "
WindowsError: [Error 6] The handle is invalid" error: detect_windows_bitness_inconsistency by juj · Pull Request #1958 · emscripten-core/emscripten · GitHub . I wonder if you have 32-bit python installed manually? Maybe removing that from PATH or uninstalling it to try could help.

Ultimately, last weekend I decided to migrate Emscripten away from using Python multiprocessing at all. See these two fixes:

That should resolve all these multiprocessing troubles for good.

Unfortunately it will take a short while until those make their way to Unity, we are targeting to get those for Unity 2021.2 release. In the meanwhile, I would recommend to work around using a different PC installation if nothing else helps. Apologies for such a nuclear suggestion, but I don’t have anything better to offer here, given that the issue is so elusive.

9

Damn…

Well, since I spent further 2 days working on it… Might as well share

I tried 3 things:

  • EMCC_CORES
  • EMCC_DEBUG
  • direct change in emscripten.py
    Like I said in last post - those did not help

But after this I started looking closely at the log and the very last thing that was linked was Live2D animation plugin, with their webgl .bc library. So I removed it (breaking the project in the process) and… it compiled fine

Now the fun part - we have another project in development that is using Live2d and builds with no issues. It’s Live2d plugin library is a bit newer so I copied it and…

It built. Almost opened the champagne but decided to give it one more try… FAIL

And that’s when we gave up to enjoy the weekend

On Monday morning I discovered that editor will not make builds either, mostly due to changes to Environment Variables and python scripts. So I removed ENV Variables, restored Python files and… NADA, FAIL

Completely reinstalling 2019.4.20 did fix it fortunately but I’m now too scared and too scarred to touch it :frowning:

PS My early tests in 2020.2 showed the game was building fine there but… There was enough broken bits that we didnt follow with it (and 2020.1 breaks the way WebGL index and template files are handled, so we would have a lot of extra work with it as well. Unstable build + those changes + release window for the game = nope, not gonna risk it now)

10

Just a quick update @jukka_j - nuclear option worked and Mac was able to build it fine. Thus we assume that so should Linux, so we will move the build process to a VM on our server. Feels safer than updating a Live game to 2020.2

11

Between 2019.4 and 2020.2 there have been no changes to the Emscripten toolchain that is used, so it is expected that whatever failures would happen in one, would also occur in the other. It sounds like there is some amount of randomness in play that the issue does not occur all the time, leading to occassional successful or failed builds.

Linux is quite different from Mac in general. On Linux it may boil down to what distro and distro version is being used, and one may need to apt/dpkg install some packages outside Unity to get Emscripten builds to work. The Handle is Invalid error should definitely not occur on Mac or Linux, that is for sure.

12

@jukka_j ok, we decided to bite the bullet and move to Unity 2020.3.3 only to… get bitten by this abomination of an error

Is there any chance this fix make it to main Unity anytime sooner?

13

@jukka_j any chance you could look here? do you maybe have a working but “not tested” Emscripten that we could replace manually just to unlock the build process? This is insanely frustrating issue :frowning:

14

I think I read conversations in our Slack about this issue just recently. If I understood the issue correctly, it could be something to do with the way that the Python interpreter takes in stdin handles, and Unity build system feeds stdin input over to the Python subprocess it spawns.

I don’t unfortunately have a fix or a workaround to propose right now, but prodded an internal conversation about this to find out more.

15

Btw, we have not received a single bug report about this, so would be good if you could raise a bug report so our QA can try reproing it as well.

16

Thanks for remembering about this

I really wish I could, but the projects are… NDAd, large and contain paid plugins. I will see what happens if I just try to build an empty test project, but since there are no errors I assume it will work without problems

Like I said before - based on the full output log I traced that the very last thing that goes to the linker is Live2d plugin. So I will also try to include in the test project and see what happens

17

@jukka_j hah. Found it

The cause is Live2d as I suspected. I tried building just an empty Unity project with their Unity plugin and nothing else - and it broke in the very same way as above. I contacted them but who knows how much time it will take them to reply, so maybe you have some idea?

It looks like during the CLI build (but not during editor build) Live2d opens… another process that… does nothing it seems, just prints the plugin version

But it IS a new process and [ERROR 6] is directly related to handling multiple processes. So I’d blame it

Unfortunately I don’t see anything within their plugin that spawns that process - so it’s probably part of the plugin’s library

Is there any way of blocking spawning additional processes during the build process? Maybe that would be enough… dirty hack to save us buying and setting up additional non-Windows build machine…

Here’s the test project - again, just empty Unity + Live2d sample
https://imdev.pl/share/d36110cb-a13c-43e0-b51e-05cc8e05f374/Live2dCLITest.zip

And it can be built using the command (adapt the paths of course)
d:\Work\Unity\2020.3.3f1\Editor\Unity.exe -quit -nographics -batchmode -buildTarget WebGL -projectPath d:\Work\Live2dCLITest -logFile D:\BuildWebGL.txt -executeMethod BuildSystem.Build_WebGL

Thanks for help

18

Sorry, cannot quite think of a way to prevent C# code from spawning a subprocess like that. It would be great if the plugin authors are able to react, and if they think the usage should be fine, have them report a bug against Unity - they can probably give more information about their required use case on the plugin.

19

@jukka_j damn, thanks anyway. Yes, I wrote them but it’s a Japanese company and I don’t expect to see much help from them :frowning:

20

@jukka_j it took some time but finally got a reply from there. Might be of interest to you so Im posting it here:

So as I understand we’re basically out of options other than Linux/Mac until… Unity 2021.2, right?