iOS 15 + WebGL 2 issue

This would be great thanks, On 15.4, disabling WebGL 2.0 and WebGL on Metal in Safari experiemental settings does not resolve the Out of Bounds error on loading.

The issue has to do with WASM and not WebGL. This failure is something we’re very concerned about, and I wish there was more we can do other than pushing on Apple and hoping they fix it soon.

Thank you for keeping us updated!

I’m curious if we’ll need to wait for a new release of iOS until this is resolved.

Is there perhaps a workaround we can implement in the meantime?

I have not been able to find a work-around. The problem seems to be from the WASM size being bigger than 9MB. If you make an empty project, it works, but as soon as the WASM gets bigger than about 9MB, it crashes.

The WebKit bugzilla report is here, 237180 – REGRESSION (iOS 15.4 beta): Unity WASM builds fail to load. We’ve had Google and people at Apple check in on it, but in typical Apple fashion there’s no feedback.

We tested with Unity2021.2.15f1 and Unity2022.1.0b11.

Projects with only simple components such as Main Camera, Directional Light, and Cube loaded successfully, but when we added Image or PlayableDirector components, the loading stopped halfway through.
(Runtime Error: Out of bonunds memory access)

We also found that the problematic components generate errors when the object is called.
In other words, the problem is not caused by being included in the project, but by being used.

Also, the size of WASM in that case did not exceed 9MB either, but was about 5.3MB, and the total size of the build was about 6.8MB.

These behaviors seem very strange and do not seem to be an Apple-only problem.

I have tested with several unity versions and build sizes. It doesn’t affect by the binary size. I can make it run with build sizes over 15 MB as well. How did I make it run is a different story: I added a debugger; to somewhere in js code. If I wait enough and press continue on my js web console it started working correctly without any wasm issue. WebGL 2 issues still remain so I was forced to work it with WebGL 1.

That’s an interesting finding @sercand - sounds like it might be something to do with the wasm tiering-up to a higher JIT whilst executing that causes the problem. I’m still digging for workarounds (and also pushing Apple on the bug report Brendan linked) as this is a pretty major issue for us.

@brendanduncan_u3d - Any chance you could try a -s WASM=0 build? Would probably be big and slow, but might actually be functional. I’m trying to dive in and debug what’s going on with your examples from the webkit bug but don’t know the Unity webgl build system, and don’t have Unity installed here at the moment

@simon_zapper ASMJS is deprecated by browsers, so it wouldn’t be a viable solution, and I’m not sure it would provide useful info for debugging. I am currently dedicating my time to finding an answer to this since Apple hasn’t been helpful thus far.

The problem is coming from some data is being evaluated differently on iOS15.4 than in other environments. From my latest investigations, an IL2CPP class type ID is saying it’s 0, where it’s never 0 on any other browser. That 0 causes the IL2CPP class object to return as null, and I suspect some following code is updating the class data, not null checking, and writing to the null ptr. Emscripten has a memory heap validation check that it does to see if something wrote to the null address. On older Emscriptens, pre Unity 2021.2, it printed this as a warning to the console. On newer versions of Emscripten, it calls abort.

For experimental purposes only (not proposing this as a solution!) if you edit the project build’s framework.js file, find the code
abort(‘Runtime error: The application has corrupted its heap memory area (address zero)!’);
and replace it with
console.warning(‘Runtime error: The application has corrupted its heap memory area (address zero)!’);
Then a theory is that it would continue running, and that “corruption”, despite not being good, wasn’t in fact critical.

But I will continue narrowing it down to exactly where this problem is coming from, and find an exact point of failure that either we can patch on our end or get Apple to fix, since they don’t seem to want to fix anything that they have to spend any time investigating themselves.

Thanks for the update Brendan.

I had noticed the stack cookie check code was where things were failing on your minimal clear-color example, but I believe other projects show other errors. Looks most likely just a Safari VM bug that corrupts random stuff when running perfectly valid code, but obviously if you are able to narrow it down to a trigger in the Unity code that can be fixed that would be brilliant.

Is it possible for users to customise the emscripten command lines for the webgl build? There’s various flags and options that I’d be happy to play around with if so. I added an issue on emscripten GitHub just mainly to get their advice too - that’s here: iOS 15.4 issue with Unity wasm - ideas for workarounds to try? · Issue #16526 · emscripten-core/emscripten · GitHub

I’m also dedicated to this at the moment, looking for workarounds. My current plan:
1 - Try a wasm2js converted build (which I expect to be functional but perhaps too slow to be useful)
2 - Look at wasm-split to see if we can separate the wasm into smaller modules and if that avoids the bug
3 - Try to bisect in the open-source webkit code to identify the breaking commit (if it is in fact broken there) to give better hints to Apple engineers

We’re also leveraging any contacts we can find to bring more pressure on Apple in the hope for a fix and a point release ASAP.

I suspected the Cookie abort might not have covered everything, and the corruption that triggered that might be more wide spread. Just looking for anything that might help in the meantime.

I’m not sure you’d find the problem in open-source webkit, since they use a separate branch for iOS, and it doesn’t happen on desktop WebKit/Safari.

We’ve also been pushing on contacts at Apple…but it’s Apple.

You can set Emscripten linker args in ProjectSettings/ProjectSettings.asset, emscriptenArgs value.

You might also be able to set Emscripten compiler flags, at least for the code generated by IL2CPP, by setting the EMCC_CFLAGS environment variable.

Thanks for the hints. Assigned myself a seat and downloading the editor now. Looks like the -s MAYBE_WASM2JS only introduces minimal runtime changes, so I’m still trying to patch your built examples first.

Haven’t tried to repro with open-source WebKit yet as finding a workaround is priority #1. Apple in general are very opaque on this stuff, but at least some in the Safari team work slightly more collaboratively on the webkit bug tracker. Not the case yet on this bug, but I live in hope - we did get a few other critical things fixed during the iOS 15.4 beta phase, but unfortunately this one wasn’t on our radar until Wednesday :frowning:

I am seeing some promising results from updates to IL2CPP I just pulled into my build of Unity, the minimal tests that were crashing, are no longer crashing or having the errors I was seeing before. I’m doing a build of a bigger project now. I’m pessimistically hopeful.

Not quite. The new IL2CPP fixed the issues with that minimal test, but larger projects still have errors. I’m still hopeful we can find a way around the issue.

Update: I may have found a solution to the problem, or at least a specific compiled file that causes the crash, which I can swap out with a working one to get a crashing build working. If it works out, this would be something we can fix on the Unity side and wouldn’t require waiting on Apple to fix whatever they did to break things.

The other bad news for iOS 15.4 is that WebKit introduced a number of very bad regressions in the WebGL graphics side of things, and even without the WASM crash, rendering will still be bad for most things. The graphics problems will be easier to work with Apple/Google to fix.

Thanks for the hard work Brendan! If I may ask, how will the fix be rolled out/shared with? I am too facing this issue where my users who are playing our browser based Unity WebGL games and upgraded to the recent iOS 15.4 build could not have their games loaded and getting stuck at 90%

Great updates Brendan, thanks for your great work on this so far. On the graphics regressions, is webgl1 still working well enough? We noticed some broken three.js webgl2 content seemingly triggered by the use of SRGB8_ALPHA8 textures in the beta. Haven’t yet noticed any regressions with our webgl1 content at least.

I’ve managed to get a wasm2js build of your minimal clear-color developer build working on iOS 15.4 too, and in a way where it’s opt in via a ?_wasm2js=1 query string for testing. Looking into applying the same thing to a more complex example project now.

I haven’t investigated webgl1 on the new iOS yet, it might work since I don’t think WebGL1 uses the Metal back-end.

I found a specific line line in a file from IL2CPP that triggers the WASM crash on iOS. In a specific circumstance, memcpy is failing. But I have a “fix” that seems to resolve the issue, even if I don’t fully understand why it fixes it. I also still don’t understand why this particular circumstance causes memcpy to fail on just iOS 15.4’s WASM VM. I’ll discuss with management about how we’ll roll out the fix, assuming others can replicate my success. The worse case, there is a simple way to do the fix yourself without waiting for a Unity release, but I need to do more testing before I can recommend it.

3 Likes

Could you post the workaround, so we can get our Games to work temporarily?

Oh and thank you for all your hard work!

Since it can’t get any worse than it already is:

In your Unity installation (on MacOS, in Unity.app/Contents), edit:
il2cpp/libil2cpp/metadata/GenericMetadata.cpp

Find the line:
const Il2CppType* GenericMetadata::InflateIfNeeded

Add:
#pragma clang optimize off
to the line before that function to turn off compiler optimizations for the function, and add
#pragma clang optimize on
after the end of that function to turn compiler optimizations back on again. This will disable compiler optimizations for that one function. In my tests, this has been enough to stop the crash.

In your Unity project, delete the directory Library/Bee/artifacts/WebGL/il2cpp to be sure it picks up the changes to that file. Also make sure your iOS Safari isn’t trying to load a cached version by going to Settings / Safari and selecting Clear History and Website Data. I’ve spent plenty of time chasing ghosts by not clearing a cache.

Memory errors are notoriously hard to track down, especially when only one platform, and one version of that platform, has the memory corruption, and that platform is iOS.

22 Likes