WebGL data caching is broken in recent Unity versions!

Unity Engine
,
1

Hi,

so I was writing a custom initializer code to split the humongous data files generated via WebGL builds into chunks of say 15MB, so that you can serve a WebGL build with free hosting services such as GitHub Pages and Cloudflare Pages. This was working fine with Unity 2021.3.2f1 in the past. But now, even if I reinstall this particular version, the UnityCache IndexedDB is using version 4 instead of version 3, which means it has a RequestMetaDataStore instead, storing not the file data, but just metadata. Worse yet, while you switched to using Cache Storage, and in fact I am able to patch that with the concatenated data, Unity keeps overwriting the RequestMetaDataStore entry with the empty file and apparently uses the size field in this metadata to determine the cached file size, which means it is unable to load the actual cached file properly.

This is a huge issue guys. There are three options for making this work, right now you don’t support any of them:

  1. Design your caching in a way that allows people to patch the files with custom data, like it was possible before. This already sucks, because its pretty complicated to do properly this way, but at least it was working before your latest updates…
  2. Use a file fetcher interface that allows us to overwrite the way Unity loads remote files. This way we can do whatever we want and just hand Unity the raw data. This would be easy to do on your end and make this a “sane” implementation, while (1) is more of an ugly hack.
  3. The gold standard would be to have an editor option “Limit file size to X MB” and split the files on your own. This is also easy to do on your end since you know how many files you generated and can just glue this easily back together. But it is definitely more involved than (2)

But for Christs sake, please fix at least (1), right now our whole deployment process is in the trash because we can’t provide our game anymore with a newer Unity version, and not even on computers with the same old Unity version, since somehow you still upgraded the UnityCache, even though the editor version is the same.

Why does this matter? Simply because providing Unity games generates an insane amount of traffic, we were seeing up to gigabytes per minute in peak traffic. We are providing free games and using paid CDNs for this amount of volume would just be infeasible. We are relying on Cloudflare Pages to provides near infinite bandwidth and throughput for free, giving us room for using money where its really needed and giving our customers the benefit of a world class CDN.

2

Oh wow, I got it working still somehow but this is a brittle house of cards now lol. This will need a lot of testing before we can ship this. Please consider providing either (2) or (3) above, it would help a ton to not turn deployment into a Frankenstein monstrosity.

3

After a long investigation, i have found the origin of the problem.

Unity 2020.3.40f1 and older version use the old Module.XMLHttpRequest (XMLHttpRequest.js) with :

var UnityCacheDatabase = { name: “UnityCache”, version: 2 };
var XMLHttpRequestStore = { name: “XMLHttpRequest”, version: 1 };
var WebAssemblyStore = { name: “WebAssembly”, version: 1 };

Unity 2021.3.11f1 and newer version use the new Module.UnityCache (UnityCache.js) with :

var UnityCacheDatabase = { name: “UnityCache”, version: 3 };
var RequestStore = { name: “RequestStore”, version: 1 };
var WebAssemblyStore = { name: “WebAssembly”, version: 1 };

The both use different method to use the indexedDB.

Example case with a project named “MyGame” :

Build it with Unity 2020.3.40f1 that use the old indexedDB system, the indexedDB will work as expected.

Now, if you build the same project with unity 2021.3.11f1 that use the new indexedDB system, when you will load your webgl game you will have the error “Could not connect to database: Timeout.”

If you delete the indexedDB and do the same but load a game that use the new indexedDB system and then load the game that use the old indexedDB system you will get the error “indexedDB access denied”

So we have a conflict between the old and new indexedDB code.

The quick solution that i have found to avoid any conflict is to change :

var UnityCacheDatabase = { name: “UnityCache”, version: 3 }; to var UnityCacheDatabase = { name: “UnityCache1”, version: 3 }; on the new indexedDB system Module.UnityCache.

It create a fresh separated DB that will not enter in conflict with the old one.

Also, some informations are missing on the new indexedDB system.
company, product, updated, revalidated, accessed field are empty compared to the old indexedDB system where you have all field with all informations.

I hope it will help.

Posted also here

https://discussions.unity.com/t/891742

3 Likes