Scene tests with LoadScene (GitLab) CI very slow due to deserialization

Unity Engine
,
1

Recently we’ve began working with running tests in GitLab CI through GameCI’s Docker images and configuration files. We are also using Zenject for our services, and are writing tests that load the scene through its testing framework (along with our service container) and test the application’s behavior this way. For now these are simple, such as loading the scene, clicking a button on a window and checking if it is visible afterwards, and clearing the scene again (to get a clean slate for the next test).

This works fine locally, and tests are reasonably fast, about one second per behavior test. For some reason, though, in GitLab CI, these are unacceptably slow - each test takes anywhere between 50 to 120 seconds.

Analyzing the Unity logs in CI, it turns out deserialization of the scene is the root cause. Here is an excerpt from a single test:

Loaded scene 'Assets/Scenes/Main.unity'
    Deserialize:            106748.398 ms
    Integration:            1540.717 ms
    Integration of assets:  507.097 ms
    Thread Wait Time:       -432.781 ms
    Total Operation Time:   108363.430 ms

The first test is usually a little slower, so all consecutive tests are similar to the following:

Loaded scene 'Assets/Scenes/Main.unity'
    Deserialize: 61155.941 ms
    Integration: 744.542 ms
    Integration of assets: 292.142 ms
    Thread Wait Time: -224.852 ms
    Total Operation Time: 61967.773 ms

Which amounts to about 60 seconds per test. I realize GitLab CI is less powerful than a development machine, but this seems extremely slow. GitLab CI has little problems with these kind of tests in npm, PHP, or other languages, which are also computationally expensive (and usually not native or use JIT, so one would expect Unity to be even faster).

We have code coverage enabled, but are limiting it to our application’s scripts, but it is possible that it is impacting it anyway (?). If that turns out to be the case, we could try disabling it, but in the end this would also be unfortunate since we would lose coverage entirely.

Also, we use the YAML scene format to get better results when merging in Git, which may also be related to (de)serialization performance.

Does anyone have any pointers towards improving the situation?

2

Not sure what you’re mean regarding “Gitlab CI”. Gitlab has runners running the CI jobs, depending on the hardware of the machine they run on. So the issue here seems to be the machine running the runners and not Gitlab itself.

At my company we run over 8 builds with tests simultaneously on a 16-Core Ryzen CPU with a 5GB/s NVMe SSD. We have none of the issues you’re describing. Aside from us not using the Code Coverage package we seem to have identical project setup.
If the issue is with your code then you should most likely also notice these issues on your local machines.

TL;DR I think the machines running your Gitlab runners are just slow? Are they your own local ones or are you using a cloud solution for the runner machines?

3

Thanks for the response @PeppeJ2 . I was indeed referring specifically to the GitLab CI runners (on GitLab’s cloud infrastructure), not GitLab CI itself necessarily. I expect the cloud runners to indeed be slower than my development hardware (where all of this is indeed very fast), but wanted to ask for input on the forums anyway, since the difference is rather extreme, which seems odd compared to other languages and technologies native to CI environments.

Hosting our own hardware using GitLab Runner and attaching them to GitLab CI is also something that we are considering, so it’s helpful for us to know that these are working well for you, thanks!

4

Update from my end: I tried using an Azure Debian VM since my original post, which improves the situation, from anywhere between 50 and 120 seconds deserialization per test to about 20 seconds.

Since we have over 100 scene tests by now, this still amounts to 2000 seconds (~30 minutes) being spent on Unity just deserializing scenes. We could use a stronger Azure Debian VM, but we already have to invest an additional configuration and running cost into these VMs just for Unity, so we’re not too keen on doing this. As a result, we’ve just stopped using CI with Unity for the time being and run tests locally again. (Personally not happy about that, but it is what it is.)

(In a related fashion, I’ve also [reported this as feedback on Unity test framework 2.x.]( Unity Test Framework 2.0 ready for feedback page-2))