Slow Unit Tests

Unity Services
1

Hello,

We are using Unity Cloud Build to build our project and we want to use unit test validation for each build as we have high quality requirements. It turns out that turning on unit tests increases the build time by an impossible amount, which seems to lead to a cancellation by timeout (cf screenshot).

1642×354 48.8 KB

Locally these tests pass in 20s seconds. I can understand these test taking 3x more time on UCB, even 10x more time. On a smaller project, the local vs UCB time factor is 35x, which is still fine at that scale (10 minutes build). But 360x more time ? Unfortunately this is unusable for us.

1/ Could the job summary indicate the reason of cancellation ? It is nowhere to be found, even in the full log. I just receive a “a user cancelled the build” email.
2/ How can I act on that run time? Must I optimise my code? Is there something you can do to speed up the test runner? Can we pay for faster CPU?
3/ BTW, is it possible to run play mode tests on UCB?

The unit tests are requirement for us and we plan to increase the quantity and variety of tests (play mode tests). If build times increase exponentially like we experiment it, we will have to find another solution.

@dannyd Can you help on this ?

2

As I keep investigating this problem, I have reduced the number of tests to see which spent too much run time, but as I am progressing, I getting closer to an unexpected conclusion: the test runner hangs pretty quickly after starting and that hanging may not be tied to a particular test.

Again, I am not done testing, and I have yet to test running only one single and simple test (Assert.True(true)) or even try the test runner with no tests, to conclude that my tests are not responsible for my UCB jobs stalling.

Has anyone else had issues specifically with running tests in the Unity Cloud Build with the “Latest 2017.2” ?
My other project using the “Latest 5.6” has no test runner problems.

3

With almost all tests removed, the test runner keeps on hanging:

15084: [Unity] Running tests for EditMode
15085: [Unity] UnityEngine.DebugLogHandler:Internal_Log(LogType, String, Object)
15086: [Unity] UnityEngine.DebugLogHandler:LogFormat(LogType, Object, String, Object[ ])
15087: [Unity] UnityEngine.Logger:Log(LogType, Object)
15088: [Unity] UnityEngine.Debug:Log(Object)
15089: [Unity] Unloading 66 Unused Serialized files (Serialized files now loaded: 0)
15090: [Unity] System memory in use before: 244.7 MB.
15091: [Unity] System memory in use after: 245.0 MB.
15092: [Unity] Unloading 60 unused Assets to reduce memory usage. Loaded Objects now: 4245.
15093: [Unity] Total: 5.884269 ms (FindLiveObjects: 0.492589 ms CreateObjectMapping: 0.175800 ms MarkObjects: 5.118917 ms DeleteObjects: 0.095841 ms)
15094: [Unity] Refreshing native plugins compatible for Editor in 20.00 ms, found 6 plugins.
15095: [Unity] Preloading 1 native plugins for Editor in 1.81 ms.
15096: [Unity] ----- Total AssetImport time: 0.186556s, AssetImport time: 0.000000s, Asset hashing: 0.000000s [0 B, 0.000000 mb/s]
15097: [Unity] DisplayProgressbar: Test Runner
15098: [Unity] UnityEngine.DebugLogHandler:Internal_Log(LogType, String, Object)
15099: [Unity] UnityEngine.DebugLogHandler:LogFormat(LogType, Object, String, Object[ ])
15100: [Unity] UnityEngine.Logger:Log(LogType, Object)
15101: [Unity] UnityEngine.Debug:Log(Object)
15102: [Unity] Refresh: detecting if any assets need to be imported or removed … Refresh: elapses 0.185593 seconds (Nothing changed)
15103: [Unity] Refresh: detecting if any assets need to be imported or removed … Refresh: elapses 0.191927 seconds (Nothing changed)
15104: [Unity] Cleanup mono

Nothing else happens after that. Once two hours have passed, the jobs cancels itself.

4

Ok, I now know for a fact that my tests are not responsible for UCB hanging: I have reduced tests to only one file, one class, one method. Here it is:

using NUnit.Framework;
namespace UnitTests
{
    public class TheMostSimpleTest
    {
        [Test]
        public void TrueIsAlwaysTrue()
        {
            Assert.True(true);
        }
    }
}

I also realised that my Unity5.6 project that “has no unit test problems” actually has one: the “Tests” tab in UCB reads

Even though the project HAS tests, and that they were run (builds with tests take longer and I can see the tests in the full log).

5

Sorry for the slow response. There are some lingering issues around Unity hanging on exit in certain versions of Unity 2017.2 (although it’s still not clear exactly which circumstances, unit tests seem to be a more likely cause in some projects). Have you tried using the latest patch release of 2017.2 rather than using “Latest 2017.2”? Those “latest” versions generally point to the latest final release, rather than latest patch release.

6

Thank you @dannyd for the answer. I’m starting builds with 2017.2.1p1 before leaving the office, I’ll check tomorrow their status.

7

Hey @dannyd , unfortunately explicitly specifying the very last version of 2017.2 didn’t help the build, it is still hanging and “time-out-ing”.

On the positive side, I can now see an explicit message at the end of the full logs:

To make it even better, the “summary” tab should show a very short reason of failure or cancellation such as: “compilation error”, “cannot checkout git repo”, “execution timeout”, “tests failed”, or whatever.

8

Trying to narrow down what was failing, I tried to compile our Unity 5.6 project also with 2017.2 (and 2017.2.1p1 explicitly yesterday). To my surprise it shows a valid build, but this is another different bug: even though the tests are enabled and required, they are actually never run.

The editor’s assembly doesn’t compile, which is actually what I expect since I didn’t even tried to port the code to 2017.2:

Which UCB doesn’t seem to mind:

I can MP the URL or ID of the project if necessary.