We have recently moved our project from Unity 5.5.0 to Unity 5.6.1 and noticed the Test Runner was changed and now supports play mode tests and updates the results as it executes. That’s great! However, we also noticed that:
Tests are running much slower now. For few lengthy tests, about 3 times slower (of course it depends); for many quick tests it’s actually much worse by orders of magnitude (see post below).
There are no icons to select only tests that failed or that were not run yet. This was very useful!
There is no number that counts the number of tests (just a detail).
Why these regressions? Will those be fixed in the future?
It seems that due to the results updating while executing, there is a large interval between running each test. If you have 1000 very quick tests they will now take a very long time, whereas before they finished almost instantly.
In fact, I tried this with 1000 tests that look like this:
[Test] public static void Test1() { Assert.True(true); }
[Test] public static void Test2() { Assert.True(true); }
[Test] public static void Test3() { Assert.True(true); }
[Test] public static void Test4() { Assert.True(true); }
They took about 21s on my machine (i7-3632QM), having only trivial code as seen above.
Most of our actual test scripts went from taking a few dozen ms each in Unity 5.5 to taking hundreds of ms in 5.6. And as I’ve just now optimized them, they should run in a few ms but take about 6 seconds total.
Can we return to the old test scheme somehow? As it stands this is having a hit on our development time.
Just reported this as a bug. At the moment I have no suggestions for other people running 5.6… I haven’t tried to get old versions of the unit testing tools into our project. Maybe that could work it in the meantime…?
EDIT: just tried the tests in Unity 2017 (2017.1.0b6) and it’s even worse. Hangs for a while before even starting to run the tests, depending on how many you have. Also reported that. D:
UPDATE: for the hang problem in Unity 2017, it had already been reported by someone else: this is the page. The general slowness in 5.6 is still a problem, however.
I would like to note that I have the exact same problem. We have around 1400 unit tests in our project and recently upgraded from 5.3.41f to 5.6.1f. The unit tests that formerly took a few seconds now take over a minute (!!) to run.
I’m seeing the same problem. As soon as you start using just a few [Values(...)] attributes or similar, as the amount of permutations grows it becomes unbearably slow.
It’s easy to see this with the following test:
using NUnit.Framework;
public class PerformanceTest
{
[Test]
public void ManyTests(
[Range(0,1000)] int testInt)
{
Assert.IsTrue(true);
}
}
This is just 1000 executions that should be sub-millisecond each, but in Unity 5.6 this takes ~18 seconds for me.
Also, the lack of filters is super annoying – we used the category filter all the time.
Also, is there any way of starting the test runner from script? I just recently implemented ITestRunnerCallback / Batch.RunTests() into our build script, it worked nicely and allowed us to check the unit tests on each build. All those classes appear to have been removed in 5.6, is there any replacement for this use case (running tests from scripts)?
We’re facing the slowness issue as well, but it seems to vary between machines, even after clean installs of Unity (including deletion of all settings).
In addition, we’ve noticed the following regression: The unity test runner no longer treats NUnit’s Explicit attribute correctly. These tests are supposed to be run if selected by a category, but now they only run if executed individually - both in batch mode (build server) and in the editor. This is the same issue as noted by @sybo_jon above, about the category dropdown disappearing from the test runner GUI. For us, this is a big deal, as we have a significant number of tests that are unity-independent and run in the IDE, so we need a way to run different sets of tests in different environments.
I just want to say that we’re experiencing the exact same problems. Its also very annoying that you can’t alt-tab away from Unity while the test runner is running (it will stop and wait until you tab back).
We’ve gone from being able to run all our tests in a few seconds, to it taking 5-8 minutes.
I’ve noticed that this has increased the amount of failing tests, since developers don’t want to wait that long for them to run.
I think its time for Unity to tell us what the plan is, and when we can expect improvements.
Another one: The XML format generated by the test runner is no longer valid NUnit XML, because the required root element is missing. Build servers tend to love that…
The comments everyone is writing just make me realize this is even worse than I thought! Would also really appreciate if someone from Unity could talk about this a bit.
The new test runner also doesn’t support tests in managed plugins (DLL assemblies). We use DLLs exclusively for our project and this is making our tests totally unusable.
It’s clear to me the new Test Runner is not quite ready for prime-time. Can someone from Unity comment about this and potentially revert back to the old Test Runner somehow?
Ad 1. We understand the tests became slower. The reason behind it is because we changes the way the runner is executing tests. Before it used to run all tests in one frame. We improved the runner to make it possible to yield/skip frames from tests and therefor we had run each test in a different frame (which runs in the same framerate as EditorApplication.update). We will try to look at improving the execution time at some point while keeping all the new features.
Ad 2 and 3. Lack of UI filter is a regression. We simply didn’t have enough resources to focus on reimplementing that feature. It’s again something we will hopefully fix.
There is no public API for it but you can reflect the Batch class and invoke it like that (keep in mind the API may change)
We are also aware of this issue. For now I suggest postprocessing the result file before parsing it.
That, again, is on our TODO list.
Apologies for regressing in funcionality in some of the areas but we needed to rewrite some parts of the runner in order to make it even better and we simply lack resources to fix it on time. Personally I don’t work on the test runner anymore but I have notified people that will look into the problems.
I thought that was the case, but it really is a huge difference for non-frame dependent tests (or non-Unity-dependent tests, for that matter). Though I realize the following may be a little maintenance nightmare, I suggest adding a toggle so that we can run tests sequentially and without breaks, using the old system (while the new one isn’t up to par).
Secondly, do you know of a way that we can, right now, run the tests in the old Editor Test Runner - or similar tool - while keeping Unity 5.6? I believe everyone in this thread would like to know this. Anything Unity Technologies can provide in separate, like the old tests module itself?
Thanks for replying. I could use the Batch-class with reflection but what about the callback interface, is that still around somewhere? In 5.5 I had a nice progress dialog that listed the number of tests that succeeded, failed etc. so I need the callback to be able to do that.
@Tomek-Paszek reading you I’m thinking that there is no dedicated team working on test runner because you broke so many things months ago and there are still broken…
I haven’t tried to get old versions of the unit testing tools into our project. Maybe that could work it in the meantime…?