I’ve just upgraded a project with extensive unit tests from 5.4 to 5.6 and the changes to unit testing feel like a backwards step to me.
In 5.4 we had a unit test runner window that was:
- moderately fast
- had search filters for finding tests by name
- had filters to hide passing/ignored tests
- ran on recompilation
In 5.6 the external tools are dropped and we have new internal tooling. I was excited to see this! However Unity seems to have merged the unit and integration aspects of the old tools, and the new UI leads a lot to be desired:
- it’s achingly slow, probably because it’s serialising a bunch under the assumption my tests will touch the scene hierarchy. However they do not.
- the UI has no filtering, so a single fail in hundreds of tests is hard to find
- it no longer runs on recompilation, which is a blessing in disguise considering the speed
Ok, I thought, maybe this is now an integration-style tool and I should look elsewhere for a unit-focused option. However monodevelop on OSX still can’t run the unit tests out of the box, failing instantly with an invocation error.
I even went so far as to try and get the old tools test runner UI running under 5.6, but it does not seem willing to cooperate with 5.6’s version of nunit.
I hope you understand that it’s infuriating to lose so much functionality. I can see how unittesting monobehaviours is now more manageable, but people like me who were writing unit tests in unity’s previously preferred manner have had their workflows greatly altered.
I’m looking to understand more about where the new tools are heading, why these decisions were made, and how I can improve my workflow. Can anyone help me?
Hey, yes, we had merged several tools together. 5.4 could not run tests in playmode if I remember correctly. Now they can run tests in both playmode and editmode. We have also updated NUnit version. Some features in UI were dropped due to new NUnit and we had several performance regressions in 5.6.
Although, in latest 2017.2-3 versions we have returned the UI features, filters, categories, passed, failed, ignored test counters. We have also fixed all performance issues in 2017.*
1 Like
Hi Andrew,
I am not the developer that worked on making test automation natively supported in Unity, but I will try to reply to your concerns. Thanks for voicing your opinion here. I can assure you it will be passed to the owners of this feature.
The native testing tools in Unity were not intended to be a 1 to 1 match with the original Unity Test Tools package. So, indeed, if you just upgraded to Unity 5.6 it feels like the functionality has regressed. Part of the reason for why this functionality seems incomplete in 5.6 is that we only added support for what was ready at that time. We did get quite a lot of feedback on that feature and we improved on it in Unity 2017.1 and we have more improvements available in the 2017.2 public beta and planned for 2017.3 and beyond.
For example, this manual page shows an overview of how the feature looks like and works in 2017.1 (including a list of its current limitations): Unity - Manual: Unit Testing
For 2017.3 we made a lot of performance improvements, added back test fitlering (looks like this: Imgur: The magic of the Internet), the option to rerun failed tests and so on.
I am not sure what you mean when you say you want to run unit tests in MonoDevelop on OSX. The tests can only run if started from Unity’s own test runner, since they are not unit tests as such. The look similar to Unit Tests, but you need a Unity process running in order to execute them.
As to improving your workflows, I would be happy to understand more about your project and maybe even take a look at it and upgrade it to the latest versions of Unity to see how much that helps. In general, we recommend you keep your project updated to the latest stable version of Unity since we don’t have the bandwidth to backport new features and improvements (we mostly backport critical bug fixes).
Please DM me if you want to share your project with us.
Thanks for your replies, both. I’m glad the feature is improving but you can understand my frustration that a feature I found useful in 5.4 was deprecated before the replacement was fully ready.
I’m not sure @ElvisAlistar what you mean by “not unit tests”. Our tests don’t touch gameobjects or monobehaviours, as has been the unity advice since the testing tools were introduced, and just need unity dll access for vector3 access and the like. They’re made to test unit-sized assumptions and to run fast. We really only used the unity runner because it was extremely convenient - setting up our own monodevelop testing project to reference everything would mean having two instances of monodevelop open, and constantly updating references! If you have any resources on how to make that outside-of-unity experience work better, I’d be interested in seeing them.
Are you implying the new tools are really only designed for integration testing? I think they’re important but you should have both short fast unit tests and bigger more expansive integration tests in a successfully-tested project.
Hi Andrew,
Im one of the developers working on the Test tools integrated into Unity. And understand your frustration.
There are some differences in how the test runner executes, compared to ealier, that makes the test execution slower. There where some performance optimizations, as Elvis says, but are not nearly as fast as we want it to be.
We know how to solve this and are working on it.
For your trouble of executing tests in MonoDevelop, it’s known. It was not really meant for developers to use our nunit version to execute in VS/MonoDevelop. We had to make some changes to nunit, and one of the changes was to remove the reference to System.Web.dll, nunit was referencing this only for one interface, and that interface is what the nunit runner uses as API. This resulted in the Invocation error that you are seeing.
We are looking into how to solve this, but I dont see it happen any time soon. One of the reasons is that the nunit runner needs to implement the same interface.
The new test tools are not only for integration testing. What I think Elvis is saying is that because you have the editor, you will still have state that we can’t promise will be cleaned up, you need to do it yourself. But that is not something new, compared to ealier.
You can test unit’s with the test tool, it’s entirely up to the user.
@HaraldNielsen thanks for replying! I appreciate the dialog. This is probably much more suited to a face-to-face conversation, but that’s the nature of things.
The frustration from my end comes from the apparent downgrade in functionality. Even if it’s temporary, it’s hard to believe that (eg) shadow rendering could be overhauled and take multiple major revisions to come back to feature parity.
From what I’ve heard so far, I’m not feeling terribly confident that the previous workflow - write simple small decoupled unit tests, run all tests on every compile - is ever going to work under the new system. Whatever changes you’re making to better support direct testing of monobehaviours is going to be too costly to make frequent runs possible. I hope I’m wrong!
It’s understandable that you want to make automated testing easier. That’s a good aim. If there’s anything I can do to help you do that while also maintaining the old workflow, let me know.
