In the enterprise development world, there seems to be a few handful leading frameworks for doing unit tests (referred to as the xUnit family of frameworks such as NUnit, xUnit, etc).
When approaching this subject with Unity3d, it seems there are multiple different options.
A short Google search reveals many different frameworks for unit testing with Unity3d, however it is unclear whether any of these are still actively being developed/used by developers.
I would like to enlist (with your help) the current available unit testing frameworks for Unity3d (or automated testing frameworks, as some other forms of testing may be required that are “larger” in scope than unit testing).
Also, i am interested in knowing what sort of automated testing are you doing for your games?
Things like proper “unit testing” where taking a single method, invoking and asserting against some conditions in an isolated way, testing of interaction between different objects (without having to start a “scene” and simluate using the engine), throwing “test” objects into real game scenes to test certain things, creating specific test scenes to check code, etc.
Any help will be of value as i’m trying to create a single and coherent source of information on this subject for myself and for other developers usage.
Frameworks/tools i’ve stumbled upon were:
*Comments/opinion/feedback on these will also be great.
I am actively developing Uniject, which is a testability framework and not merely a test runner.
Uniject lets you write genuinely testable code because it actually mocks UnityEngine. It lets you execute and verify almost all your code without even firing up the Unity editor.
Every other testing framework I’ve seen merely shoehorns an XUnit runner into the editor, without addressing the more fundamental problem of the impossibility of unit testing code that has dependencies on concrete types.
Take a look at my most recent post that shows some of the goodness on offer.
On the more general question of what unit testing practices are prevalent in the community, I have tried to answer the very same question.
My conclusion to the lack of response was that almost nobody is unit testing.
I believe this is because of my earlier point - out of the box Unity scripts are fundamentally untestable without a framework like Uniject.
I will definitely check out your project, and am also willing to participate during my free time to donate to the project in case i find it fits my needs
In reply to your assumption that people don’t unit test, and that unity scripts are non unit testable – My question was more broad in the sense that i am not strictly aiming at unit tests, but in fact i am interested in all types of automated testing frameworks that are available to Unity (this includes integration testing for example, which is not aiming to isolate a particular method or a piece of logic).
Integration testing may be more prevalent. Even with Uniject I find myself writing integration tests more often than unit tests, since I’m normally interested in the interactions between two or more of my own entities.
Uniject makes integration testing very straightforward because it can instantiate any portion of your codebase from a single component to an entire scene - as ever, without any dependence on Unity.
I’m very interested in Uniject. I’m come from TDD mindset and have always followed standards and best practices to write software.
I was shocked knowing that the game development practices do not highlight testing. I actively asked famous indie game devs (Jonathan Blow, Tommy Refenes …etc.) about testing their games and they basically said that they only do build testing and that’s about it!
I’m more than willing to even donate money to help you out with this project!
I have few things that I would like to ask:
1- Can you run the tests from the command line? Effectively this would be useful to hook up with a CI server such as Jenkins or Bamboo.
2- Do you have examples of complex interactions that are tested with your test framework? You only have a simple example on your website.
3- Can you test network behaviour as well? e.g. Loading an asset from a server then using the asset in a scene.
4- Does your test framework generate reports after the test has finished? what kind of formats for reports does it output? (i.e HTML, JSON, junit xml …etc.)
5- Do you have documentation for using your test framework?
6- Does it support a specific Unity3d version? or works on all ?
I find the approach of mocking away all the engine very unhealthy. TDD aim not only in having a tested code base in the end, but also a code base that is better coded because you had to decouple and think about how to get your code testable in the first place. The theory is that code that is easy to test is a good code altogether. By mocking all the engine, you are not enforcing any decoupling between the engine and your code, not that you’re build a multi-engine game or something, but decoupling your code brings a lot of other good stuff to your code base(among them, testability). In the end, you may have a bunch of tests testing your code against the mocked up unity3D, but a lot of behaviors may be different between the mock and the real deal, and since you’re not decoupled, you rely on those behaviors… It’s better than no tests.
The approach that I used, and that my tool supports is decoupling your game code from the engine. No business logic class should inherit from MonoBehaviour, unity is just the presentation layer of your application, decouple it the best you can. If you got to mock up 300 classes just to test a component, you’re not doing unit testing at all.
Having totally decoupled code (not even relying on MonoBehaviour) makes your code testable, however it gives your more concerns to deal with in order to execute your tests…
For example - Unity automatically calls Awake, Start, etc for you. In order to “simulate” this behavior you’d have to perform these yourself for your test.
That’s just one example, there are probably many others.
Also, while unit tests are important, the bigger picture (integration between different parts to form the game) is also important, what techniques did you use for these sorts of tests?
It’s important to keep in mind why are you writing automated tests in the first place. For me, it is to be able to change things easily and rest assured I didn’t brake anything(refactoring). 90% of the bugs should be catch-able on the unit level of testing and using TDD, if not, you probably are not decoupled enough. For that 10% that go thru, generally they are 'cause I did something wrong with my TDD, forget to write a test or something, so I write a unit test for that case and go on with my life. But there are some bugs, few, disastrous and hard to catch bugs that just don’t go down with simple unit tests, on that rare occasions, I do one of 2 things:
1 - Hardcode a scenario; be it a unity scene, a giant “unit” test, an integration test or end to end test
2 - Trust my QA team to catch it =)
The problem is that people forget that most of your tests should be on the unit level and try to come up clever ways to do integration tests and end to end tests instead of making their code unit testable. It would all be fine if were not for the fact that end to end and integration tests make your code more brittle, every change you do, you have to basically rewrite most of you integration and end to end tests. That fact hurts the main purpose for what I do testing, LESS of this tests is better. So I only do tests that are not unit tests when I admit that I’m not good enough to crack that nut, I’ve failed, so I write an integration test. I do so little “other” tests that I don’t need a special tool to do it.
I used uunit for some time to notice the problems in unit testing monobehaviour, then to make it work I´ve decoupled test game logic from monobehaviours. That works for most of situations, with some exceptions, like updates and movements, coroutines with wait for seconds, and the biggest problem the WWW call.
Never used uniject but I guess it will be hard to convince my team to use a framework that decouples from unity to unit test a game, for me unit testing is just like Erick said, this week I´ve decided to create my own unit test framework, and just today had the time to start the job.
Never used another framework, and go interested in the utest(it will be hard to convince a company to buy, even by the price of just 10 dollars, is hard to convince a company that unit test will work great in games, mainly because anybody saw any efficiency in it).
Yet there are some features needed that is missing, and monobehaviour is my greatest problem.
I’ll be glad if anyone can suggest any new stuff.
Things like WaitForSeconds and WWW should always be mocked for client testing, you need to check that your code is calling them correctly and responds correctly to various outputs (e.g. server errors) but you don’t need to check that your actual server behaves correctly, at least not through unit testing from the Unity client - you should have a separate test suite for the server.
I wish I knew Uniject existed, may have helped during development of a tool I worked on. I spent a couple months working on something called Testity: https://github.com/HelloKitty/Testity which I’ve decided to temporarily cancel since it was so large. The idea was to create interfaces for all components and services in the engine, some are implemented, and write adapters for the Unity3D types.
Then on dll import it used Rosyln code generation to build out MonoBehaviour equivalents and boxed/adapted the Unity3D types into the adapters and injected them into the classes you’d write with Testity that were sort of like MonoBehaviours but without the dependency of the UnityEngine. This let you write everything against interfaces and unit test with mocks.
I stopped because it was just too large in scope and I ran into some cases where I wasn’t sure how I should handle the code generation for things such as: collections and how to handle classes marked [Serializable] as single fields or collections.
Most things were handled, even UnityEvent, but the above was left unfinished and the Unity3D components and services weren’t fully fleshed out.
Edit: Why can’t Unity Technologies add interfaces to their components so we can mock them? Would solve this problem much easier.
So I’ve recently been getting into automated testing with Unity. I was a bit disappointed with the built in unit testing because it can’t be run in playmode. Maybe it’s because I don’t come from a previous domain where unit testing had its philosophies, but I don’t see the point to mocking things out to that extent. Why not run tests you can yield in playmode?
I bought both uUnit and FUnit and will be trying both out. I have a new state machine class I build and I might try testing with both classes. I’m keeping NSubstitute around but moving it out of the Editor directory. I’m also going to keep the integration tests tools and encourage our non-programmer devs to learn those tools. However, for unit testing, I want to run in playmode. I don’t have to jump through hoops to have the game logic completely decouple, at least not when it doesn’t make sense to. Sure, I’ll keep an eye out and try to make as many of my classes POCOs, but I’ve seen some examples of people going to extremes of separating MonoBehaviours from the logic when clearly their logic is mostly time based.
I’d love to hear people’s opinions on either FUnit or uUnit.
What are uUnit and FUnit? can you add links to these? are they still being actively maintained?
Regarding unit tests / integration tests – i know that there’s going to be a new kind of built-in test called a “play mode” test. It had surfaced somewhere around the 5.5 betas, but i guess it’s still not ready for use. Basically i think it’s going to be support for running NUnit still tests, only during play mode.
You can achieve something similar to this already today, by using the Unity Test tools package (it’s called “integration tests”). These are tests that are made of a scene, game objects and code that asserts for specific conditions. You can check out my posts on the matter here:
Thanks so much for the reply. This will be a large post, I’ll add some headers…
FUnit and uUnit
So I don’t think either are maintained anymore, but they work. However, I tested both last night. Here are the packages;
The promise of both is the ability to do “yielding” tests. They run in playmode and you can use yield statements to control the timing of the tests. This is great, cause now you can more easily test MonoBehaviours.
So both have their issues; uUnit is the more fully featured of the two, having a better assertion library and more attributes. However, it has a major flaw; you can’t select which tests you want to run, it seems to just run all tests found in an assembly. I don’t know if I can just return a namespace, I could investigate that further. If I can get around that issue, I would probably stick to uUnit, because obviously, when we get into tests ranging in the hundreds and some are time based yielding tests, that’s going to be a big issue. There’s a manager that uses “System.Reflection.Assembly[ ]” in order to find all unit tests.
FUnit is a little bit more bare, although it’s more fully featured as a unit testing and assertion library compared to Integraion Testing. You can select which tests to run, so that at least gets around uUnits problem. Another thing I didn’t like as much is that you have to separate out unit tests and yielding tests into different classes. You can get around that by simply yield breaking at the end of a yielding tests, so it’s treated like a unit test, but it’s still lacking in features compared to uUnit and even more so NUnit.
Verdict
I think I’m going to use all test tools from here on out; for non MonoBehaviours, I’ll stick to NUnit and the editor test runner. For MonoBehaviour unit tests, I’ll use FUnit, since I can pick which tests I want to run. For true integration level tests (multiple components interacting with each other), I’ll use Unity’s integration tests.
My take on MonoBehaviours and Unit Testing
I am of the opinion that while you can have your game logic not implement MonoBehaviours, in many instances, it just doesn’t make sense. Like how do you test an FSM that uses coroutines in Enter/Exit states? Or a Motor with a MoveTo method? Sure, I can jump through hoops to separate out the Unity part, but why? The idea behind unit testing is to save time and become more productive in the long run, plus ensure a higher quality product. Why not test a real world MonoBehaviour in it’s natural environment and not heavily modify either an existing code base or change the way you program to include more classes and more overhead to implement code? The former is the big one; how do you deal with a legacy codebases (defined as non-unit tested code)? It’s one thing to start a brand new project in TDD fashion and do all the Unity separation from the start, another is to refactor legacy code.
Potential Playmode tests
So the Playmode tests sound promising but I couldn’t find any information on it. Just found this thread;
I’d appreciate any other info you could provide. Running unit tests in playmode sounds great, but I really hope that it’s easy to do the equivalent “yielding” tests in the tools mentioned. The issue I see with Integration Test Tools in Unity is that the code based API just isn’t friendly or as full featured as NUnit, making it a poor choice for code based unit testing, even if it has the playmode option. So hopefully playmode tests will be that sweet in-between pure unit tests and multi-component integration tests. Maybe as Unity developers we simply have to redefine automated testing by adding a new category of testing to the testing pyramid.
I just noticed that in the Editor Test Runner, in 5.5.0 f3, there’s PlayMode and EditMode tests. Any idea what this is about? Is there any documentation on the PlayMode tests? I’d love to get started working on those, maybe I don’t need FUnit anymore.
This is not functional yet. it’s been since early 5.5 betas i believe. I think it’s some sort of mixture between editor (“unit”) tests and integration tests (that run in a scene, using game objects, etc).
Not quite ready yet, but definitely sounds interesting.
