Test runner API questions / feedback

Unity Engine
,
1

I am going over the documentation for Unity Test Runner and some things aren’t totally clear to me. here’s my feedback:

  1. Play mode vs. edit mode tests
    What is the technical distinction between the two types? what is allowed to be done in edit mode which is not allowed in play mode tests ? (and vice versa). i also do not understand the motivation for selecting either type (when should i use each of these?)

In Unity test tools (which is now deprecated), unit tests could not perform certain things (e.g: create new game objects). They were also executed as editor code (e.g: not in “play mode”) so they could not start coroutines and such things.

With these 2 new test types, it’s not clear anymore (to me).

  1. Editor restart after enabling play mode tests
    After enabling play mode tests, the editor must be restarted. kind of a nitpick, but that’s pretty annoying, and i don’t recall any other feature that behaves the same. wasn’t there any other solution for this? (why is it necessary anyway ?). Also, it is mentioned that this adds some overhead to the game at runtime - can someone clarify what does this add and whether i should actually worry about enabling this? (if it adds 10kb to the build i am totally fine with it).

  2. PrebuildSetupAttribute
    (what an awful name BTW) - why should i prefer this over NUnit’s SetUp attribute? what is the difference between the two and which one should i prefer to use ?

  3. MonoBehaviourTest
    while i believe the intention was good (probably due to lots of feedback about how it’s hard to test MonoBehaviour code), i don’t really see how this thing could be useful for real world scenarios.Also, it “asks” you to mix your runtime (MonoBehaviour) code with your tests (IMonoBehaviourTest interface).

  4. Command line execution
    does this work in conjunction with -batchmode or -nographics? we would like to run editor / play mode tests from our build server. so far i haven’t been able to execute it via command line.Also, it can be nice to have more control over which tests will get executed (by name, category, etc).

  5. Improve the documentation
    documentation on the docs page (linked above) is not clear at times. also links coming from this page to various attributes or other code related stuff is very under-documented and missing examples or includes confusing descriptions.

  6. Please Open source the test runner
    I think we (the dev community) can benefit from it as well as improve on it and send it back as PRs.

2

It’s pretty funny that nobody replied to this important thread. I’m currently looking into TDD on Unity but there’s so little to read.

3

That’s highly subjective.
TDD isn’t first class citizen in game development for better or for worse, that’s matter of taste. Personally I don’t like TDD.

4

Do you dislike TDD at all or only in Gamedev? Do you want to argue about it? I’m quite interested to know other people opinions and experience.

disclosure: I use TDD or a edulcorated version of it in all my development projects.

5

I dislike TDD everywhere.

I’m an enterprise software engineer, so in my day-time job, I write tests, after I wrote the actual code to ensure I don’t mistakenly change something. But I don’t write BS tests in advance.

I value the functional tests much more, since I usually work on applications, where the only thing matters is what the users experience. If the application is working properly, we’re good at the moment.
This applies to the game development as well. I value QA manual tests more than artificial unit tests. (Although obviously for an API you don’t really have a choice).

6

Hi @liortal ,
I’m nowhere near a test expert, but I can give you some info based on my personal experience writing tests for Unity code.

I use edit mode tests to test Unity APIs that are part of UnityEditor, or code that sits in Editor folders. Since these tests use Editor code, they will not compile as playmode tests if you try to run them anywhere but in the Editor.

Playmode tests run in playmode / standalone, so if you use them in combination with the [UnityTest] attribute, you can test the interaction between your code and the rest of the Player loop in one coroutine, which might be possible with Editor tests, but will be more complicated. So this makes them a bit more suited to Integration tests.

There are a lot of simpler tests that you can write as either type, but you need to keep in mind that Editor tests will not run outside of the Editor, so you will miss platform coverage by writing your tests as Editor tests. That might not be an issue if you write strict unit tests, but as soon as you start interacting with Unity APIs, having multi-platform coverage is always good.

Finally, Edit mode tests don’t have the game loop running, so there are less variables involved. If what you are testing doesn’t depend on the game loop running, or MonoBehaviours executing, then you can definitely implement them as Editor Tests.

7

Just dug up some more info:

This is a setup that’s invoked in the Editor, before the playmode test builds a standalone player to execute your tests on-platform. This is useful if you need to test things that require generating assets/scenes in the editor.

8

Thanks @DavidGeoffroy , some time has passed since i originally wrote this post.
I do realize now what’s the different between edit and playmode tests. it seems that playmode tests are more suitable to what i’m after (as i am testing our game’s components rather than some editor code that we created).

Having worked with the test framework a bit more, there are more things that i find missing, off the top of my head:

1. No editor API for running tests
The only ways i know are either from the editor UI, or by launching the editor using the relevant command line args.
We are using an automated build system so we do not manually interact with the UI. The only option we have is running Unity with the -runtests command line arg.

Our build consists of running tests first and then building the player. This means we have to launch the editor twice (once for tests, once for building the player). This slows down our build process. i wish there was an editor API (e.g: TestRunner.ExecuteTests) that will allow me to run all tests.

2. UI changes to test runner window
There are a lot of visual improvements that can be done to this window, for example:

  • Show the number of tests (executes, passed, failed, etc)
  • Allow grouping by categories (built-in support in NUnit using the [Category] attribute
  • Better marking for ignored tests (also, it seems that ignored tests take so much time to skip over, even if they are not run)

3. Playmode tests don’t work with -nographics
Not sure this is 100% correct, but our CI system runs these tests and it cannot execute playmode tests on Windows based machines, since when running them it requires access to the “desktop” since Unity cannot be run with -nographics (our build agents run as a service that doesn’t have that access). Why can’t i run Unity without any graphics, but only logically simulate running those tests?

4. Allow building a Playmode test build
Playmode tests can be executed on a target device, but this only works if you build and run it through the test runner UI. it could be more flexible to be able to run a complete build that will be generated to run playmode tests, and then install that build wherever i want (any device, device farm, etc).

5. All tests are run in the same “isolated” scene
When starting test execution, a new scene is created. this is often times not enough, as we’d like a “clean slate” for every test fixture or even every test that is run. It could be great to have that baked right into the framework (currently i have to implement this myself).

I have a lot of other feedback points, i really wish there were more updates about the test runner UI and framework (open sourcing it, updates, etc).

1 Like
9

Maybe @ElvisAlistar can comment on any upcoming updates to the test runner?

10

I would refer you to Christian’s reply here: Feedback for LogAssert class - Unity Engine - Unity Discussions

Looking to hire one more person internally to look after this feature. Also, working hard to getting it released as a package and very close to that.

At a minimum, you will have access to the source code and do anything you like with it. We plan to address most feedback and concerns in the future, but it’s going much slower than we would like to.

We’re also going to set up a separate forum section dedicated to testing and we’re going to be more closely monitoring that.

1 Like
11

Great news. Thanks for replying.
I will definitely be waiting for all of that.

12

Is there any progress? We need to have C# API to run tests from the editor. At least could you provide an example of how to use reflection to call Run All tests? This is very useful to do before any commit to the repository. Thanks.

13

Yes. The test runner api has been released a few weeks ago. More details here: https://discussions.unity.com/t/753560

14

@Warnecke there’s a lot of obvious bugs in the test-framework docs (from typos, to broken/incorrect/impossible code examples, to missing/incomprehensible explanations/sentences, to incorrectly-tagged HTML headings/links).

Is there a way to report / correct these? Since it’s outside the main docs, the normal links for “there’s incorrect information here” aren’t present. And there’s no contact information included.

15

I just post my feedback / comments here on the forums.

16

Yes. Leaving your feedback for the documentation here on the forum will work fine.

17

OK. @Warnecke : Current “causes problems multiple times for at least one team member on a day to day basis” issues in the framework / docs:

  1. The testRunner window keeps resizing everytime you reload assemblies. Specifically: the detail area (bottom of the window) keeps shrinking to a size that is useless: it stops at precisely the height BEFORE you can see the line number of any assertion-fail. If it has to be hardcoded to something (it shouldn’t!) then at least make it a useful position :).

  2. Assert.AreEqual : has no override for Quaternion, Vector2/3/4 … these are very common things to need to compare equality on, I’m not sure why you’d leave them out? We wrote our own, and it seems everyone else has to also.

  3. Assert.AreNotEqual : almost useless. Since Unity Engine’s architecture uses float everywhere for almost everything, this method needs to have a version that matches Assert.AreEqual’s variant with the extra “delta” argument. Again: wrote our own to fix the gap.

  4. TestRunner has no “right click a test name to select the class in project view” (which exists for all materials, shaders, scripts, etc in Unity, in multiple places throughout the editor). When you have a few toy tests (5, 10) then it’s fine, no-one cares; but if you’re actually using the Test Framework properly, on a real project, with thousands of tests, then this is a HUGE missing feature.

  5. Missing docs: MonoBehaviourTest … ??? … This thing has 3 sentences of doc that I can’t understand, and no-one else seems to either, and a single code example that equally makes no sense - what is this for? Why would you do that? On forums, most people seem to make the same assumption: it’s “something” related to mocking MB’s, but … how? (and is that even true? I don’t think so - I think the name is misleading?)

  6. The “Overview” is missing critical background information, for instance: what happens with Scenes? (Trial and error: TestRunner auto-creates a scene, which is shared for all tests within a run (or is it all tests within a namespace? Who knows!). But it also allows you to override that and create your own scene (if needed). But the scene it creates isn’t a proper Unity Scene (it has no name! Which breaks some Unity APIs that need a name or a path) and confuses the Unity SceneManager/EditorSceneManager classes).

(there are some detailed forums posts here from other people who’ve trial-and-error tried to understand the missing docs around scene-creation with UTF. Personally, once I realised how confusing it is, I just rewrote our game code to fit UTF’s useage of scenes - it’s a bit ugly, but depending on undocumented behaviour in Unity UTF was a bigger risk)

  1. Frontpage of project declares that Parameterized Tests aren’t supported except for ValueSource. This is false (and it would be a disaster if it were true! Trying to use ValueSource without Sequential is a nightmare. But Sequential actually works fine, which is a relief.)

  2. The official manual page for tests and testing has been deleted, and there’s no link / reference to the new UTF. Team members upgrading from Unity 2019.1 or 2018.4 were lost. Google indexes the 2019.1 and 2018.3 pages as the top hits - and because Unity didn’t link the UTF docs into the main manual, that’s all Google values, unless you know the magic words to search for. As they say: “A URL is for life, not just for Christmas” :wink: (i.e.: the testing page shouldn’t be deleted from the manual, it should be updated to point to the UTF sure, but not just wiped out!)

… there’s also typos, grammatical errors, and missing docs for key classes / attributes, but they are minor compared to the above.

18

But - to be clear - overall it’s very useable and we’re getting genuine benefit out of it. Things that we’ve been unable to test with Unity in the past (because of missing features in unity’s unit-testing) and had been manually testing (ugh!) and/or using custom editor scripts for … I’ve mostly been able to port across and they run fine in Unity TestRunner.

Also: I’ve been pleasantly suprised by how fast the testrunner runs. There’s an annoying delay on startup for every test-run (3-5 seconds), but where I was afraid it would increase with the number of tests and test classes it seems fairly constant so far, and I’ve been throwing larger and larger numbers of tests at it.

I’ve only had one crash - I used 6 ValueSource variables, each with 6 values, and forgot to mark Sequential. The UnityEditor hung. It was attempting to render 46,000 tests in the TestRunner, I guess? I consider that a bug - it meant I had to kill the process and lose unsaved scene edits (fortunately very little in this situation - but in future that could bite someone) - so it would be better if the pre-process step that generates the set of tests especially for combinatorial ValueSource setups had some kind of early-out backstop that detected an insane number of tests and output a warning / required a manual override in the testrunner (an “Are you SURE you want to run 10,000 combinatorial tests?” button or something like that).

(also … 46,000 tests isn’t actually that much. I am disappointed that the Editor can’t cope. But I know that the UnityEditor’s GUI has some very bad bugs when it comes to rendering large numbers of items - it’s taken many years for the Unity Profiler to stop hanging on simple profiles, and it’s still slow for medium sized game projects, so I appreciate this is probably an issue with the Editor rather than with TestRunner)

19

First, a correction: the namespace collision between Unity’s proprietary class “Assert” and the same-named methods with NUnit’s class “Assert” + methods caused me to be wrong in one of the complaints above.

(NB: that is IMHO another annoying bug. Why has Unity decided to create a name collision, while both libraries are packaged together in Unity by default, and both are live at once? Since Unity’s version is NOT identical (but has some idenitcally named methods!) the whole thing becomes a huge mess)

In reality, Unity does have an AssertNotEqual method that works on float/double, but they decided to break the naming convention of NUnit (again … why?!! This just makes life harder for all of us :() and so I didn’t know it existed.

Now that I’ve pushed more tests over to using Unity’s Assert class, I’m finding it’s significantly worse than NUnit’s :(. For instance … The implementation of Assert.AreEqual … is not good. The message it outputs is:

UnityEngine.Assertions.AssertionException : Assertion failure. Values are not equal.
Expected: 0 == 10

What does that mean?

(hint: it doesn’t mean what it says ;). It definitely DID NOT expect 0)

Digging in the decompiled code, the problem seems to be “UnityEngine.Assertions.AssertionMessageUtil”, which has this method:

public static string GetEqualityMessage(object actual, object expected, bool expectEqual)
{
  return AssertionMessageUtil.GetMessage(UnityString.Format("Values are {0}equal.", !expectEqual ? (object) "" : (object) "not "), UnityString.Format("{0} {2} {1}", actual, expected, !expectEqual ? (object) "!=" : (object) "=="));
}

…the problem is that that method has no text saying which value is which, and it’s called from a context that prefixes it with the text (from same class):

private const string k_Expected = "Expected:";

So the net effect is that Unity outputs:

Expected: [I](not the expected value)[/I] == [I](the expected value)[/I]

In contrast, the NUnit implementations of Assert.AreEqual output simple, clear, messages that are wholly correct and don’t confuse the reader. IT would be great if Unity dropped their messages and used the NUnit ones.

20

Another issue (bug?):

  • Create ValueSource array that’s pre-filled with ScriptableObject instances
  • The first test you run will work
  • All other tests now hang the TestRunner, you cannot use TestRunner again with these tests until you reload assemblies

(the test never ends - neither succeeds nor fails - and Unity returns to the original scene as though the test had run, but it hasn’t been executed (verified by outputting Debug.Log during the test))