How to write unit tests that can be visually inspected?

Unity Engine
,
1

I’m trying to write unit tests for my camera scripts and they are working as expected mostly. However, I realized that sometimes it’s a bit hard to debug a failure since they, at least in their current form, don’t provide any visual feedback.

To be precise, it provides some visual feedbacks in the scene window, but I cannot examine them because it clears the whole window as soon as the test finishes.

When I run tests, the scene section in my editor flashes constantly, showing glimpse of the camera view which is controlled by my tests. But as soon as it finishes, it displays the summarization of test result briefly and switches back to the normal editing mode immediately, so I can’t really read the result or inspect the scene at the time when something goes wrong:

3021604--225730--test.png
3021604--225730--test.png726×659 90.8 KB

The screenshot was captured the moment before the editor switches back to its normal editing mode, thus clearning everything in the scene window.

Of course, I can see what tests failed from the left tree view, but I also want to examine the content of the scene visually, and I suspect it could be the way it should be used, considering it shows some information in the scene window.

So, how can I examine what went wrong during a test visually? Is there any way that I can pause the test execution when there’s a failure then inspect the content of the scene, or read what is displayed in the scene window?

Here’s the test code I used, which shows such a problem :

Thanks in advance!

2

Anyone please? :sweat_smile:

3

Instead of (or in addition to) displaying the results in the game view (presumable with OnGUI or similar), you should just Debug.LogError. This will pause the game (assuming you have standard editor settings), which makes it much easier to see what’s going on, and also make any failures hard to ignore (as they should be!).

4

I tried your suggestion but it doesn’t seem to pause the test execution. I checked my editor settings also but could not find relevant options.

Could you instruct me how I could make my editor work as you suggested?

5

In the Console window, make sure the Error Pause button is always down.

850×357 107 KB

It’s up in this screenshot (taken from the Unity manual). It should look just like the Clear on Play button.

Also note that it doesn’t pause your script right where the error happens, but rather at the end of the frame. Still, that’s usually good enough for you to inspect the state of things and see what’s going on.

6

@JoeStrout Ah… now I get it, thanks!

However, the Error Pause button seems to revert to unselected state when I run any tests. It works as intended when I hit the play button though. Any ideas?

7

No idea. I run my tests by hitting the Play button. But I guess you’re using some sort of unit-testing framework? It must be turning that off. You should contact the author of that framework (or, stop using it, and just call your unit tests from the Awake method of some object in your scene).

8

I’m just using what Unity provides for unit testing, and the screenshot shows the default test hierarchy view from the editor, so I assumed I was doing it correctly.

Maybe I’m missing something, but I have no idea as there’s so little information on this subject :frowning:

9

I honestly didn’t know Unity provided any support for unit testing! :eyes:

What I do is just write a static method to run my unit tests in each of my modules, and then call that from some handy Awake event. The tests invoke Debug.LogError if any test fails.

EDIT: I’ve never seen the value of unit testing frameworks, in general, because they seem to be all about providing you the ability to not run the tests at all, or to skip specific tests — two things that you should never do. Every test runs every time, or their value is greatly reduced (IMHO).

10

But in general, I think IDE integration is quite cool (in this case, Unity integration… even though I can’t figure out how to pause the execution), and fixture related functionalities could be quite handy, like this :

        [Test, Description("Changing Heading/Elevation/Distance should move the camera to a proper position.")]
        public void ShouldMoveCameraToProperPosition(
            [Values(5, 10)] float distance,
            [TestRange(-180, 180, 45)] float heading,
            [TestRange(-80, 80, 40)] float elevation)
{

And lastly, I prefer to make my assert statements in the most descriptive form as possible as many people seem to do. It maybe not so expressive as DSLs in other languages, but I suppose it’s better than nothing as NUnit makes me write such a code:

Expect(Camera.Distance, Is.EqualTo(Camera.DistanceSettings.Maximum).Within(Tolerance))

Anyway, I’m also trying to be pragmatic about unit testing, but I think it’s nice that Unity supports such a widely used framework like NUnit, even though it looks to be rather clunky at the moment :slight_smile:

11

I don’t know if there’s a builtin way to pause from the test runner, but you can pause the editor from code like this:

UnityEditor.EditorApplication.isPlaying = false;

So you could at least wrap the troublesome code in a try/catch:

public void TheTest() {
  try {
    //Do the thing
  }
  catch {
    UnityEditor.EditorApplication.isPlaying = false;
    throw;
  }
}

There might be a way to have a method run any time a test fails, but I haven’t really used the integration test runner that much so I’m not sure.

12

Thanks for the suggestion. But it doesn’t seem to work, as it just throws the following exception: