Unity Test Tools

Unity Engine
,
21

Hello!

I’m trying to make some unit tests.

I created a C# file in MonoDevelop:

using System;
using System.Threading;
using NUnit.Framework;

class TextProcessorTests {
}

But I run into an error:

The type or namespace ‘NUnit’ could not be found. Are you missing a using directive or an assembly reference?

There are no other errors. The sample tests are running properly, with their ‘using NUnit.Framework’ statement.
What am I missing?

UPDATE:

I noticed that the nunit-DLL:s are only available in Assembly-CSharp-Editor, not in Assembly-CSharp where my tests are located. Is this the problem? If so, how do I fix it?

22

Place your file with tests under ‘editor’ folder. Read more about special folders in Unity here: http://docs.unity3d.com/Documentation/Manual/ScriptCompileOrderFolders.html

Tomek

23

Thanks, it is working! I put it in Assets/Editor/Tests/ … Can the Editor folder be placed anywhere?
It doesn’t really make sense :slight_smile: Lets hope TDD becomes part of Unity in the future, and not just an asset.

Also, the PDF contains no instructions on how to run the tests in the terminal on OSX.

24

I’m hitting an issue with running the tests from the command line. If I put the -quit in the command line arguments, the test results .xml file never gets output. This happens for unit tests and integration tests. Is this by design? I am trying to make this part of my build process and I need Unity to shut down between calls since you can’t have two instances open at the same time.

These both work if I take out the -quit command, but with the -quit I see the window pop open for a second and no output appears.

Here is what I am trying to run for unit tests:

“C:\Program Files (x86)\Unity\Editor\Unity.exe” -projectPath [ProjectPath] -testscene=IntegrationTests -batchmode -quit -executeMethod UnityTest.UnitTestView.RunAllTestsBatch

And for integration tests:
“C:\Program Files (x86)\Unity\Editor\Unity.exe” -projectPath [ProjectPath] -testscene=IntegrationTests -quit -executeMethod UnityTest.IntegrationTestsRunnerWindow.RunAllTestsMenu

I also tried encapsulating these methods in a static Editor methods that call EditorApplication.Exit(0) after calling the run tests methods, that also did not work. Has anyone else tried this?

25

Yes, you can place it anywhere.

Look here how to start unity from the terminal on MacOS. Just pass the same arguments.

26

The reason why the Editor quits and no results are produced is because the tests never get to run. It happens because the runner’s start is asynchronous and happens in a certain frame. The -quit parameter will make the editor quit once the static method is executed and that happens almost instantly since the method will not wait for the runner to finish. However, the runner is designed to quit the application, the play mode (in non batch runs) or even the editor (in batch mode runs) by itself after it’s finished.

What you can do is to implement an asynchronous mechanism for detecting if the run has ended (the results are generated or the application is not running anymore) in your continuous integration framework.

Tomek

27

There is an error in the v1.1.1 batchmode integration test example in the documentation. The argument in the doc is “-testscene” but the BatchTestRunner looks for “-testscenes”.

I have a Unity project with multiple scenes using Unity Test Tools v1.1.1. I’m trying to run my integration test scene with the following command…

Unity -batchmode -projectPath /path/to/my/unity/project -executeMethod BatchTestRunner.RunIntegrationTests -testscenes=IntegrationTestScene

Application.LoadLevel(1) in BatchTestRunner.Start() appears to load only the first scene configured in the build settings rather than my IntegrationTestScene. I can see in Editor.log that this is true.

The following line in BatchTestRunner appears to have no effect. Debugging shows that the array of scenes it creates is correct, but that Application.LoadLevel() just ignores the new scenes array.

EditorBuildSettings.scenes = sceneList.Select (s => new EditorBuildSettingsScene (s, true)).ToArray ();

Any thoughts on a fix or workaround?
Thanks.

28

My CI environment will know that Unity has exited and to continue on. I have the Unit Test runner working in my CI system with -batchmode, but integration tests are still giving me trouble.

When I run the integration tests with -batchmode, I hit a weird issue where I see the Unit Test Runner Window pop up, but Unity never shows up. The tests are not run. I can see an instance of Unity running in the Process Monitor, but there is no editor window, just the Test Runner window. I have to go into the Process Monitor and actually kill the process manually.

29

testscene parameter was renamed to testscenes after we added the possibility to run multiple scenes… aaand we forgot to update the docs. Sorry about that!

About your issue, which Editor version you are using?

30

Which OS and Unity version?
The window pops up because the run is initiated ‘through’ the window, thus it’s being opened (and it’s visible because it’s how Unity behaves, although it’s in batch mode). The window popping up it’s probably something that can be avoided.
The issue with the Editor not closing is related to a failure with the test runner. If you provide log from the run I will be able to tell you more details. http://docs.unity3d.com/Documentation/Manual/LogFiles.html

Tomek

31

I’m using Unity for Mac. I’ve tried this on version 4.3.0f4 and 4.3.4f1. It behaves the same way in both versions. If I only want to run (in batch mode) my integration tests, I need to change my build settings so my IntegrationTestScene is first in the list and make it the only scene with its checkbox checked.

32

Hi,
I 've just updated the UnityTestTools package. ( i like the new GUI ;))
But now all the tests are waiting the unity editor to get the focus in order to play them.
It seems that you are making a pause ( i guess a time.scale =0) when the focus is lost…

How can i remove that ?

Thanks

PS : You should also put Icons.cs on Editor/ or WebPlayer won"'t build…

33

Importing Unity Test Tools makes building Windows Store Apps player fail with the following errors:

Will there by a version for Windows Store Apps with #defines and Metro dlls?

34

It default behaviour of Unity. If you build a Standalone player, it it will also pause by default on lost focus. Go to Player Settings (Edit > Project Settings > Player) and check Run In Background. That should do the trick.

PS. We go the icons to Editor folder now :slight_smile:

35

npruehs, we are working on it.

36

Hi! The Test Tools are a great upgrade. Thanks!

However, NUnit test inheritance does not work for me. When I’m trying to unit test interfaces, I create an abstract class testing the interface and a specialized class with just overridden instantiation code (override protected ISomething NewSomething() { return new Something(); }) for each class implementing the interface. As far as I can tell, it should be a pretty common practice, but it does not work for me in Unity.

When there is no other test in the specialized class, I get a pretty ugly exception every time I try to run the tests:
ArgumentOutOfRangeException: Argument is out of range.
System.Linq.Enumerable.ElementAt[UnitTestResult] (IEnumerable`1 source, Int32 index)
UnityTest.UnitTestView.FindTestResultByName (System.String name) (at Assets/UnityTestTools/UnitTesting/Editor/TestRunner/UnitTestView.cs:392)
UnityTest.UnitTestView.UpdateTestInfo (ITestResult result) (at Assets/UnityTestTools/UnitTesting/Editor/TestRunner/UnitTestView.cs:386)
UnityTest.UnitTestView+TestRunnerEventListener.TestFinished (ITestResult result) (at Assets/UnityTestTools/UnitTesting/Editor/TestRunner/UnitTestView.cs:547)
UnityTest.UnitTestRunner.TestRunnerCallbackList.TestFinished (ITestResult fullName) (at Assets/UnityTestTools/UnitTesting/Editor/TestRunner/TestRunnerCallbackList.cs:22)
UnityTest.NUnitTestEngine+TestRunnerEventListener.TestFinished (NUnit.Core.TestResult result) (at Assets/UnityTestTools/UnitTesting/Editor/TestRunner/NUnitTestEngine.cs:129)
NUnit.Core.TestMethod.Run (EventListener listener, ITestFilter filter)
NUnit.Core.TestSuite.RunAllTests (NUnit.Core.TestResult suiteResult, EventListener listener, ITestFilter filter)
NUnit.Core.TestSuite.RunSuite (EventListener listener, ITestFilter filter)
NUnit.Core.TestSuite.RunSuiteInContext (EventListener listener, ITestFilter filter)
NUnit.Core.TestSuite.Run (EventListener listener, ITestFilter filter)
NUnit.Core.TestFixture.Run (EventListener listener, ITestFilter filter)
NUnit.Core.TestSuite.RunAllTests (NUnit.Core.TestResult suiteResult, EventListener listener, ITestFilter filter)
NUnit.Core.TestSuite.RunSuite (EventListener listener, ITestFilter filter)
NUnit.Core.TestSuite.RunSuiteInContext (EventListener listener, ITestFilter filter)
NUnit.Core.TestSuite.Run (EventListener listener, ITestFilter filter)
NUnit.Core.TestSuite.RunAllTests (NUnit.Core.TestResult suiteResult, EventListener listener, ITestFilter filter)
NUnit.Core.TestSuite.RunSuite (EventListener listener, ITestFilter filter)
NUnit.Core.TestSuite.RunSuiteInContext (EventListener listener, ITestFilter filter)
NUnit.Core.TestSuite.Run (EventListener listener, ITestFilter filter)
NUnit.Core.TestSuite.RunAllTests (NUnit.Core.TestResult suiteResult, EventListener listener, ITestFilter filter)
NUnit.Core.TestSuite.RunSuite (EventListener listener, ITestFilter filter)
NUnit.Core.TestSuite.RunSuiteInContext (EventListener listener, ITestFilter filter)
NUnit.Core.TestSuite.Run (EventListener listener, ITestFilter filter)
NUnit.Core.TestSuite.RunAllTests (NUnit.Core.TestResult suiteResult, EventListener listener, ITestFilter filter)
NUnit.Core.TestSuite.RunSuite (EventListener listener, ITestFilter filter)
NUnit.Core.TestSuite.RunSuiteInContext (EventListener listener, ITestFilter filter)
NUnit.Core.TestSuite.Run (EventListener listener, ITestFilter filter)
NUnit.Core.TestSuite.RunAllTests (NUnit.Core.TestResult suiteResult, EventListener listener, ITestFilter filter)
NUnit.Core.TestSuite.RunSuite (EventListener listener, ITestFilter filter)
NUnit.Core.TestSuite.RunSuiteInContext (EventListener listener, ITestFilter filter)
NUnit.Core.TestSuite.Run (EventListener listener, ITestFilter filter)
UnityTest.NUnitTestEngine.ExecuteTestSuite (NUnit.Core.TestSuite suite, ITestRunnerCallback testRunnerEventListener, ITestFilter filter) (at Assets/UnityTestTools/UnitTesting/Editor/TestRunner/NUnitTestEngine.cs:93)
UnityTest.NUnitTestEngine.RunTests (System.String[ ] tests, ITestRunnerCallback testRunnerEventListener) (at Assets/UnityTestTools/UnitTesting/Editor/TestRunner/NUnitTestEngine.cs:38)
UnityTest.UnitTestView.StartTestRun () (at Assets/UnityTestTools/UnitTesting/Editor/TestRunner/UnitTestView.cs:462)
UnityEngine.Debug:LogException(Exception)
UnityTest.UnitTestView:StartTestRun() (at Assets/UnityTestTools/UnitTesting/Editor/TestRunner/UnitTestView.cs:468)
UnityTest.UnitTestView:Update() (at Assets/UnityTestTools/UnitTesting/Editor/TestRunner/UnitTestView.cs:400)
UnityEditor.EditorApplication:Internal_CallUpdateFunctions()

If I add some test to the specialized class, the exception goes away and the test runs OK. BUT still none of the inherited tests is executed.

Has anyone else experienced this? Or is the inheritance working for you?

Thanks for reply,
Adam

37

Hi Adam!

Thanks, we are happy you like the update!
Could you please post some of the code that causes the exception and the OS you are using?

Tomek

38

I’m on Windows 8 Pro, 64 bit, using Unity 4.3.3f1, Test Tools 1.1.1.

Well, when I have tried to reproduce the exception in a fresh project just with Test Tools installed, I didn’t get the exception… And now when I have reloaded the old project, it does no appear even if I remove the additional tests. Strange… (The exception appeared when there was no TestOtherStuff() in the code below, but disappeared when this was uncommented.)

However, I’m more concerned about the inheritance of the test methods. Should they work this way? (I have each class or interface in a separate file.)

public interface ISomething 
{
    int ReturnOne();
}

public abstract class TestISomething
{
    public abstract ISomething NewSomething();

    [Test]
    void TestReturnOne()
    {
        ISomething s = NewSomething();
        Assert.That(s.ReturnOne(), Is.EqualTo(1));
    }
}

public class Something : ISomething
{
    public int ReturnOne() { return 1; }
    ... other stuff
}

public class TestSomething : TestISomething
{
    public override ISomething NewSomething() { return new Something(); }

     [Test]
     public void TestOtherStuff() { ... }
}

From what I have read, the NUnit should execute both TestReturnOne() and TestOtherStuff() on TestSomething. But only TestOtherStuff() is actually executed. TestReturnOne() does not produce any output to console, unit test runner, or XML.

My main question is: Do I have something wrong? If yes, what’s the best way to test multiple classes with the same interface without code duplication?

Thanks!

39

It states that it requires 4.0.0, but I am getting a bunch of errors on 4.0.1 (to begin with there where two which I fixed, but after that there where about 26 new compile errors). An example of one of the compile errors are:

Assets/UnityTestTools/Assertions/Editor/AssertionComponentEditor.cs(38,59): error CS0103: The name `target’ does not exist in the current context

or

Assets/UnityTestTools/Assertions/Editor/AssertionComponentEditor.cs(48,33): error CS1502: The best overloaded method match for `UnityTest.AssertionComponentEditor.DrawOptionsForAfterPeriodOfTime(UnityTest.AssertionComponent)’ has some invalid arguments

40

I’m having the same issue. The only way to get the integration tests running in batch mode is by manually selecting only the integration scenes in the editor build settings. Tested on OS X and Windows (Unity 4.3.4f1 and Unity Test Tools 1.1.1). Is there any workaround?

No problem with unit tests though. Awesome tools, by the way. Thanks for sharing.