Changing the bundled NUnit version

Unity Engine
,
1

Currently, Unity automatically references a built-in NUnit assembly from the Unity installation folder.
I am not 100% sure which NUnit version is that, but is it going to be possible to drop whatever NUnit version i want so my tests can use that ?

2

We are currently moving the Test Runner functionality into a package. The built-in NUnit is going to be a separate package referenced by the Test Runner one. So, in theory, you should be able to swap that one with any other NUnit version, but in practice, things might not work as expected if you do that or might not works at all.

The reason is that the NUnit version we use contains some modifications compared to the original:

  • it removes System.Web.dll
  • changes some internal NUnit APIs to public, so that we could make our own test runner to use instead of their default one (required for dealing with platforms like WebGL, where multithreading isn’t supported)
  • NUnit did not respect equality overloading, so checking for null on UnityEngine.Object and other things like that were added
  • maybe other things I can’t remember

I think the version of NUnit we currently use is 2.6.4 and we want to update to at least NUnit 3

3

The latest version is compatible with .NET Standard 2. I don’t know if it still references System.Web in that assembly

https://www.nuget.org/packages/NUnit/

Also about your fork and modifications… Did you suggested a pull request at that time with that valid points so you could use the original assembly still? It will be better for everyone.

4

@ElvisAlistar We currently ship Unit 3.5-ish.

The biggest thing we changed in NUnit was restructuring the execution engine to make it possible to write tests as coroutines (i.e. support for the [UnityTest] attribute). I did look at whether there was a nice way to upstream this but couldn’t come up with anything that I thought would be appealing to Charlie and the team. Maybe once it’s all packaged up, it’ll be easier to figure something out.

5

@ElvisAlistar @superpig I see that in 2019.2 alpha Test Runner it’s already a separate package.

I have two questions:

  1. Have you considered updating NUnit to the latest version? Now that it is “separated from Unity” (a separate package) will it make it easier for you to update NUnit from time to time and be able to be close to the latest version?
  2. Will it ever be possible to test with async/await/Tasks? Please see: https://forum.unity.com/threads/async-await-in-unit-tests.671608/
6

Hey.
Updating nunit is something that we would like to look into when we have the bandwidth for it. However it is currently unknown for us how feasible it will be to do so and how much will break when we upgrade it. We cannot promise anything yet, but it is on our radar.

7

I would also like to express my support for a more recent version of NUnit. Particularly because of the Order attribute for fixtures (instead of just methods) that was introduced in NUnit 3.8. This would be useful to ensure that e.g. the tests defined in SmokeTests are run first. Would also be nice to be able to toggle between different sort modes in the test runner (e.g. by Name, by Order)

4 Likes
8

I would like a newer version too. By now Unity’s NUnit version is ancient (I cant even find a version that old on NUnit GitHub which goes 5 years back). I’d also like a .Net 4.5 build of it so I can run async tests. Maybe you can leverage the async support to handle coroutines?
The docs for v2 is also way worse than the v3 docs.

2 Likes
9

+1

I also would want to have async tests.

1 Like
10

I’d be happy to just have Assert.Multiple.

3 Likes
11

+1

12

One of the reason why we desperately need an NUnit update on our side is to support async tests correctly. The embedded NUnit version does not accept test methods that returns Tasks, e.g. this does not run:

[Test]
public async Task SimpleAsyncTest()
{
    var value = false;

    await Task.Delay(2000).ContinueWith((t) => {
        value = true;
    });

    Assert.IsTrue(value);
}

And using void is not an option since Nunit will think the test is done at the first await:

[Test]
public async void SimpleAsyncTest()
{
    await Task.Delay(2000);
 
    UnityEngine.Debug.LogWarning("This executes after the test completes, making the test a false positive");
    Assert.IsTrue(false);
}

If Unity does not have the resources to maintain a custom Nunit version, is there a possibility to integrate another test runner ?

5 Likes
13

I’ve run into the same issue as remi-sormain-vrdirect mentioned. In Unity 2019.3.6f1 version and Test Framework package 1.1.13 version from March 17, 2020.
The tests with Task return type fails by default.
So I wrote task.Wait() in tests for async methods. It seems to work.

14

I would also prefer async unit tests now that Unity has better support for async features in general.

1 Like
15

A year has passed since this reply. It seems like bandwidth should be made for this, it’s quite painful to have such an old version of NUnit.

16

And another 3 months. Poking here in the hopes of getting a Unity dev’s attention on this.

1 Like
17

It’s still on our to-do list, but no ETA yet.

2 Likes
18

I assume you proposed a task var with Task.Run(Action to await for), then .Wait() on this task var and then use .Result().
That’s not working to me 'cause it says my action to await for is only capable to be executed from the main thread, so I’m on a deadlock thus I can’t test async code in my project. :frowning:
The transient solution I came with is just return void so all tests pass but they log the exceptions. My decision relies on a proper solution updating NUnit bundled so Task can be returned where on a [Test] method.

19

I will too ask for this to be prioritised, lack of proper async/await support is holding us back from adopting it. Plus, with newer APIs like Addressables supporting the pattern and libraries like UniTask getting better with every update, I believe it would be a huge upgrade for many workflows.

2 Likes
20

+1
I really want this to be prioritised, [UnityTest] and other workarounds bloat tests too much and lead to garbage code