Unity Test Framework 2.0 ready for feedback

Unity Engine
, ,
1

Unity Test Framework v2.0
[updated on May 2nd]

Hi everyone, we have a new experimental version of Unity Test Framework for you to try out in hope of getting feedback regarding the new features. At this point we have no timeline for fully releasing this version, but once we do, we will make sure to keep you updated.

For those of you who are unfamiliar with what Experimental packages are, they are defined as:
Experimental packages are new packages or experimental modifications made to mature packages. Unity does not support Experimental packages because they are in the early stages of development.

You can read more about them in this article.

How to activate v2.0
If you’re using Unity 2019.4 and beyond, open your package.json and update the com.unity.test-framework line to “com.unity.test-framework”: “2.0.1-exp.1”

Combine Edit Mode and Play Mode tests
This version removes the previous requirement to keep Edit Mode and Play Mode tests in separate assemblies by introducing the RequiresPlayModeAttribute. An Editor-only test assembly can now include tests that will run in the Editor’s Play Mode if given the [RequiresPlayMode] attribute. Likewise, a platform-specific assembly can include Edit Mode tests and exempt them from running in Play Mode if the tests are given the [RequiresPlayMode(False)] attribute.

Async tests
This version introduces support for writing asynchronous tests with the dotnet Task asynchronous programming model. See Async tests. Feedback on this feature is especially welcome.

Ignore tests based on arguments
This version introduces the ParameterizedIgnoreAttribute which allows ignoring tests based on arguments which were passed to the test method of a parameterized test.

Revised Test Runner UI
This version includes a revised Test Runner window with support for combined Edit Mode and Play Mode tests and several usability improvements:

2307Ă—657 57.4 KB

This version includes many additional bug fixes and improvements. For a full list of changes and updates in this version, see the Unity Test Framework package changelog.

Looking forward to hearing from you

  • UTF team
12 Likes
2

This sounds really good! I’ll be sure to give it a try soon.

3

Good job!
Do you have a good solution for simulating the advance of time for Tasks in tests?
Like TestScheduler in UniRx for example

4

Will this version be supported in earlier Unity versions as well ?

5

I’m very interested in this as well

6

Yes, it is available all the way back to 2019.4. Depending on the Unity version, you will have to add it to the package.json manually, by changing the test-framework line to "com.unity.test-framework": "2.0.1-pre.18"

1 Like
7

There’s a few things that are annoying in V1, and I’m wondering if they’ll be solved in V2:

  • The inability to write tests targeting Assembly.CSharp, forcing an assembly-based workflow. Our experience is that assembly-based workflows lead to worse code and slower compile times, so we’d like to avoid it.

  • No support for [Timeout]. Right now every test that yields anything else than WaitForSeconds has to include a manual timeout, otherwise tests can hang forever.

  • The way runtime tests work. Right now Unity makes a temporary scene asset and loads it when we start tests. After that, we typically load a different scene. This causes a bunch of knock-on issues:
    – If Unity crashes during execution, we get a new scene asset in Assets/
    – Any system that scans for asset changes (like Quick Search and version control plugins) starts doing a bunch of work
    – We’re often in the scene that the Unit test is supposed to run in (especially when iterating on fixing a test). Unloading that scene, loading a new scene, and then loading the original scene back is slow, and slows down iteration.

  • Ideally all of this would be fixed by two things:
    – the ability to tell Unity which scene(s) to run a test in the context of, and for Unity to not do any scene loading if those are already open
    – when we don’t run the tests in the context of a scene, Unity should use an in-memory normal empty scene instead of an asset. Unity is perfectly capable of entering play mode with a clean scene that’s not an asset.

10 Likes
8

Timeout issues will be solved in later iterations of v2.0, before the full release of 2022.2.

These two items you mentioned are not addressed in v2.0 at all. While for the latter we discussed a potential solution of enabling users to specify which scene should be loaded for a test, I’d be curious to learn more about your experience regarding assembly-based workflows. How does it lead to worse code and slower compile times? What would be a better way of working for you and your team?

9

I’ve had that discussion before here , and most of that still holds. Note that we didn’t have enter play mode options for that project, so more assemblies meant taking longer to enter play mode even when we didn’t change code!

For our current project, we ended up having a single, big asmdef that just covers all of our Assets/ code in order to support tests, and another one for Editor/ stuff that we asmref all over the place. But I’d really love for that to just be Assembly-CSharp and then be able to target that with our test assemblies.

The clean solution that’d fix both the testing problem, and all of the other problems we have around assembly definitions, would be to just allow Assembly-CSharp to be an assembly we could add a reference to from other assemblies. That shouldn’t be very hard!

We do like to extract stable, independent parts of our code base out of the general compile/reload flow, but in those cases moving them into a package is a lot cleaner, since that allows versioning and reuse between projects (we use Verdaccio). If something’s changed often enough that it’s in Assets/, it’s changed often enough that we’re not saving compile/reload times by segmenting it into it’s own assembly. We’re just adding to the average case number of assemblies that has to be reloaded by a code change.

1 Like
10

Oh, and if you have a bunch of plans and ideas, @thewerku , maybe there should be a testing tab on Engineering tools roadmap | Unity ?

1 Like
11

Since you’re interested @thewerku I agree with @Baste here. I work for a big mobile company and we use modules for every part of the game, and each module has its own Asmdef. (Shop, Dashboard, Each game mode, Social, Rewards, etc).

Since we started using Asmdef for each module, everytime you change any single line of the code, going back to unity takes around 20 seconds which makes the workflow very tedious and hard. It gets 10 times worse when you’re developing using TDD. Everytime you change something on the test you have to wait 20 seconds, this makes a simple 2 minute discovery test, something that can take up to 15 minutes.

5 Likes
12

I gave this a spin, and here are my findings:

  • The new UI is nice, especially when filtering tests!
  • However, the UI does not show pass/fail status while executing, which I find somewhat annoying, as it’s nice to start looking at failing tests already while the rest of the tests are executing.
  • The first bigger issue I found is that the new Custom NUnit version has Auto Reference enabled. This makes all assemblies look like test assemblies. This is a problem for some of our custom code, SonarCloud analysis, and also the test runner code itself seems to use this criterion to scan assemblies for tests (see bullet below).
  • When running all tests in our largest project with lots of assemblies and over 7000 tests, starting up takes extremely long: it took over 2 minutes to get past “Create Bootstrap Scene”, and at least another 2 minutes to get past “Prebuild Setup”. I’m guessing this is at least partially because all assemblies are treated as test assemblies now (see bullet above). These stages can’t even be canceled, making the situation worse.
4 Likes
13

There’s a workaround for this called StoppableCoroutines. Trilleon Unit Test & UI Automation framework implements it, and coroutines will never hang/fail. Try implementing it yourself. Here is the specific file. Look at the AutomationMaster.cs file to see how it is used.

Although it may be difficult to implement in an NUnit or UnityTest framework if you can’t control the test runner. If you need this functionality, it may make sense just to switch your tests to Trilleon tests, or make your own test runner.

1 Like
14

Error occurred when run all Edit Mode and Play Mode tests on Test Runner window.

ArgumentException: An item with the same key has already been added. Key: RequiresPlayModePerformance.Tests.Editor.EditModeTests.DummyEditModeTest_AlwaysPassed(1)
UnityEditor.TestTools.TestRunner.GUI.TestListGUI.get_ResultsByKey () (at Library/PackageCache/com.unity.test-framework@2.0.1-pre.18/UnityEditor.TestRunner/GUI/Views/TestListGUIBase.cs:56)
(snip)

Reproduction Project

This error will no longer be reproduced after the following actions:
Causing Assembly-CSharp-Editor assembly recompilation by editing Assets/Editor/RunTestsPerformance.cs.

15

Bugs:

  • In the Test Runner window, the IgnoreAttribute and DescriptionAttribute string arguments of tests are not displayed. In the case of ExplicitAttribute, they are displayed reason after “Run All”

Requests:

  • I hope to be added string argument (reason) to the ParameterizedIgnoreAttribute
  • In the Test Runner window, I hope multi-word search. And if possible, ideographic space (U+3000) should also be recognized as separators.
  • I hope SceneManager.LoadScene should be able to load scenes that are not in “Scenes in Build” (there is a workaround, but it would be nice to have it implemented in the framework)
2 Likes
16

I’d say “SceneManager.LoadScene` should be able to load scenes that are not in “Scenes in Build” (there is a workaround, but it would be nice to have it implemented in the framework” is quite important

2 Likes
17

The issues with Coroutines and the Unity Test Framework run deeper than just timeouts, I commented on it in this thread . It’s totally possible to write properly timing out and cancelable code within the framework, it just requires you to wrap everything asynchronous in something else than a raw coroutine/enumerator.

1 Like
18

Hey all, thanks for the replies! I’ll try to address the points we haven’t yet responded to:

@unity_t3DjhpF93gNHLg the only thing that has come up in our discussions was potentially using Time.timeScale - have you looked into it to see if it meets your expectations?

Our plans for this year are very much dependent on v2.0 feedback, so we don’t really have a fully fledged roadmap to put on the main page. What we’re going to do is collect improvement ideas and then present them to you (again through Forum) to help us prioritise the work - once priorities are clear, we’ll factor in the time we need for bugs and maintenance work to plan it against the release deadline. Either way, we might look into actually having a public roadmap once the v2.0 plans are validated with the help of this community.

@sbergen do you mean that the status icons don’t show up at all while tests are running? Would you be able to share a screenshot of how it looks like on your end? Feel free to ping me directly via Forum messaging option if you don’t want to share such content publicly.

Thanks for reporting @nowsprinting . We logged these bugs on our backlog.

We will take these suggestions into planning. As mentioned above, we’ll collect all requests and get back to you to understand the priority and value, that will then help us decide on the work plan for the next quarter.

The last thing I wanted to address is feedback around working within assembly-based workflow. We’re taking in what you’ve shared here and in other threads, and will start an internal discussion about this with the Scripting team, to better understand current approach and what is the best way to alleviate pain for you all. If there will be any work planned, it will be separate from UTF 2.0 release.

To be continued. Appreciate the discussion here so far, keep it coming!

5 Likes
19

In these years using the test framework, this would be my main requests:

  • Coroutine version of OneTimeSetUp

  • Discussed here: Add coroutine version of OneTimeSetUp

  • This is important for loading a test scene just one time

  • Support for loading scenes that are not included in the build index

  • Important for loading a test scene that you only need it at testing

  • It would be also nice to have serialized fields in the tests. Sometimes you need to reference some project asset in the test and it would be nice to get it by a serialized field.

Thank you

2 Likes
20

This can be done through EditorSceneManager.LoadSceneInPlayMode. We write tests like this:

public abstract class TestsForScene {
    public abstract string pathToScene { get; }
    protected string sceneName => Path.GetFileNameWithoutExtension(pathToScene);

[OneTimeSetUp]
public void OneTimeSetup() {
#if UNITY_EDITOR
        UnityEditor.SceneManagement.EditorSceneManager.LoadSceneInPlayMode(pathToScene, new LoadSceneParameters(LoadSceneMode.Single));
#else
        SceneManager.LoadScene(sceneName, LoadSceneMode.Single);
#endif
    }

But having a real way to do this would be much nicer. What if we could:

[UnityTest(scene="path/to/scene")]
5 Likes