Running tests consecutively

Unity Engine
1

Hi,

I’ve got a load of tests that I need to run consecutively and not concurrently. When I select them individually and run them they run fine, and if I select a few of them (say a fixture or a namespare with a couple of fixtures below it) then they run fine as well. But if I select more they seem to run consecutively but some of my plugins don’t work (DOTween if it helps to know), and if I ask all of them to run then they all start to run at the same time and clash with each other causing the results to flicker back and forth across random tests.

Is it possible to run all the tests but force it to run all the tests concurrently and such that each test is completely isolated (as if I entered or exited playmode)? And is there a way to do this via the command line?

Cheers!

1 Like
2

What do you mean by consecutively? all tests are being executed one after the other, never concurrently.

3

Mine don’t seem to, they seem to run all at the same time and their results change flicker back and forth between each pass/fail.

(These are playmode tests btw).

Running them individually means that they pass fine though.

I’m loading and unloading a scene for some of the tests and for those I’m using a superclass which has a [UnitySetUp] and [UnityTearDown] to load in my scene and then load back the test scene respectively, which all the other fixtures derive from. That’s the only thing I think that could be causing the issue.

Are we also allowed to “yield break” at the end of tests if the tests should be completable within a single frame?

4

I am having the same issue when running tests consecutively. Did you find the problem?

5

I’m guessing you’re using the [UnityTest] attribute for your test methods? If so, your tests are run as coroutines, which might explain the behaviour. You can still use the [Test] attribute even in PlayMode tests, so if your test doesn’t need to use yield then use the [Test] attribute instead.

6

My tests do require yields unfortunately

7

Gonna bump this thread. My problem is that I have a legacy codebase, which likes to leave around a ton of DontDestroyOnLoad gameobjects, and probably some static state too. Running my playmode tests in isolation work fine, but once I try to chain two, the lingering objects and state starts causing issues. If I could make my tests each end with a proper play mode exit, things would be sufficiently cleaned up. Is there a straightforward way to do this? I realise I’m sacrificing the ability to run these in a build.

The alternative I could try is restructuring them as edit mode tests, and have each architect a call to play mode enter/exit. Does that sound feasible?

1 Like
8

(in the end I bit the bullet and made my levels play nicely when flowing from one to the other. That’s obviously the Proper Fix and worth the investment)

9

You might need to properly setup and teardown each test:

        [SetUp]
        public void Init()
        {
            // loads a new scene for each test
            SceneManager.LoadScene("TestScene", LoadSceneMode.Single);
        
        }

        [TearDown]
        public void TearDown()
        {
            // disposes training instance from the Academy singleton
            Academy.Instance.Dispose();
        }

I also ran into trouble when setting up tests due to the immediate calling of the delegate passed to the event: Academy.Instance.OnEnvironmentReset

10

BTW this fails and it should not if you run both tests at the same time. If you run them one by one they pass.

This is a real issue, none of NUnit and MSTests (nor any test framework ever) are working this way…

This should be documented and have an easy work around …

    [TestFixture]
    public class PlayModeCleanContextTest
    {
        [UnityTest]
        public IEnumerator TestA()
        {
            yield return new EnterPlayMode();
            var a = new GameObject("A");
            Object.DontDestroyOnLoad(a);
            yield return new ExitPlayMode();
        }
        [UnityTest]
        public IEnumerator TestB()
        {
            yield return new EnterPlayMode();
            var a = Object.FindObjectOfType<GameObject>();
            if(a) Assert.AreNotEqual("A", a.name);
        }
    }
1 Like
11

This is not really feasible with heavy integration tests though. Also InputSystem Setup/TearDown is flawed and errors stack up after each couple of tests.

12

You can do
yield return new Enter/ExitPlayMode(), but it’s not cleaning anything as you can see in the previous example

1 Like
13

I tried adding this in the SetUp of tests. It just breaks the InputSystem even more with an error flood

            for (var i = 0; i < SceneManager.sceneCount; ++i)
            {
                foreach (var rootGameObject in SceneManager.GetSceneAt(i).GetRootGameObjects())
                {
                    UnityEngine.Object.Destroy(rootGameObject);
                }
            }
14

There’s really no easy way to clear static state (you should just avoid it in general).

However, for destroying all GameObjects, you might want to take a look at how Zenject test utilities do it.

1 Like
15

Yeah, I already cleaned up any static code. My issues actually come from unity own modules (InputSystem) that get all wacky once an error occured in a test. It comes from the fact that they use static based code with a push/pop mecanism.

I’ll try Zenject code though. Thanks

16

I pulled my hair out on this one, do not forgot to NOT destroy the “ResourceManagerCallbacks” gameObject. If using the addressable package or async operation will not finish and hang the test silently. I’ll add a PR to Zenject

1 Like
17

Hi there,

I am bumping this thread because I am encountering the exact same issue, @Whatever560 did you finally manage to solve it ?

1 Like
18

I’m using the workaround I wrote about based on Zenject unit test cleaner. Basically I also filtered out the “ResourceManagerCallbacks” object that is in the DontDestroyOnLoad scene. So between tests both the Test Runner and the Addressable manager are kept.

Is it clear enough ?

I’m pretty sure I’ve filled the bug report on the subject.

The PR on the Zenject project, not using Zenject myself, just the cleaner for now https://github.com/modesttree/Zenject/pull/249

1 Like
19

Hi @Whatever560

I went a different path by changing my tests to Playmode tests, then went back to Editor tests, and could not reproduce the issue.

I marked the commit which had the issue, and made no changes so far, I will try to investigate the differences between the two commit to understand better what is causing it

20

I’m trying the new version of the test FWK (2.0.1, at Unity Test Framework overview | Test Framework | 2.0.1-exp.2), and it seems that I just need to clean up my own “Don’t destroy on load” objects now. No need to tag the “tests runner” as dont destroy on load anymore (beware of its new name, was previously “code-base tests runner”)

1 Like