Bug: Unity Test Runner swallows exceptions from child coroutines

I've seen this bug now twice, so I thought that I would mention it. Currently if you have a coroutine like:

public IEnumerator Parent(){

    yield return LoadScene("Foo.unity");
    SceneManager.GetSceneByName("Foo").GetRoots(); // Test runner reports invalid scene error.
}

public IEnumerator LoadScene(string name) {
    yield return null;
    Assert.AreNotEqual(name, "Foo.unity", "Please don't load foo"); // Test runner swallows this assert.
    yield return SceneManager.LoadSceneAsync(name);
}

The exception thrown by the child LoadScene will be swallowed. Instead, you only get an error in Parent(), line 2, that the loaded scene is invalid.

In the TestRunner package, this is because the function ExecuteEnumerableAndRecordExceptions only records one exception at a time. New exceptions overwrite the old exceptions recorded, rather than maintaining a list. When LoadScene() throws an exception, it doesn't stop the parent coroutine from running (which it probably should).

This bug makes the test errors panel un-useful and means test errors (especially in a CI machine) are very hard to parse / track down.

2 Likes

I think this bug was fixed for a short while, but has since regressed. There's at least this report open ATM: https://issuetracker.unity3d.com/issues/unitytests-do-not-fail-when-nested-coroutines-throws-an-exception

Shameless self-promotion: If you want better error reports in CI for highly asynchronous code, check out Responsible, which also works around that bug :)

1 Like

Argh, has that been open since 2018? Is anyone actively working on the testing package?

1 Like

Man, nested coroutine is a mess to debug ...

Yes, I ended up writing my own Coroutine runner and stack trace debugging system. >.>

[Edit: Fixed see next post]

Would you mind sharing some thoughts here ?

Here is some of my code :

        /// <summary>
        /// Test each game scene to check if fallback context works correctly
        /// </summary>
        /// <returns></returns>
        [UnityTest, TimeoutDebug]
        public IEnumerator DirectSceneTest()
        {
            yield return BaseAppScenesTest.DirectSceneTest();
        }
 public IEnumerator DirectSceneTest()
        {
            var dummyScene = "DummyScene.unity";
            string[] scenes =
            {
                "SceneA",
                "SceneB"
            };

            yield return new WaitForEndOfFrame();

            foreach (var scene in scenes)
            {
                yield return Addressables.LoadSceneAsync(scene);

                Logger.Debug(nameof(DirectSceneTest), "Scene loaded Async");

                yield return null;

                //Wait for any on going transition to finish. This is internal magic
                yield return AppInjectorBh.CurrentExporter
                             .GetExportedValue<IStateMachine>()
                             .GetCurrentStateTransitionAsync().AsUniTask().ToCoroutine();

                //Clean up by loading an empty scene and destroy things Zenject style
                yield return Addressables.LoadSceneAsync(dummyScene);
                UnityTestUtils.DestroyEverythingExceptTestRunner();

                yield return null;
            }
        }

I don't get why but when I debug, I go through the loop once without issue. But then on Scene B load, It stops on L.14 and never continues on the second yield.

The scenes are correctly loaded but the test stops silently, the log message never gets printed, I enabled all exceptions in the debugger but nothing is thrown.

Hey I fixed it : https://discussions.unity.com/t/766528/16

It's probably because L14 throws an exception, which is then caught and swallowed by the coroutine runner you're using (this is running via the Test Runner?). I would:
- Step into L14 to figure out which of the code within it throws the error.
- Check the test output to see if you see the exception.

Ahh, haha. Awesome! Oof, silent hangs are rough. Should have though of that given that Addressables are very asynchronous.

2 Likes


Yeah unfortunatelly none of it was working. It just occured to me that I was seeing some GO poping in the scene list, and after investigating on its role found out it was a part of Addressable workflow. I just wild guessed it was the issue. And it was.