How to restructure tests which open and iterate over multiple scenes

Unity Engine
1

I have a series of tests which I’d like to refactor because they contain a lot of duplicated code. The current structure is like so:

public class SceneValidationTests
{
    [Test]
    public void LevelScenes_OnStart_ContainObjectBla()
    {
        // Get list of all level scenes.
        // Iterate and load all scenes.
        // Assert GameObject is present.
    }

    [Test]
    public void LevelScenes_OnStart_ContainObjectBla2()
    {
        // Get list of all level scenes.
        // Iterate and load all scenes.
        // Assert GameObject 2 is present.
    }
}

Every test verifies that a specific object must be present or some settings need to make sense in its context (kind of like sanity checks for level designers). Each tests needs to verify these conditions for all scenes, so they all start by opening the scenes, checking the condition and iterating through the list of scenes. This means, that for each test, all scenes are opened, which can become quite slow.

It would be much better, if I could open every scene only once and then run all tests for this scene. Similar to this:

public class SceneValidationTests
{
    [TestEntry] // This attribute does not exist, but it would be called as a parent to all other tests.
    public void RunForAllScenes()
    {
        // Iterate and load all scenes
        // Automatically call all test methods
    }

    [Test]
    public void LevelScenes_OnStart_ContainObjectBla(Scene scene)
    {
        // Assert GameObject is present.
    }

    [Test]
    public void LevelScenes_OnStart_ContainObjectBla2(Scene scene)
    {
        // Assert GameObject 2 is present.
    }
}

What’s a good way to structure tests in this manner? I would like to keep each test its own [Test]-case so it shows up as a single entry in the test runner and each test should be able to run on its own. So, ideally, I would like to define this kind of method similar to TestSetUp, which runs before a test or kind of wraps it, but also runs only once if multiple tests should be run.

Is there an easy solution to this or would this be a new feature possible to implement in the future?

2

A bit unrelated, but how are you going about testing multiple scenes? Have you figured out any specific code to load a scene in Play mode tests?

3

I’m opening and checking the scenes in Edit Mode like this:

public class SceneValidationTests
{
    [Test]
    public void LevelScenes_Setup_HasValidSettingsObject()
    {
        // Load and iterate over all scenes in our special folder.
        var sceneGUIDs = AssetDatabase.FindAssets("t: Scene", new[] { "Assets/MyScenes/" });
        for (int i = 0; i < sceneGUIDs.Length; i++)
        {
            string path = AssetDatabase.GUIDToAssetPath(sceneGUIDs[i]);
            var scene = EditorSceneManager.OpenScene(path);

            // Check that each scene contains a special object.
            var specialObject = Object.FindObjectOfType<MySpecialMonoBehaviour>();
            Assert.IsNotNull(specialObject);
        }
    }
}

This is much faster than Play Mode tests, but it works similar to Play Mode. You need to setup a test assembly for Play Mode tests and then change the code to load scenes as you would in your game via SceneManager.

4

I demoed pretty much exactly this scenario at Unite. The trick is to use a parametric test fixture, which you set up to get instantiated once for each of your scenes. You pass in the scene name/path/whatever in the constructor and store it, and then in the OneTimeSetUp you do the actual load of the scene. Then in each test you just do the check for that one scene.

1 Like
5

@superpig TestFixtureSource is exactly what I was hoping for! And I looked up the talk, it’s an incredibly helpful and inspirational to get me started thinking about how to test things. Thank you! :slight_smile: I already knew about parameterized test cases, but never knew what a TestFixture was used for.

6

Edit: Solution below.

After my first excitement about the TestFixtureSource and your example, I ran into a strange issue. My tests all used to pass, then I refactored them to use the TestFixtureSource following your Unite talk. I really copied the same approach, but for some reason the tests now randomly seem to fail without error messages and most of the tests actually appear as skipped.

5426793--552348--upload_2020-1-30_21-57-19.png

Any idea what could be going wrong here?

I’m having troubles finding an issue. If I remove some of the tests, all others pass again, but some of them are still skipped. If I remove other tests, again other tests randomly fail or are skipped.

namespace Nementic.X.SoloMode
{
    using Nementic.X.Worldmap;
    using NUnit.Framework;
    using System.Collections;
    using System.Collections.Generic;
    using UnityEditor;
    using UnityEditor.SceneManagement;
    using UnityEngine;
    using UnityEngine.SceneManagement;

    public class ZodiacScenesProvider : IEnumerable<string>
    {
        IEnumerator<string> IEnumerable<string>.GetEnumerator()
        {
            var sceneGUIDs = AssetDatabase.FindAssets("t: Scene", new[] { "Assets/Runtime/SoloMode/LevelSelectionScenes" });
            for (int i = 0; i < sceneGUIDs.Length; i++)
            {
                string path = AssetDatabase.GUIDToAssetPath(sceneGUIDs[i]);
                yield return path;
            }
        }

        public IEnumerator GetEnumerator() => ((IEnumerable<string>)this).GetEnumerator();
    }

    [TestFixture]
    [TestFixtureSource(typeof(ZodiacScenesProvider))]
    public class SaveKeyTests
    {
        private readonly string scenePath;
        private Scene scene;

        public SaveKeyTests(string scenePath)
        {
            this.scenePath = scenePath;
        }

        [OneTimeSetUp]
        public void LoadScene()
        {
            this.scene = EditorSceneManager.OpenScene(this.scenePath, OpenSceneMode.Single);
        }

        [Test]
        public void ValidateMultipleBoardsFilePathCorrect()
        {
            var multipleBoardsVariant = AssetDatabase.LoadAssetAtPath<LevelVariant>(
                "Assets/Runtime/SoloMode/SoloManagement/LV_ScriptableObjects/LV-02_MultipleBoards.asset");

            multipleBoardsVariant.hideFlags = HideFlags.DontUnloadUnusedAsset;

            Assert.That(multipleBoardsVariant != null, "No level variant source found.");

            foreach (var soloLevel in Object.FindObjectsOfType<SoloLevel>())
            {
                // Ignore tutorials (indicated by presence of a guide).
                if (string.IsNullOrEmpty(soloLevel.guideCharacterKey) &&
                    soloLevel.LevelVariant == multipleBoardsVariant)
                {
                    Assert.That(soloLevel.FilePaths.Length == 1);
                    Assert.That(soloLevel.FilePaths[0].Contains("Multiple_Boards"),
                        AssertMessage("Incorrect file path for multiple boards.", scene, soloLevel));
                }
            }

            Resources.UnloadAsset(multipleBoardsVariant);
        }

        private static string AssertMessage(string message, Scene scene, Object context)
        {
            return $"{message} Context: '{context.name}'. Scene: {scene.name}.";
        }

        [Test]
        public void ValidateMultipleBoardsCorrectSettings()
        {
            var multipleBoardsVariant = AssetDatabase.LoadAssetAtPath<LevelVariant>(
                "Assets/Runtime/SoloMode/SoloManagement/LV_ScriptableObjects/LV-02_MultipleBoards.asset");

            Assert.NotNull(multipleBoardsVariant);

            foreach (var soloLevel in Object.FindObjectsOfType<SoloLevel>())
            {
                if (soloLevel.FilePaths.Length == 1)
                {
                    string ubongoPath = soloLevel.FilePaths[0];
                    if (ubongoPath.Contains("Multiple_Boards"))
                    {
                        Assert.That(soloLevel.LevelVariant == multipleBoardsVariant, "Solo Level with incorrect level variant: " + soloLevel.name + " - " + scene.name);
                        Assert.That(soloLevel.RandomizeBoardRotation == false, "Solo Level with incorrect RandomizeBoardRotation: " + soloLevel.name + " - " + scene.name);
                        Assert.That(soloLevel.RandomizeBoardFlip == true, "Solo Level with incorrect RandomizeBoardFlip: " + soloLevel.name + " - " + scene.name);
                        Assert.That(soloLevel.HasTimer == false, "Solo Level with incorrect HasTimer: " + soloLevel.name + " - " + scene.name);
                    }
                }
            }
        }

        [Test]
        public void ValidateZodiacTotalCrystalCount()
        {
            int expectedSum = Object.FindObjectsOfType<SoloLevel>().Length * 3;
            var label = Object.FindObjectOfType<WorldmapCrystalCount>();
            Assert.AreEqual(expectedSum, label.maxCount, "Mismatched crystal count for zodiac HUD: " + scene.name);
        }

        [Test]
        public void ValidateZodiacFinalLevelCrystalCount()
        {
            int totalSum = Object.FindObjectsOfType<SoloLevel>().Length * 3;
            foreach (var gate in Object.FindObjectsOfType<CrystalGateLock>())
            {
                Assert.Less(gate.crystalUnlockCount, totalSum - 3, "Gate unlock count must be less than total crystal count of all level minus the last one: " + scene.name + " - " + gate.name);
            }
        }

        [Test]
        public void ValidateZodiacHUDCrystalCount()
        {
            var crystalCount = Object.FindObjectOfType<WorldmapCrystalCount>();
            var nodeKey = Object.FindObjectOfType<SoloLevel>().saveKey;
            string animalName = nodeKey.Split('/')[1];

            Assert.That(crystalCount.saveKey.Contains(animalName), "Zodiac HUD crystal count save key should contain the animal name as specified in the nodes save keys. Scene: " + scene.name);
        }
    }
}

After quite some investigation I found my issue. My example code only shows a couple of the tests, there are a few more methods which are very similar. The entire class is not refactored well, since I was only now starting to reorganize the code. In one of the tests, I opened another unrelated scene to test something, which used to work in the old structure, but now with the TestFixtureSource it breaks, since the scene reference suddenly becomes null. However, the issue was hidden quite well since it only appeared for specific ordering of the tests and sometimes not at all.

Additionally, I was experiencing a visual glitch in the test runner, where some of the tests showed as failed, but without error message. When I toggled the display state of failed/passing tests in the top right corner of the runner window, the visuals were updated and showed the correct symbols again.