Trying the test framework but having some issues with it.

Hello. I've started looking into implementing tests for my game and I'm having some issues with it.

I figured, I would start testing the menu navigation at first. My plan is to find buttons by hierarchy and inject input to the UI somehow but I'm not quite there yet.

So, I have this little editor test:

public class MenuNavigationTests
{
    [UnitySetUp]
    public IEnumerator SetUp()
    {
        Debug.Log("Entering play mode");
        yield return new EnterPlayMode();
    }

    [Test]
    public void MyTest()
    {
        Debug.Log("This runs inside playmode");
    }

    [UnityTearDown]
    public IEnumerator TearDown()
    {
        Debug.Log("exiting play mode");
        yield return new ExitPlayMode();
    }
}

Now, the first issue, is that "Entering play mode" is printed twice in the logs, with no explanation.

Second, I have some hooks when the play mode changes, and some extra UI around the play button to start the game with certain setups (start from the menu, jump right in, load a save game, stuff like this). The reason I have hooks is so that regardless the scene I have opened, it will start from the menu scene and then when the play mode is done, it will switch back to the scene which was originally opened.
All this has been working fine for months now but when going through the Test Runner, when OnPlayModeChanged gets called with PlayModeStateChange.ExitingEditMode,
EditorSceneManager.GetActiveScene().path returns an empty string. The opened scene is ofc one of the serialized scenes and it still works fine if I just press the play button myself.

Could anyone help me out here?

I’m still puzzled by the behaviour of the test framework.

I figured, I’ll just use some code to disable the custom toolbar and playmode hooks and just to what needs to be done manually.

I put this in my Editor test:

    [UnitySetUp]
    public IEnumerator SetUp()
    {
        CustomToolbar.Enabled = false;
        if (EditorSceneManager.SaveCurrentModifiedScenesIfUserWantsTo())
        {
            EditorSceneManager.OpenScene("Assets/Scenes/Menu.unity");
            yield return new EnterPlayMode();
        }
        else
        {
            yield return null;
        }
       
    }

and I somehow get this output

MyTest (0,148s)
---
SetUp : System.InvalidOperationException : This cannot be used during play mode.
---
--SetUp
  at (wrapper managed-to-native) UnityEditor.SceneManagement.EditorSceneManager.SaveCurrentModifiedScenesIfUserWantsTo()

Given the fact that EnterPlayMode was not called yet and that it’s an Editor Test, I’m not sure why would this error be thrown.

Without this, I’m not sure how am I supposed to safely switch to a different scene from a script.

I’d understand this behaviour if the test framework was in preview, but it looks like it’s marked as production ready and yet, it seems buggy and unusable.

Could you share how your asmdef for the tests is set up? (Either a screenshot of it in the Inspector, or the text contents of the file - it’s just JSON).

@superpig

Here it is:

{
    "name": "Tests",
    "rootNamespace": "",
    "references": [
        "UnityEngine.TestRunner",
        "UnityEditor.TestRunner",
        "MainEditorAssembly"
    ],
    "includePlatforms": [
        "Editor"
    ],
    "excludePlatforms": [],
    "allowUnsafeCode": false,
    "overrideReferences": true,
    "precompiledReferences": [
        "nunit.framework.dll"
    ],
    "autoReferenced": false,
    "defineConstraints": [
        "UNITY_INCLUDE_TESTS"
    ],
    "versionDefines": [],
    "noEngineReferences": false
}

MainEditorAssembly is another asmdef that I use to compile all my editor scripts and then that one has a reference to "MainAssembly" which has all my runtime code.

Hmm. @Warnecke , what do you think?

@superpig @Warnecke Any news on this one? I can still skimp by not automating the testing of my game but it’d be really nice if I could start working on it. Would it make sense for me to look into the internals of the test framework without being able to also look at Unity’s source code?

Well, you have the test framework source code (in the package), plus the C# sources for UnityEngine/UnityEditor.dll… you might be able to get more insight with that.

But I think this is something the team should take a look at. Could you possibly file a bug report through the Unity Bug Reporter app, with a repro case based on one of the scripts you’ve shown in this thread?

You could also simply use the new Unity automation-testing framework designed around making automated UI testing simple. This framework makes loading into a scene and interacting with GameObject simple. Based on what I am seeing in your comments here, you should be able to set this up and accomplish what you are doing in a matter of a few minutes.

@superpig Sorry for replying rather late, I’ve had some bigger issues with my game and I only work on it after my day job and as you can imagine progress is slow.

I found this rather hard to debug myself since I also have an issue where I can’t keep a debugger attached when unity switches from editor mode to play mode.

I successfully reproduced the issue in a very small example which I’ve submitted as a bug. It’s a 100% reproducible.

The case is 1395077

@tsibisky I appreciate trying to help but that package does not look like it could help me out. I’d like to have full programmatic control over all tests, entering and exiting play mode from code as needed, and asserting various conditions as the tests progress.

Excellent, thanks for taking the time to do that! I've put the bug into our QA fasttrack path so someone should start looking at it on Monday.

@superpig Awesome. If anything, please ask on the email that I've used to submit the bug with. I'm likely to reply very fast on that one if I'm in front of the PC.

@superpig I’ve submitted a new bug (CASE 1398944) about the fact that calling "
EditorSceneManager.GetActiveScene().path" in a test setup returns an empty string. Without something like this I can’t swich to a specific scene and then switch back the end of my tests.

Here’s the whole code if anyone is curious:

using System.Collections;
using UnityEngine;
using UnityEngine.TestTools;
using UnityEditor.SceneManagement;

public class BugTests
{
    [UnitySetUp]
    public IEnumerator SetUp()
    {
        Debug.Log(EditorSceneManager.GetActiveScene().path);
        yield return null;
    }
   
    [UnityTest]
    public IEnumerator MyTest()
    {
        yield return null;
    }

    [UnityTearDown]
    public IEnumerator TearDown()
    {
        yield return null;
    }
}