Input when the game does not have focus (for integration tests)

Baste · October 20, 2020, 7:42am

Hi!

I’m running some automated tests, and I’m using the new Input system’s test setup (InputTestFixture) to test things.

It’s pretty great - being able to say “this is what needs to happen when the jump button is pressed” rather than “this is what needs to happen when the jump method is called” allows the game code to have the proper access modifiers, while still working with tests.

But I have a rather large problem. It seems like that if the game view (or Unity in general) doesn’t have focus, the inputs doesn’t get piped through to the game. This makes sense in general - if I tab away from the game and write stuff, I don’t want the player to run around.

But for tests - especially when there’s a lot of them - I want to be able to start them and then tab away. But now all my tests fail, because the input’s not going through!

Is there a way to work around this? I was trying to look for settings that had to do with focus, but I couldn’t find them. Ideally, this is something we could turn on in tests only.

Baste · October 20, 2020, 8:02am

One instance where this is really annoying is when I’m attaching my debugger and trying to step through the code during tests. Since the inputs are not active, I can’t really do that, which makes figuring out stuff a lot harder.

Rene-Damm · October 20, 2020, 12:42pm

Could you post the code for a test that is affected by this? Would like to have a closer look. I assume it’s a [UnityTest]?

Overall, InputTestFixture is set to ignore focus. In the editor, it will always toggle “Lock Input to Game View” on and InputTestRuntime has its own focus status (which is true by default and only changes when set explicitly in code) that is not tied to Application.isFocused (though looking at it, we don’t currently expose control over that through InputTestFixture; something to add).

But… for [UnityTest] tests, InputTextFixture hooks itself into the player loops… which is indeed affected by GameView focus. So my guess is this is where the problem is originating from. If that is indeed the case, it’s probably something that should be solved in the Unity test runner. IMO runs in the Test Runner window within the editor shouldn’t be affected by GameView focus. If this is indeed the source of trouble, I’ll go have a chat about it with the test framework people.

Baste · October 20, 2020, 1:18pm

When I’m testing, it seems like it’s been attempted to be fixed. If I start the tests and then click into a different window before the first test runs, all the tests work. It’s only if I alt-tab or click after the tests has started that things stop working properly.

The code looks like this. I’ve trimmed it down to only be a single unit-test, and removed a ton of extraneous details:

namespace Tests {
public class TestLedgeGrab {
    private bool sceneReady;
    private bool referencesSetup;

    private Gamepad gamePad;
    private InputTestFixture inputTester = new InputTestFixture();

    [OneTimeSetUp]
    public void OneTimeSetup() {
        EditorSceneManager.LoadSceneInPlayMode("Assets/Scenes/Unit Test Scenes/TestLedgeGrab.unity", new LoadSceneParameters(LoadSceneMode.Single));
        SceneManager.sceneLoaded += OnSceneLoaded;

        gamePad = InputSystem.AddDevice<Gamepad>();

        Time.captureFramerate = 60;
    }

    [OneTimeTearDown]
    public void OneTimeTearDown() {
        InputSystem.RemoveDevice(gamePad);
    }

    [TearDown]
    public void TearDown() {
        inputTester.Set(gamePad.leftStick, Vector2.zero);
        inputTester.Release(gamePad.buttonSouth);
    }

    void OnSceneLoaded(Scene scene, LoadSceneMode mode) {
        sceneReady = true;
    }

    [UnityTest]
    public IEnumerator TestLedgeHandling() {
        yield return new WaitWhile(() => sceneReady == false);

        var playerGO = Instantiate(...); // details skipped for brevity

        inputTester.PressAndRelease(gamePad.buttonSouth); //makes the player jump

        yield return new WaitForSeconds(.5f);

        Assert.IsTrue(PlayerHasGrabbedLedge()); // details skipped for brevity
    }

When testing, it’s clear that PressAndRelease gets called, but the code that listens doesn’t get any calls. Here’s the listener code, from a MonoBehaviour attached to the instantiated player object:

    //The type Teslagrad2Inputs is generated from an input actions asset with "Generate C# class" enabled.
    private Teslagrad2Inputs teslagrad2Inputs;

    private void OnEnable() {
        if (teslagrad2Inputs == null) {
            teslagrad2Inputs = new Teslagrad2Inputs();
            teslagrad2Inputs.Player.SetCallbacks(this);
        }

        teslagrad2Inputs.Player.Enable();
    }

...

    public void OnJump(InputAction.CallbackContext context) {
        // handle jump! This does not get called if the game loses focus during tests
    }

Baste · October 20, 2020, 1:23pm

Oh, and:
Unity 2020.1.2f1
Input System 1.0.0
Test Framework 1.1.18

Rene-Damm · October 20, 2020, 2:02pm

Thanks for the details. Much appreciated.

Just to make sure it’s not that, could you give 1.1-preview.1 a try? There was a fix for [UnityTest] tests running over more than a single frame in there and I’d like to make sure it’s not that the same issue.

Baste · October 20, 2020, 2:57pm

Bug’s still there when I activate 1.1.0-preview.1 in the package manager.

I can look into creating a repro project if you think this’ll be considered a bug.

Rene-Damm · October 20, 2020, 3:08pm

Thanks for giving it a go.

This just caught my eye

Looking only at the test setup here, this will not isolate the input system. Instantiating InputTestFixture is not enough. It needs to have its Setup() and TearDown() methods called. My guess is you’re actually running with just input system as is and thus get the full impact of focus switching and such.

Note that calling InputTestFixture.TearDown() also makes it unnecessary to do any input-related cleanup. All devices and custom registrations will be thrown away automatically.

With those changes, my guess would be the problem you’re seeing will go away. (note you’ll still need 1.1-preview.1 for the run-UnityTests-over-multiple-frames fix)

For 1.1, we’ve clarified this better (hopefully) in the docs.

Baste · October 20, 2020, 4:12pm

Thanks, I’ll give it a try tomorrow!

Am I supposed to call those methods in [SetUp]/[TearDown] or [OneTimeSetup]/[OneTimeTeardown]?

Rene-Damm · October 20, 2020, 4:15pm

Depends a bit on what you’re going for. Personally, would recommend setting up a test environment for each test and tearing it down after each one so that each test gets clean, known state and there’s no bleeding from one test into another. In that case it’d be [SetUp] and [TearDown]. But the choice is up to you. The fixture should do its job just fine sticking around for an entire test run.

Baste · October 21, 2020, 1:21pm

Welp, that broke everything!

I added this to my test setup:

    [SetUp]
    public void SetUp() {
        inputTester.Setup();
    }

    [TearDown]
    public void TearDown() {
        inputTester.TearDown();
    }

When that’s there, interacting with the inputTester during tests causes nullRefs deep in the input system. This line of code:

inputTester.PressAndRelease(gamePad.buttonSouth);

Gives this error:

TestLedgeHandling_Blink1 (0,316s)
---
System.NullReferenceException : Object reference not set to an instance of an object
---
at UnityEngine.InputSystem.LowLevel.InputStateBuffers+DoubleBuffers.GetFrontBuffer (System.Int32 deviceIndex) [0x00001] in E:\Teslagrad2\Library\PackageCache\com.unity.inputsystem@1.1.0-preview.1\InputSystem\State\InputStateBuffers.cs:73
  at UnityEngine.InputSystem.LowLevel.InputStateBuffers.GetFrontBufferForDevice (System.Int32 deviceIndex) [0x00001] in E:\Teslagrad2\Library\PackageCache\com.unity.inputsystem@1.1.0-preview.1\InputSystem\State\InputStateBuffers.cs:126
  at UnityEngine.InputSystem.InputControl.get_currentStatePtr () [0x00000] in E:\Teslagrad2\Library\PackageCache\com.unity.inputsystem@1.1.0-preview.1\InputSystem\Controls\InputControl.cs:788
  at UnityEngine.InputSystem.LowLevel.DeltaStateEvent.From (UnityEngine.InputSystem.InputControl control, UnityEngine.InputSystem.LowLevel.InputEventPtr& eventPtr, Unity.Collections.Allocator allocator) [0x00074] in E:\Teslagrad2\Library\PackageCache\com.unity.inputsystem@1.1.0-preview.1\InputSystem\Events\DeltaStateEvent.cs:88
  at UnityEngine.InputSystem.InputTestFixture.Set[TValue] (UnityEngine.InputSystem.InputControl`1[TValue] control, TValue state, System.Double time, System.Double timeOffset, System.Boolean queueEventOnly) [0x000b4] in E:\Teslagrad2\Library\PackageCache\com.unity.inputsystem@1.1.0-preview.1\Tests\TestFixture\InputTestFixture.cs:457
  at UnityEngine.InputSystem.InputTestFixture.Press (UnityEngine.InputSystem.Controls.ButtonControl button, System.Double time, System.Double timeOffset, System.Boolean queueEventOnly) [0x00001] in E:\Teslagrad2\Library\PackageCache\com.unity.inputsystem@1.1.0-preview.1\Tests\TestFixture\InputTestFixture.cs:347
  at UnityEngine.InputSystem.InputTestFixture.PressAndRelease (UnityEngine.InputSystem.Controls.ButtonControl button, System.Double time, System.Double timeOffset, System.Boolean queueEventOnly) [0x00001] in E:\Teslagrad2\Library\PackageCache\com.unity.inputsystem@1.1.0-preview.1\Tests\TestFixture\InputTestFixture.cs:359
  at Tests.TestLedgeGrab+<TestLedgeHandlingForSpawnPoint>d__24.MoveNext () [0x00186] in E:\Teslagrad2\Assets\Scripts\Editor\Unit Tests\Runtime Tests\TestLedgeGrab.cs:89
  at UnityEngine.TestTools.TestEnumerator+<Execute>d__6.MoveNext () [0x0004c] in E:\Teslagrad2\Library\PackageCache\com.unity.test-framework@1.1.18\UnityEngine.TestRunner\NUnitExtensions\Attributes\TestEnumerator.cs:36

I also managed to break Unity completely when doing things. I did it by messing up stuff when experimenting, but it’s a very easy repro; run this code:

    [MenuItem("Test/Break Unity Kinda")]
    public static void EnoughToBreakUnity() {
        new InputTestFixture().TearDown();
    }

And then recompile anything. Unity will start spamming two errors outside of play mode until the editor is restarted completely:

Error 1:

TypeInitializationException during event processing of Editor update; resetting event buffer
UnityEngine.InputSystem.LowLevel.<>c__DisplayClass7_0:<set_onUpdate>b__0(NativeInputUpdateType, NativeInputEventBuffer*)
UnityEngineInternal.Input.NativeInputSystem:NotifyUpdate(NativeInputUpdateType, IntPtr)

Error 2:

NullReferenceException: Object reference not set to an instance of an object
UnityEngine.InputSystem.InputSystem.InitializeInEditor (UnityEngine.InputSystem.LowLevel.IInputRuntime runtime) (at Library/PackageCache/com.unity.inputsystem@1.1.0-preview.1/InputSystem/InputSystem.cs:2999)
UnityEngine.InputSystem.InputSystem..cctor () (at Library/PackageCache/com.unity.inputsystem@1.1.0-preview.1/InputSystem/InputSystem.cs:2916)
Rethrow as TypeInitializationException: The type initializer for 'UnityEngine.InputSystem.InputSystem' threw an exception.
UnityEngine.InputSystem.InputAnalytics.OnStartup (UnityEngine.InputSystem.InputManager manager) (at Library/PackageCache/com.unity.inputsystem@1.1.0-preview.1/InputSystem/InputAnalytics.cs:25)
UnityEngine.InputSystem.InputManager.OnUpdate (UnityEngine.InputSystem.LowLevel.InputUpdateType updateType, UnityEngine.InputSystem.LowLevel.InputEventBuffer& eventBuffer) (at Library/PackageCache/com.unity.inputsystem@1.1.0-preview.1/InputSystem/InputManager.cs:2537)
UnityEngine.InputSystem.LowLevel.NativeInputRuntime+<>c__DisplayClass7_0.<set_onUpdate>b__0 (UnityEngineInternal.Input.NativeInputUpdateType updateType, UnityEngineInternal.Input.NativeInputEventBuffer* eventBufferPtr) (at Library/PackageCache/com.unity.inputsystem@1.1.0-preview.1/InputSystem/NativeInputRuntime.cs:60)
UnityEngine.InputSystem.LowLevel.<>c__DisplayClass7_0:<set_onUpdate>b__0(NativeInputUpdateType, NativeInputEventBuffer*)
UnityEngineInternal.Input.NativeInputSystem:NotifyUpdate(NativeInputUpdateType, IntPtr)

So right now it looks to me like any call to InputTestFixture.SetUp or TearDown will break the tests, and also maybe Unity.

Rene-Damm · October 21, 2020, 2:22pm

The one time setup is interferring with the per-test setup. Would recommend going one way or the other. What the mix does is run the one-time setup which adds the Gamepad to the non-isolated system and then the per-test setup saves and wipes the system state to put input in isolation mode. Thus the exception when you access the Gamepad that belongs to the non-isolated system.

Sorry, but that code doesn’t make sense InputTestFixture is touching global system state in its setup and teardown work, so yes, invoking them arbitrarily will wreak havoc on the global system state. That’s expected. (////EDIT: I do believe that in 1.1-preview.2 there’s now some check that will, as a side-effect, make TearDown not do any work if Setup wasn’t called)

Baste · October 21, 2020, 2:41pm

Thanks for all the help so far! I’ve changed the setup code:

    private bool sceneReady;

    private Gamepad gamePad;
    private InputTestFixture inputTester;

    [OneTimeSetUp]
    public void OneTimeSetup() {
        EditorSceneManager.LoadSceneInPlayMode("Assets/Scenes/Unit Test Scenes/TestLedgeGrab.unity", new LoadSceneParameters(LoadSceneMode.Single));
        SceneManager.sceneLoaded += OnSceneLoaded;
    }

    [SetUp]
    public void Setup() {
        inputTester = new InputTestFixture();
        inputTester.Setup();
        gamePad = InputSystem.AddDevice<Gamepad>();
    }

    [TearDown]
    public void TearDown() {
        inputTester.TearDown();
    }

    void OnSceneLoaded(Scene scene, LoadSceneMode mode) {
        sceneReady = true;
    }

This made me run into a new problem:

If I run a bunch of tests, the first one succeeds, and the other ones fail. They fail in a pretty strange way, though: the scene doesn’t have any objects anymore. For all tests after the first one, SceneManager.GetActiveScene().GetRootGameObjects() is an empty list. My tests rely on finding spawn points through GameObject.Find and instantiating the player prefab there, so all the tests after the first one is failing on that.

Commenting out inputTester.TearDown(); causes the issue to disappear (and the tests to work), so it seems like inputTester.TearDown somehow breaks the loaded scene or something?

Baste · October 21, 2020, 2:43pm

That’s good to hear! I was aware that the code was nonsensical, I wrote it originally by commenting out one line of code and forgetting to comment out another one. The error was expected, but having to restart the editor by calling methods in an unexpected order should probably not happen

Rene-Damm · October 21, 2020, 3:26pm

Ah, I kinda feared that would become an issue And yup, it’s InputTestFixture messing around. In particular, this code. Pretty sure the fixture shouldn’t do that. Makes sense for our own tests but not necessarily for other test suites. Think it’s the kind of thing that just crept in there and managed to stick around.

I’ll go see if I can manage to sneak a PR into 1.1-preview.2 last second to get this addressed in some form (even if for now it’s just a way to toggle the behavior off).

As a workaround, temporarily adding an empty scene while calling TearDown() should probably prevent the code from doing damage for now.

Baste · October 21, 2020, 3:41pm

Hah, yes that would break our tests Thanks for the help again!

I think I’ll make a subclass of InputTestFixture that overrides TearDown and just… doesn’t do that. That shouldn’t lead to any problems, right?

Rene-Damm · October 21, 2020, 3:51pm

You’d still probably want the rest of what the method does. If TearDown() isn’t called but Setup() is, the input system is left hanging in an isolated state and the “real” system state won’t get swapped back in.

Baste · October 22, 2020, 8:07am

Hacky hacky: I noticed that the TestFixture only deletes objects with HideFlags set to 0, so I simply set all the hide flags to something else, run base.TearDown(), and restore the hide flags.

… With a big, fat, comment about how this will all be fixed in the future!

    public override void TearDown() {
        var rootGameObjects = SceneManager.GetActiveScene().GetRootGameObjects();

        var hideFlags = rootGameObjects.Select(go => go.hideFlags).ToArray();
        rootGameObjects.Each(go => go.hideFlags = HideFlags.DontSave);

        base.TearDown();

        rootGameObjects.EachIndex(i => rootGameObjects[i].hideFlags = hideFlags[i]);
    }

willwolfram18 · December 11, 2021, 2:29pm

Wanted to drop by and say thank you for this little clue! I’ve been trying to figure out why some of my tests were failing when running from CLI but could never get them to fail when running from the editor. I feel silly that I didn’t notice InputTestFixture has it’s own [SetUp] that needs to be called. Appreciate your feedback on this thread!