Has anyone seen this exception before?
I was running all my PlayMode tests when they ended early with this error.
NullReferenceException: Object reference not set to an instance of an object
UnityEngine.TestTools.TestRunner.PlaymodeTestsController+d__15.MoveNext () (at Library/PackageCache/com.unity.test-framework@1.1.3/UnityEngine.TestRunner/TestRunner/PlaymodeTestsController.cs:74)
All tests pass individually or in their suites.
But running as a large group by selecting all and running causes this error.
It keeps breaking after the same number of tests so the error is consistent.
I tried deleting the Library folder but no luck.
It’s never happened before.
I did make a change to one of my tests but:
- The method signature is the same. Iow, it’s not the UnityTest with void return versus Test with IEnumerator return error
- The change only had to do with the removal of an Assert so I’m not sure what caused it.
- This test runs to completion sucessfully and additional test suites are ran afterwards.
But it does have something to do with my code changes apparently, because the exception does not occur on my develop branch, only the one with my changes.
As an update to this…
Is there a max number of frames the test runner is willing to process?
Long story short, my test had a loop waiting for a variable to change which has it’s maxFrame count set to 1000.
I dropped it down to 100 and the exception is gone…
[Update] As far as I can tell, this is definitely not caused by a code change.
I cloned the branch into another directory structure and I don’t see the problem.
I had the same issue and I can only agree. I changed a lot but in the end it was just the order of the PlayModeTest (without any dependencies in my script).
[Edit] Some changes later the exception showed up again. I deleted “yield return new WaitForSeconds(4); in the PlayModeTest” and it was running again.
It seems I encountered the same error now. When executing a test individually it always passes. When pressing “Run All” it most of the time does not (though even more confusingly, very rarely it does). Given that it is rather a simple test, and it always passes individually, I don’t think it itself is flaky.
I tried making the test longer (eg. with WaitForSeconds(10) or so) so I can pause and see what is in the Objects Hierarchy, but it fails immediately.
Annoying is that the error it fails with seems to be coming from my Assert (so I was trying to look for state that is not correctly cleaned up or something like that) but after this thread I think it might be a red herring.
It seems running the “Run All” tests in editor is simply too flaky to be actually useful 
If not that one then another fails.
Since it seems that running it from command line (like described here Running tests from the command line | Test Framework | 1.1.33) still works properly (though I tried only a few times, so who knows…), I guess I am just going to ever run them individually in the editor and rely on continuous integration tests to catch any regressions.
But I would be really happy If anyone found a reason for that flakiness!
1 Like
If your tests are that simple you could start a bug report. It may help to improve the quality of the test system.