I’ve recently been forced to upgrade the Unity Test Framework from 1.1.x to 1.4.x due to Unity.Collections package minor version bump, and I do have some feedback. It was a huge downgrade that broke workflow and introduced a handful of annoying bugs.
Related bug report: IN-88016
Main feedback: Splitting PlayMode and Player tests into separate tabs is bad. Imagine a scenario - you want to run 300 play mode tests in the build. Build has say 30 seconds overhead to get compiled before it starts to run. If 295 tests passed, there is no easy way to rerun failed tests in PlayMode, because test results and even search filters are separate in PlayMode and Player tabs. So each test you want to rerun in PlayMode has to be manually found in PlayMode tab. This is extremely annoying!
Additionally, here are a few bugs. Note: I have my tests split into multiple categories, and I always run with some categories filtered out. Most bugs are related to running tests in Player or PlayMode.
- Stopping a play mode test mid run hangs the runner state until Cancel Run is pressed (playmode video clip, player video clip)
- Rerun failed doesn’t work if categories are selected, it runs a random set of tests, some from unselected categories (video clip)
- When Running multiple tests, parent icons don’t get refreshed until all tests have finished running (video clip, related video clip)
- Parent categories / groups don’t show timing anymore (Shows: Tests (300 tests) (0.000s)) (video clip part 1, clip part 2)
- When a group of tests run, sometimes last tests shows as not yet finished and wait for player run is hanging, even though test has finished and player shows that all tests are completed, and never refreshes if test run was aborted (video clip of hanging, video clip of never refreshin after aborting)
- Tests from unrelated packages are showing up in my tests and no way to remove them (e.g. input system integration tests) (video clip)
- If failed tests were filtered out, when all test finish running, test runner shows 0 / 0 / 0 until you click one of the buttons to toggle filtering (video clip)
- Sometimes I’m getting “An unexpected error happened while runnign tests”:
NullReferenceException: Object reference not set to an instance of an object
UnityEditor.TestTools.TestRunner.TestRun.Tasks.PlayModeRunTask+d__1.MoveNext () (at ./Library/PackageCache/com.unity.test-framework@1.4.5/UnityEditor.TestRunner/TestRun/Tasks/PlayModeRunTask.cs:52)
UnityEditor.TestTools.TestRunner.TestRun.TestJobRunner.ExecuteStep () (at ./Library/PackageCache/com.unity.test-framework@1.4.5/UnityEditor.TestRunner/TestRun/TestJobRunner.cs:173)
UnityEditor.EditorApplication:Internal_CallUpdateFunctions()