In the Unite talk, Richard @superpig showed a technique he called “Content Validation” - these are tests that are executed on a scene or prefab to verify that it meets a certain set of expected conditions.
This is a very powerful technique, especially when baking it right into the build process (as shown in the video as well).
We are actually doing this already (although i refer to these kind of validations as “Integrity Tests” since they verify the scene or prefab ‘integrity’). Our validations run when processing the scenes before a build or even when saving assets to disk (to prevent ‘faulty’ assets being checked in to source control).
Since we are not on 2019.x we are still using the built-in test runner, so all of our tests are not implemented as “pure” NUnit tests, instead we have our own infrastructure for it.
My question is: what is the benefit of setting up these validations as tests ? what can be gained from this that we don’t already have ?
You can always write tests as ‘just some code that does a bunch of if-guarded LogError statements.’ So why use a test framework at all? Ultimately I think it’s always about workflow - workflow for authoring tests, workflow for running tests, and workflow for investigating failures.
It sounds like you already have a workflow you’re happy with for running tests, so the main advantages that remain are in authoring tests (would it be easier for your team to plug in new tests if they didn’t have to do it in a central place?) and in investigating failure. In particular: using the TestRunner window to see which validations actually pass/failed (and which were skipped entirely) and to look at error messages more easily than in the Console; and, being able to save out the results to standard NUnit XML, which can be consumed by various CI systems so that you get your content report directly in the UI of your build overview page.
If you’re already running the tests on every save, I assume they’re pretty fast. Perhaps there are tests that you’ve not added because you didn’t want to slow down saving too much, but they could still be useful to run at build-time, or even just run manually when requested by an artist/designer - for example, testing that a level is solvable? Using categories (and possibly the TestRunner window) could be a good way to have those extra tests and integrate them into workflow, without slowing down the most commonly-used situations.
1 Like