With Unity 2019.2, the Unity Test Framework (aka Unity Test Runner) will be a package. As a part of it, we’ve given the documentation a very much needed overhaul.
The latest version of the documentation is available here: Redirecting to latest version of com.unity.test-framework?
Any feedback and questions to the documentation are welcome. We will be keeping the documentation up to date as we release new versions of the package.
Beautiful; a hell of an improvement over the current documentation.
One request; can you describe the order in which external actions within the same category run with respect to each other? E.g. in which order will implementations of OuterUnityTestAction run with respect to one another? Or implementations of IWrapSetUpTearDown?
EDIT: And maybe while you’re at it, why would I want to use certain stages over others? Like, why would I want to use OuterUnityTestAction over IWrapSetUpTearDown/IWrapTestMethod, or vice versa?
Thanks.
In regards to the order of each set of actions (e.g. OuterUnityTestAction), then they are run in the same order as other nunit attribute, which again is the order that System.Reflection provide the attributes. It is however not documented.
In my experience, then the order seems to be first it takes the attributes in the order they are written, staring with the base most class, working upwards.
Regarding OuterUnityTestAction vs IWrapSetUpTearDown/IWrapTestMethod, then they are a lot alike, but the need for the OuterUnityTestAction arose for performance measurements, which would preferably be at the outer most scope. Hence this can be used as the outer most action.
Thanks for updating the docs. here is some feedback from me (BEWARE!!! nitpicking ahead! also this is going to be very long).
I am not using .asmdef but still, my editor code (which contains my editor tests) reference my “runtime” scripts. This phrasing (to me at least) made it sound as if you can only access editor code in these tests.
We have a lot of code that is not MonoBehaviour based but still used at runtime in our game. This makes it perfectly suitable to be run as an edit mode test, so it’s not really only for editor extensions.
Not sure why this would be needed, but sounds like a bit of a hack isn’t it? creating an edit mode test only to let it go into play mode.
The page later mentions how to set up playmode vs. edit mode tests.
For edit mode:
And for play mode:
Should probably describe both cases in a similar way to avoid any confusion.
This section is not clear to me (sorry!)
https://docs.unity3d.com/Packages/com.unity.test-framework@1.1/manual/workflow-create-test.html
This is not a category… NUnit allows you to tag tests using the Category attribute. What’s shown on the documentation page is the actual class name containing different tests. using this attribute has no effect for me.
The categories are shown on the top right of the window:
https://docs.unity3d.com/Packages/com.unity.test-framework@1.1/manual/workflow-run-playmode-test-standalone.html
“Running as standalone” is a bit confusing, since it sounds like it’s only possible to run on a standalone (e.g: Desktop) build.
The UI itself calls it “Run in player” so i think the docs should be aligned with that.
TestPlayerBuildModifier
(not docs related, sorry…) First of all, why do we need this attribute? In order to modify the BuildPlayerOptions we already had this option available for a while.
Also - this method of setting an attribute to point to a given type is a bit cumbersome. why is it designed that way? if my implementation needs to implement a certain interface, i suppose you can find that out via reflection and instantiate it.
“Modify the Player build options for Play Mode tests” – why is this needed? why does the location of the build player matters when executing PlayMode tests?
Split build and run for Play Mode tests
The code example is not very clear:
These are my comments so far. again, thank you for creating such a detailed guide with examples and API references. great step indeed 
Hey. Thanks for your thorough feedback, it is much appreciated.
I have applied changes as part of it and it should be part of the next version of the package.
Regarding the split build and run:
Hi, I try to start the Test runs with a script like described here:
https://docs.unity3d.com/Packages/com.unity.test-framework@1.1/manual/extension-run-tests.html
using UnityEditor.TestTools.TestRunner.Api;
using UnityEngine;
namespace Editor
{
public static class IntegrationTests
{
public static void StartTests()
{
Debug.Log("Game4Automation Tests started");
var testRunnerApi = ScriptableObject.CreateInstance<TestRunnerApi>();
But I am getting told that the Internal Class TestRunnerApi cannot be accessed. How could I start the tests with code=
Best regards
Thomas
This means you’re using an older version of the com.unity.test-framework package. You need to upgrade to a newer version through the Package Manager.
Hmmm…
I am using Unity 2018.4.8f1 and the Test-Framkework in the Package manager seems also to be updated (package manager just gives me 0.0.4)
Any advice?
Best regards
Thomas
Does your manifest.json file have your project using a custom registry, by any chance?
This is my manifest.json (before installing Test-Framework):
{
"dependencies": {
"com.unity.ads": "2.0.8",
"com.unity.analytics": "3.2.2",
"com.unity.collab-proxy": "1.2.15",
"com.unity.package-manager-ui": "2.0.7",
"com.unity.purchasing": "2.0.3",
"com.unity.textmeshpro": "1.4.1",
"com.unity.modules.ai": "1.0.0",
"com.unity.modules.animation": "1.0.0",
"com.unity.modules.assetbundle": "1.0.0",
"com.unity.modules.audio": "1.0.0",
"com.unity.modules.cloth": "1.0.0",
"com.unity.modules.director": "1.0.0",
"com.unity.modules.imageconversion": "1.0.0",
"com.unity.modules.imgui": "1.0.0",
"com.unity.modules.jsonserialize": "1.0.0",
"com.unity.modules.particlesystem": "1.0.0",
"com.unity.modules.physics": "1.0.0",
"com.unity.modules.physics2d": "1.0.0",
"com.unity.modules.screencapture": "1.0.0",
"com.unity.modules.terrain": "1.0.0",
"com.unity.modules.terrainphysics": "1.0.0",
"com.unity.modules.tilemap": "1.0.0",
"com.unity.modules.ui": "1.0.0",
"com.unity.modules.uielements": "1.0.0",
"com.unity.modules.umbra": "1.0.0",
"com.unity.modules.unityanalytics": "1.0.0",
"com.unity.modules.unitywebrequest": "1.0.0",
"com.unity.modules.unitywebrequestassetbundle": "1.0.0",
"com.unity.modules.unitywebrequestaudio": "1.0.0",
"com.unity.modules.unitywebrequesttexture": "1.0.0",
"com.unity.modules.unitywebrequestwww": "1.0.0",
"com.unity.modules.vehicles": "1.0.0",
"com.unity.modules.video": "1.0.0",
"com.unity.modules.vr": "1.0.0",
"com.unity.modules.wind": "1.0.0",
"com.unity.modules.xr": "1.0.0"
}
}
Hi, I checked again yesterday. It seems that even in a new empty project with Unity 2018… there is only 0.0.4 available.
With 2019… I was able to download newer versions (1.0,1.1…).
Hey. The unity test framework package is only available with 2019.2 and forward. The 0.0.4 version you see is visible by mistake. It was an intimidate version we had when moved from the build in module, into being a package, so it will not work. In 2018 the test framework is build into unity and you do not need a package. When you later upgrade to 2019.2 or newer, then the test framework will be automatically added.
Ah I see. So the API for the Test framework seems to be only available from 2019.2 upwards. Sorry - documentation wasn’t clear about that. Thanks
The API and the documentation is coming along nicely!
I’ve looked up the TestStatus Enum and didn’t find any recommendations in the Test Framework scripting API about when to use Inconclusive or Skipped as a result. I did find information on the NUnit pages, but maybe we can add examples to the Unity docs for our specific context. For example, I found it helpful to know that it is possible to run a suite of tests to validate a Unity scene and then return Skipped/Ignored for tests which do not apply in very specific circumstances (e.g. no jump platforms present in the scene). Next, when loading a resource from the AssetDatabase that’s no longer at that location or when anything goes wrong that breaks the test but does not indicate an issue with the scene or entity to be verified, we can return Inconclusive (like a warning to the user, but no error).
Additionally, should we be more consistent with the naming between Assert.Ignore and TestStatus.Skipped? Or are these different because of NUnit reasons?
Hi. I have asked for clarification on the UNITY_INCLUDE_TESTS here: Info on UNITY_INCLUDE_TESTS define . Maybe this is a better place to ask - could you shed some light on that define and if it could be used as described in the linked post? Thanks.
Hey, I just started using the Unity Testing Framework. I understood that all tested code has to be placed in its own Assembly for making the tests to work. You can’t just test code that “lives” in the default assembly, correct?
However, in a “real world” project that uses a lot of external assets and libraries, this turned into a real nightmare, because as soon as my own code is placed in an assembly, it can’t access these 3rd party libraries anymore, so I have to define even more assemblies for them. Which sometimes doesn’t seem to be possible at all. For instance, the Facebook SDK doesn’t use an assembly by default, but as soon as I place it in one, it won’t be able to access code in its own plugins, although the docs state that all plugin libraries are auto-referenced by assembly defs.
Thus, every time I add a new assembly def, I get masses of (new) compile errors. And this continues like forever. Tbh, I am completely lost now. I am wondering why there isn’t an easier way to define tests that can directly access all scripts that are already available in the project?
I totally get that assemblies allow to exclude these tests from a build, but couldn’t it just be done in a similar way that editor folder scripts are excluded from the build, without all that “assembly hell”? I am about to give up now, and I would consider myself as a pretty experienced developer…
Also, the docs only describe basic Unit testing. But how to test a Monobehavior in the context of its scene where it belongs to, i.e. all the other Monobehaviors that are placed on the same game object? I mean actual UX tests, not just isolated units.
You can have tests mix with your regular code without issues, I just tested this although I wouldn’t recommend it. The downside would be that you would also be shipping your test code with the final build.
Did you keep in mind that you can reference DLLs directly without having to create ASMDEF? But other than that, yea 3rd party is a nightmare, but that’s not much worse in Unity than it is in other environments. The issue with many Unity plugins is that they haven’t upgraded to ASMDEF although they should. So, as a developer, you have to fix this by hand and it sucks, but I do it all the time. It’s not that bad if you just do it and then look back and say “well ok that was an hour of work, but now the entire project works again, then next plugin we add will only take 10 minutes to setup”.
Don’t! I was at that point with assembly hell, but then just went through it and know it doesn’t bother me anymore. In every project I set up three main ASMDEF: Editor, Runtime and Tests. Editor references Runtime. Tests references Runtime. Then I add 3rd party plugins into a non-magic “Plugins” folder with a Plugins ASMDEF. Runtime references the Plugins ASMDEF. This covers 90% of all plugins/cases. Basically, they are all bundles up into one assembly, unless they define their own ASMDEF or have editor code. If they already have ASMDEF, well just reference the ones you need (if this gets out of hand, think about using less 3rd party code, it might bite you in many other ways). If the plugins haven’t set up ASMDEF, but use the magig “Editor” folder, bite the bullet and create the ASMDEFs for the plugin. Usually, this just means creating two. And remember for everything else, if they use DLL you can simply reference this in your runtime ASMDEF via the dropdown. And finally, the separate test assembly is a proven idea. You simply reference runtime, because everything else should not be tested. Don’t write tests that include the Facebook SDK, they ought to test their stuff on their own, etc.
Hi! The documentation says that Unity Test Framework has NUnit 3.5.
And apparently NUnit 3.5 has support for tasks.
But this test fails with the message “Method has non-void return value, but no result is expected”:
[Test]
public async Task MyTest()
{
var result = await AsyncOperation();
Assert.AreEqual(10, result);
}
private async Task<int> AsyncOperation()
{
await Task.Yield();
return 10;
}
Why is that?
You encourage the use of Tasks, but I couldn’t find any documentation to test it properly.
Is there any tutorial for dumbs that show how can I do a test in a build? so I build it, and it tests…
The UTF docs demonstrate how to do this: https://docs.unity3d.com/Packages/com.unity.test-framework@1.1/manual/workflow-run-playmode-test-standalone.html
Hope that’s what you wanted.