I wanted to create this topic as a registry, listing all of the prominent options for automating UI testing in Unity. I want to give people a high level comparison of what each package offers. People may not realize that there are now many choices to automate UI testing in Unity, along with additional alternative options for unit testing!
Full disclosure! I created and maintain the Trilleon framework mentioned below. I am also a contributor to the Automated QA package.
Here are some categories I will mark with a âś“ or an X if the frameworks provide them:
Automation Testing
Unit Testing
Full Framework (Meaning that it fully replicates something like Selenium, instead of extending an existing framework, or representing a shell of one to build off of.)
CI Integration
External Driver
Report Generation
Parallel Execution
Custom Editor Window(s)
– In alphabetical order –
AltTester
It’s an end-to-end UI test automation tool allowing you to run tests on real devices (mobile, PCs, etc.) or inside the Unity Editor. It supports tests written in C#, Python and Java.It’ll help you to find elements and get all their properties and also to simulate any kind of device input. The desktop app has recording feature, allowing you to generate tests automatically based on recorded actions.
Automation Testing: âś“
Unit Testing: X
Full Framework: X
CI Integration: âś“
External Driver: âś“ (it can use Appium to interact with the game)
Report Generation: âś“ (test results and reports inside unity editor for C#)
Parallel Execution: âś“
Custom Editor Window(s): âś“
Multiplayer Testing: âś“
website & download link: Tools | Test automation tools for Unity apps and games - AltTester®
github: AltTester · GitHub
wiki: AltTester® — AltTester® Unity SDK 2.1.2 documentation
discord server for questions and support: AltTester® - Game Test Automation
Automated QA (Unity)
[Shelved / Development Paused]
The Automated QA package enables users to record and playback touch or drag interactions with the UI of a Unity Project and optionally use recordings to drive Unity Tests - in the editor, on a local iOS or Android device, or on an Android device managed by Unity in the cloud.
See the full alpha announcement post here .
Automation Testing: âś“ (Record & Playback w/ Customization of generated C# tests)
Unit Testing: âś“X (Full Unit Testing in the associated Unity Test Framework; see below)
Full Framework: âś“X (Links to Unity Test Tools & ML Agents to make larger test ecosystem) (Development in progress)
CI Integration: âś“ (Development in progress)
Cloud & Device Farm: âś“ (Development in progress)
External Driver: âś“X (Uses Appium in Cloud Device Farm test runs to interact with devices)
Report Generation: âś“
Custom Editor Window(s): âś“
Multiplayer Testing: X (On the horizon)
asset store link: Disabled…
github: Disabled…
wiki: Disabled…
product page: Disabled…
Minefield
Not even a wrapper to Unity’s own Test Framework, it is just an extension library and guidelines to help you write pure Unity Test Framework test cases. Its focus is that you should be able to code up a scene navigation test fast enough that you could do it while creating the scene without disrupting your creation flow. It is possible with help from “test beacon” components added to the scene as a metadata and a various yieldable methods to help click on those beacons in the test. Easy installation via Unity Package Manager. Because you are still writing Unity tests, of course you could use it together with something like Performance Testing package to create a performance test of scenes while navigating around.
Automation Testing: âś“
Unit Testing: X
Full Framework: X
CI Integration: X
External Driver: X
Report Generation: X (possible with performance testing package)
Parallel Execution: X
Custom Editor Window(s): X
github: GitHub - 5argon/Minefield: Program a concise navigation play mode test in Unity.
wiki: GitHub - 5argon/Minefield: Program a concise navigation play mode test in Unity. (README.md)
Responsible
Responsible is a reactive asynchronous testing utility primarily designed for, but not limited to be used in high level system tests. It’s inspired by Rx, and uses UniRx under the hood, but can be used without any Rx knowledge. It was designed to run as part of Unity Test Framework execution, but should also work with any other test framework that supports coroutines or observables.
The primary benefits of using Responsible are:
- Detailed output on test failures and timeouts (operation statuses and async “stack traces”)
- Declarative, composable, and reusable test code
- Using Responsible will circumvent a Unity Test Framework bug, where test execution will continue after errors within nested coroutines, sometimes even hiding the first exception.
Automation Testing: âś“
Unit Testing: âś“ (Not the primary use case)
Full Framework: X
CI Integration: X
External Driver: X
Report Generation: X
Parallel Execution: X
Custom Editor Window(s): X
github: GitHub - sbergen/Responsible: Reactive asynchronous automated testing utility for .NET and Unity
wiki: n/a
The Puppetry
The Puppetry gives ability to test a game in Unity Editor and when built and deployed to a device. This is done via additional integration with a device’s driver (like Appium for mobile, SDK for consoles or SDK for VR glasses). It is cross platform framework primarily written in C#.
The Puppetry doesn’t have dependencies from any specific technology so it can be combined with any test runner(Nunit, XUnit, MSTest), any CI system, any reporting tool set, and other required libraries or frameworks.
The main purpose of driver approach is to separate the tests from the application code base, so that you don’t worry about how changing tests will affect the application.
And finally, The Puppetry supports parallel execution of tests right out of the box. So you can run your tests simultaneously on different Unity Editors or builds.
Automation Testing: âś“
Unit Testing: X (Perhaps possible, but not explicitly designed for it)
Full Framework: âś“
CI Integration: âś“__X (Ostensibly capable, but not included)
External Driver: âś“X__ (Custom driver w/ Appium for devices)
Report Generation: X (Could not find sign of reporting. Likely has basic result output.)
Parallel Execution: âś“
Custom Editor Window(s): âś“
github: GitHub - TestUnitLab/Puppetry: Framework for automated UI testing of Unity3d application
wiki: ttps://github.com/GameUnitLab/Puppetry/wiki
Trilleon
A complete, internal automation framework that is entirely self-contained in your game binary. Communicates with CI process (Jenkins, Bamboo, Pipelines) over simple pubsub messaging framework and sockets. Accepts commands to run tests, and then takes care of the rest. Entirely white box testing solution. Adds massive number of additional options for controlling the order that tests run, along with dependencies between tests and test classes. Extensive editor window, “The Nexus”, where you can organize, launch tests, review test results, and customize the framework.
Just as easy to use as a Unit test framework alternative too. Trilleon can handle any Unit test logic that requires the use of Coroutines, or any context specific game code that cannot be tested on compile time!
Automation Testing: âś“
Unit Testing: âś“ (Atypical; not run at compile time.)
Full Framework: âś“
CI Integration: âś“ (Scripts supplied fully for Mac, partially for Windows)
External Driver: âś“__X__ (Uses Appium to load apps onto devices; no further driver usage)
Report Generation: âś“ (Friendly Html/Js/Css report, Standard XML report, In-editor results window)
Parallel Execution: âś“ (Supports through Selenium Grid)
Custom Editor Window(s): âś“
github: GitHub - disruptorbeam/trilleon: Automate all the things.
wiki: Welcome to Trilleon · disruptorbeam/trilleon Wiki · GitHub
Unity Test Framework
(formerly Unity Test Runner)
The built-in way of running unit tests, and scenario-based automation tests. The latter requires test scenes, and does not test the real scene, on devices, as a user would.
Automation Testing: âś“X (Yes & no; it is not true automation tests, but more like a Unit-Integration/Automation hybrid)
Unit Testing: âś“
Full Framework: âś“
CI Integration: âś“ (CLI provided here)
External Driver: X
Report Generation: âś“ (Standard xml report, In-editor results window.)
Parallel Execution: X****âś“ (Not for automation tests)
Custom Editor Window(s): âś“
wiki: About Unity Test Framework | Test Framework | 1.0.18
Unium
Unium is an experimental library for the purposes of facilitating automated testing and tool development for your Unity games.
Automation Testing: âś“
Unit Testing: X
Full Framework: X
CI Integration: âś“__X__ (Ostensibly capable, but not explicitly)
External Driver: X (Facilities use of external drivers. No dependencies on them.)
Report Generation: X (Facilitates integration of existing testing frameworks; ex: mocha, jest, pytest, specflow, etc…)
Parallel Execution: ? (Unknown)
Custom Editor Window(s): âś“
github: GitHub - gwaredd/unium: Automation for Unity games
wiki: n/a
Unity UI Test Automation
This is a basic wrapper for running ui automation tests that works with the Unity Test Framework. The information below describes what this tool offers only, and not what it offers in addition to the Unity Test Framework.
Automation Testing: âś“
Unit Testing: X
Full Framework: X
CI Integration: X
External Driver: X
Report Generation: X
Parallel Execution: X
Custom Editor Window(s): âś“
github: GitHub - taphos/unity-uitest: Unity UI Test Automation Framework
wiki: n/a
If you are an owner of one of these repositories, feel free to provide me an updated elevator pitch for your toolset. Correct me on any mistakes I make. Additionally, if you have an alternative tool to add to this list, let me know as well.
