Please note, Automated QA Development is On Hold. Please click here for more information.
The Unity DevOps Product Team
The Automated QA package enables users to record and playback touch or drag interactions with the UI of a Unity Project and optionally use recordings to drive Unity Tests - in the editor, on a local iOS or Android device, or on an iOS or Android device managed by Unity in the cloud (email us at AutomatedQA@unity3d.com for access to this feature).
Key Features
- Object based recording: record once in the editor and play back on many different devices and aspect ratios
- Recorded playbacks are stored as time stamped sequences of objects identified by name and tags
- Composite recorded playbacks: create modular recordings to minimize duplication and automatically keep recorded playbacks up to date when a component is updated
- Automatic recorded test generation: create Unity Test Framework Tests from recorded playbacks with the click of one button
- Run tests from your CI pipeline: recorded tests can be run on your build machine, like any other Unity Test, or use the cloud on-device testing feature to run tests on our cloud Android device farm invoked from your build pipeline.
delicioussamedeermouse
Recorded Playback Creation â Unity Test Execution
Limitations
- This is an experimental package with volatile features. New versions may include breaking changes.
- Unity 2019.4 or above required
- Recording is currently limited to UI elements implemented with Unity UI
Installation
- With your project open in the Unity Editor, open the Package Manager (Window > Package Manager).
- Press the plus button (+) in the top left of Package Manager and then select âAdd package from git URLâŚâ.
- Enter com.unity.automated-testing in the text box and then press âAddâ.
See our package documentation for getting started instructions and more information. Please email us at AutomatedTesting@unity3d.com with any questions!
Changelog
[Development On Hold] - 2021-12-10
[0.7.0] - 2021-10-27
Major Features and Improvements
-
Added RecordableInput class that can be used as a drop in replacement for the Input class that works with recorded playback. Currently mouse clicks, touch, keyboard keys, and joystick buttons are supported. Pass-through methods are provided for the remaining functionality for ease of use.
-
Added scriptable object CloudTestDeviceInput that lets a user specify specific devices on which to execute their tests in the cloud.
-
Added support to retrieve HTML reports for cloud device runs
-
Added new report to aggregate display of results for multi-device cloud runs, with links to each individual deviceâs run results in an existing HTML report.
-
Added Heap size performance sampling to reporting charts found in the existing test report.
-
Added Page Objects to test generation tool.
-
Every GameObject that code interacts with has its query string locator stored in a single Page Object class, referenced by any number of tests.
-
Page Object properties are organized in different Page Object classes based on the scene they are located in.
-
When generating tests, the tool will see if an existing Page Object property exists. It will only add a property to a class if no reference exists.
-
All menu options moved to a single âAutomated QA Hubâ window
-
Moving Cloud Testing behind a feature flag temporarily as we redesign the UX. Please email us at AutomatedQA@unity3d.com for early access.
Bug Fixes and Minor Changes
- Added support for custom report data.
- Fixed error when generating full test from a recording that starts with a number.
- Added option to generate simpler code from the Test Generator by utilizing the Driver logicâs new query string approach to finding GameObjects.
- Rewrote Recorded Playback editor window in new UIElement logic.
- Redesigned look of window.
- Added GameCrawler âCrawlâ button to window, which executes a default crawl and generates a recording json file.
- Added tracking for the object index in the hierarchy to help differentiate objects with the same name
- Added fix to ensure automation waits for a GameObject animating from off screen to finish moving into the camera frustum before interacting with it.
- Added enforcement of an en-US culture in CodeGenerator logic.
- Added a context menu with the ability to generate tests in the main recorded playback window.
- Added a script for creating and managing service accounts to use in CI workflows.
- Fixed an issue where spaces in recording json file names were used when generating a class name in the Test Generator.
- Fixed an issue that causes backslashes to be used in generated code for Staged Runs.
- Added âTryPerformâ to Driver class. If an action cannot be performed, the test will not fail.
- Added drop downs for builds and jobs in cloud test window
- Added GameElement âAutoIDâ - an automatically-generated GUID to uniquely identify GameElements.
- Adds GameElement script to all interactable GameObjects and assigns a unique GUID automatically.
- Allows for GameObject to be renamed, moved, etc. without breaking a test that references this ID using Driver.cs.
- Added menu option to automatically add GameElements to interactable UI Elements in a scene. Menu â âAutomated QAâ â âAdd Game Elements To Scene Objectsâ
- Renamed AutomatedQAEditorSettings to AutomatedQABuildConfig and AutomatedQARuntimeSettings to AutomatedQASettings.
- Fixed an issue generating iOS cloud testing builds when there are spaces in the build name.
Breaking Changes
- Removed âFull Buildâ cloud testing support.
[0.6.1] - 2021-08-12
Major Features and Improvements
- New GameElement and ElementQuery classes for reliably finding GameObjects, no matter where they are moved in a scene.
- Uses Xpath & JQuery-like query string selectors for identifying GameObjects.
- New Driver class that allows for much simpler ways to write custom code and perform actions on objects (ex: Driver.Perform.Click(â#SubmitButtonâ)).
- Added GameCrawler that allows users to create an AutomatedRun that plays randomly through a game, recording warnings, errors, and notifying users when and where it gets stuck.
Bug Fixes and Minor Changes
- Reverting change that removed the ability to generate âSimple Testsâ, which is a test that points to a recording file.
- Fixed an issue with temp directories not being properly deleted.
- Adding FPS tracking to reports. FPS sampled over duration of test execution is displayed in a graph accessible from the html report (FPS Data button).
- Fixed issue allowing objects off screen to be interacted with.
- Add support for recording right and middle mouse clicks.
- Added functionality to allow both recording and playback while editor is in play mode.
- Both record & playback can be done repeatedly in the same session.
- Editor does not stop playmode on completion, but both record & playback can still be started outside of playmode.
- Added support for running cloud tests from the command line using BuildAndRunTests.
- Fixed issue where tests may timeout during command line batch mode runs.
- Fixed issue with the InputField text listener not being added right away.
- Screenshot timing after each playback action completes is now configurable in settings.
- Exposed âQuit on finishâ in Automated Run Config.
- Added ability to generate a âFullâ test from Automated Runs.
- Added settings for log level
- Added support for TextMeshPro TMP_InputFields and TMP_Text fields
- Can record and playback input into TMP_InputFields, and find objects in ElementQuery by TMP_InputField & TMP_Text text content.
Breaking Changes
- Several APIs were moved from the test base class to Driver.cs. âFullâ generated tests from v0.5.0 will need their PerformAction and RegisterStep method invocations updated.
- The experimental Composite Recordings window has been removed. Instead, Automated Runs can be used to playback multiple recordings in sequence.
[0.5.0] - 2021-06-28
Major Features and Improvements
-
Support for running tests on real iOS devices in the cloud
-
New [CloudTest] attribute can be used to specify any Unity Test Framework test to run on real devices in the cloud
-
Automated QA > Test Generation⌠allows for step-by-step Unity Test Framework test generation (C# code).
-
Generates code for every step/action taken in a recording.
-
Allows for assertions or additional custom logic between each step.
-
Can select which recordings to generate tests for.
-
Editor Window warns user if about to overwrite custom edits in a recordingâs test when re-generating test.
Breaking changes
-
Recording file in RecordedPlaybackAutomator is now an asset reference instead of path string (Note: Only relevant if youâre using the experimental Automated Runs feature)
-
Migration: Update the file path in AutomatedRuns using RecordedPlaybackAutomator
Bug Fixes and Minor Changes
- Added dynamic wait logic so that differences in load times or animations does not result in automation failing to interact with the target object.
- Delayed loading of the recorded playback controller to avoid issues with initializing too early.
[0.4.0] - 2021-05-25
Major Features and Improvements
-
HTML & XML reports for AutomatedQA runs.
-
XML report is designed for loading tests results into a CI process run.
-
Latest HTML reportcan be opened in editor from the âRecorded Playbackâ and âComposite Recordingsâ editor windows after playback.
-
Both currently stored in Application.persistentDataPath. CI/CD must copy report from path (Will be integrated with Cloud services & modified to take raw test data from devices so that explicit file extraction is not necessary).
-
Added settings file for editable values used in Automated Qa package, thus making various Automated QA behaviors customizable (more to be added as time goes on).
-
Added new editor window to edit the settings file (Automated QA > Settings).
-
Multiple config files with varying values can be stored in Assets/AutomatedQA/Resources, and desired settings can be requested from cloud config via file name.
Bug Fixes and Minor Changes
- Added placeholder segment file names in Composite Recording window when selecting âSave Segmentâ.
- Updated VisualFx to prevent visual clutter from a rapid series of events.
- Updated validation of GameObject that is about to be clicked. Increased performance. Now checks that an object is off screen or under another object that would intercept the click.
- Updated recorded tests to load an empty scene after execution.
- Fixed ignoring depth surface warnings when taking screenshots on Mac.
- Added new Automators: Scene Automator and Text Input Automator.
- Fixed event timings and scene loading with composite recordings
[0.3.1] - 2021-05-04
Major Features and Improvements
-
New AutomatedRun object to link together recordings and custom C# scripts to automate gameplay.
-
Create an AutomatedRun with Create â Automated QA â Automated Run
-
New Automator class. Extend this class to create custom automators
-
Extend the AutomatorConfig class for your Automator to expose it in the AutomatedRun inspector.
Bug Fixes and Minor Changes
- Fixed an issue where drop events had extra delay
- Fixed an issue with the Upload Recording window requiring entitlements
- Removed dependency on com.unity.nuget.newtonsoft-json package
- Fixed entry scene to work with initialization scenes
- Moved menu items to top level Automated QA menu
- Added fix to CloudTestResults.cs that prevents ITestRunCallback from being eagerly stripped by the âahead of timeâ compilation in IL2CPP builds.
- Added logic to cleanup temporary files accumulating from previous recordings and stored in the persistent data path.
- Removed Linq usage from package as Linq can be a heavy library for mobile game development.
[0.3.0] - 2021-04-21
Breaking changes
-
Simplified assembly definitions
-
Migration: Please update asmdefs to reference Unity.AutomatedQA
-
Renamed asset directory to AutomatedQA
-
Migration: Please delete the old AutomatedTesting folder
Bug Fixes and Minor Changes
- Package directory restructure
- Added SettingsManager package and AutomatedQAEditorSettings/AutomatedQARuntimeSettings to wrap package settings
- Disabled Cloud Testing window for unsupported platforms
- Fixed an issue with recorded tests using composite recordings
- Fixed an issue causing two simultaneously active EventSystem components, resulting in a flood of console warnings
- Fixed an issue where multi-clicking âRecordâ or âPlayâ buttons creates multiple RecordedPlaybackController GameObjects