Trilleon
Built and owned by DisruptorBeam! (www.DisruptorBeam.com)
Further supported by GSN (Game Show Network) Games! (www.worldwinner.com & www.gsn.com)
Download Now: v1.1.4
- Please DO NOT send any reports of issues to DisruptorBeam or GSN. Send them to me, or post them on the Trilleon Github: Issues · disruptorbeam/trilleon · GitHub.
- The minimum requirement to begin working with Trilleon is to insert the Automation folder into your Assets folder, and add “TrilleonAutomation.AutomationMaster.Initialize()” to an appropriate bootstrap method that begins loading your game.
- I will be making some tutorial videos and updating the Trilleon wiki over the near future.
New! (Building) **Trilleon Wiki (**I need to update the wiki with the link to our github, it currently says “coming soon”)
Traditional QA Test Automation Framework AND Unit Test Framework
The Trilleon test automation and unit test framework is a comprehensive library of code built for C# in the Unity engine. Modeled (*1) “spiritually” after Selenium, it is the first comprehensive Unity toolset that allows you to build a complete automation (integration) test framework inside of Unity. Additionally, it allows you to write non-standard unit tests that can test code that cannot be tested at compile time (such as IEnumerators or context-specific logic).
The Short and Sweet List – What do I get with Trilleon?
- On Devices, Test Execution Both Locally and In the Cloud (Mac, Android, IOS, WebGL - Execution via a server using Windows will currently require you to modify the script, developed on MacOS, to work in the Windows shell)
- Plays Your Game As Close As Currently Possible To How Any Real User Would
- Near Plug-and-Play Setup Experience
- Selenium-inspired, although notable differences exist
- Actively Sends Test Results and Performance Metrics So You Can Tune Your Game
- Unit Test/Unit-like Support. (Does Not Run Tests At Compile Time. It Is Effectively a Hybrid of Unit Tests and Integration Automation Tests)
- Record and Playback (Beta state)
- Out-Of-The-Box, Jenkins-Ready Server Framework (Appium/Python/Shell Scripts Supplied)
- Framework Impact On Memory Usage Negligible (Great Even For Mobile Games!)
- Several Editor Windows For Easy Development and Launching Of Tests
- Reactive, Visually-Appealing, Informative Test Reports (Html Reports and Simple JUnit Xml). I have formerly done full stack web engineer work, and have used this experience to make all html and css-based logic as appealing as possible.
- Game-Agnostic Client Code Library (C#)
- Wide Array Of Options That Allow Complete Control Over Test Interactions And Dependencies
- Easy To Strip In Production Build Process, But Lightweight If You Choose Not To
- Approximately ~110 files in framework, 28,000 lines of code, and with a footprint of 8.5mb (not including third party code)
- Abstraction For Easy Integration Of Developer-Specific Code (Ex: TextMeshProUGUI?)
- Auto-Generates Test Function Stubs For Interacting With Selected Hierarchy Objects (Soft-disabled in 1.0.0 to receive extensive refactor and upgrades)
- IL2CPP and Mono2x Ready
- Does not work with Web Player platform (No longer supported by Unity)
- Unity 5.x+ Tested; Likely Works With No Or Minimal Modifications 4.x+
- Works In Both .Net 2.0 and .Net 2.0 (Subset)
- Considerate Of Earlier Unity Version Foibles (ex. No ForEach Usage - fixed in Unity 5.5p3+)
- FREE and OPEN SOURCE! The entire core and extension libraries are fully available for your edification. Make any changes you like, anywhere. Submit changes you think others will benefit from in the GitHub repository.
- Trilleon is designed as a basis for automation in all types of games. But you can design and add your own Starter Kits (must be free and open source), such as modified driver classes that are highly specialized for specific genres of games, such as FPS Shooters, Turn-based Strategy RPG’s, or Mobile Scrollers.
- Currently Supports Single Player Or Turn-Based Multiplayer Testing (Multiplayer not recently used on my end due to lacking a multiplayer app, and thus this may need several small updates to work properly)
- You can finally Unit Test your Coroutines, and anything else that cannot be validated at compile time.
- Offers two default options for server ↔ client communications:
----- (default) Can use a simple python ↔ c# socket and threading strategy as a simple pubsub service.
----- Can use a free implementation of ©Pubnub as a pubsub service for communicating between a server and client. ©Pubnub is arguably the best pubsub service available for Unity, and has supported the process Trilleon uses to communicate between Jenkins and Unity client applications. Simple, initial implementation is free, and can be scaled to a paid model as needed.
----- However, you can implement any service you already use, easily, using the provided wrapper classes.
What Are My Options?
Unity Test Tools is the only other full QA test integration framework available to Unity developers. However, it does not provide a substantial footing for development of traditional test automation suites. Unity Test Tools primarily allows you to build scenarios, and to test those scenarios around situations that approximate the game as a user would experience it. The ultimate result becomes an amalgamation of unit testing and automation testing. Additionally, the ability to run such automation outside of the Unity Editor (on devices, and playable builds), as part of an automatic build process, is limited or non-existent.
For years, people have tried developing traditional automation frameworks for their video games that can interact with any game using the same engine. Some have resorted to broadcasting xy coordinates to an external application through a debug log. The external test driver would then parse, interpret, and click the specified coordinates for an intended target. In this way, your automation only has insight into information you explicitly tell the game code to broadcast; setting such a framework up demands that the game be built around this requirement every step of the way. This isn’t a timely, developer-friendly, or scalable approach. Not to mention that heavy use of logging can have a satistically-significant effect on performance.
Other people have built frameworks around image recognition; programmatically comparing baseline images to what the game is rendering to find coordinates of buttons, or confirm text on screen. This process is incredibly unreliable. Your baseline images must be replaced whenever the interface elements change. These baselines must account for aspect ratio, device resolution, and asset quality variants. Anything dynamic or unexpected will throw off the process. Tests developed in this way cannot be relied on as part of a build process.
Enter Trilleon: With the new Disruptor Beam® framework, you will find the tried and true automation approaches of browser applications and standard native applications have been ported into the world of video game development.
Why Should I Use Trilleon?
Trilleon is modeled syntactically after Selenium, offering a multitude of tools built to interact with every aspect of your game, while remaining independent of the actual game code (with the exception of Page/Test Objects that wrap game code interactions in a single, organized place).
Selenium relies heavily on open source support to develop and maintain driver executables. This means that an additional application needs to be run to broker interaction between a test suite and most browsers or applications.
In web and native application automation, this external driver has access to a DOM consisting of XML formatted in child-parent hierarchical relationships. This simple, but highly-ordered structure is easy for an external process to read through and find the object that it needs to interact with. Both the application and driver develop a mutual understanding of what object is being referenced, and the driver in turn is delivered the information needed to simulate an interaction. Video games are a completely different animal. The volatile and mutative nature of a video game is not conducive to a DOM. Therefore is no easy way to retrieve information on game objects from outside the game. Indeed, if there were, it could be a security hazard.
Because of this “middle man” driver in web and native app automation, it becomes difficult to completely rely on the results of your automation, such that you can integrate your automation suite into a Jenkins build step that marks the build as unstable if any test failures occur. What this ultimately means is that, in very large automation suites, it is almost guaranteed that one or more tests will fail simply because the driver encountered a transient error that is out of your control. Because of this, gated builds relying on automation results are never as reliable as unit tests are.
This is a liability that Trilleon eliminates; there is no true driver for interaction between tests and the game (Although appium is used within Jenkins CI to load and launch the game, and dismiss device level alerts. But it does not execute any test code). With Trilleon, your game tests itself. The only thing that prevents you from investing full confidence in the results of your tests is your confidence in the quality of the test code (and how it interacts with your game).
Trilleon provides a diverse selection of test-manipulation tools; some of which even Selenium does not offer. An example of this supra-Selenium functionality is the Dependency Architecture system. With this option, you can map relationships between tests throughout your framework, or exert complete control over the order tests run within the context of the entire suite, linked test classes, or individual test classes. The framework has several layers of validation that prevents and eliminates circular dependencies; supporting you as you scale your framework from a few dozen to a few hundred tests. You can even view an interactive, graphical representation of these dependencies in the form of a custom editor window. Another functionality example is the BuddySystem Architecture which allows you to test game code that relies on more than one client (multiplayer [Turn-based currently supported, and real time support is a WIP]). More information on these topics can be found below.
Trilleon offers easy setup with cloud device farm integration for your automation suite. The code you will need is all there; simply plug it in and customize as needed. In no time, you will be ready to run your tests in TestDroid, AWS Device Farm, Xamarin, or any other provider that offers appium-python integration options. Just as important, full instructions/code are provided to set up launching of test runs on local devices through a Jenkins build step.
There are several Unity Editor Window tools included in Trilleon. For example, there’s the “Assistant” tool which allows inspection of elements in the hierarchy window of the editor. This displays all automation-relevant details, including links to the objects and scripts that reference it, and allows you to choose an action or assertion before generating code stubs that can be pasted into your tests. This code accounts for references to the selected game object, and provides a direct link to that reference. In cases where no references are found, a generic search script will use the finder functionality to locate it based on the requested attribute.
Additionally, a Record and Playback tool has just been developed that catches automation-relevant actions taken in your editor, generating a full test script that you can paste into a C# file and immediately playback. In this manner, no manual test-writing will be required to make automation tests. Therefore people of any technical skill level can generate complete, flexible, and reliable automation test scripts.
Try it out, and decide for yourself if you are ready to take advantage of the contribution to quality that a scalable automation test suite provides.
Here is the original trailer demo (more videos to come shortly):
Important Info For Initial Release!
Note: Trilleon is in Beta. It is not available yet from the Asset Store. It is initially only available on Github @ Releases · disruptorbeam/trilleon · GitHub. Nothing can be guaranteed flawless under all scenarios, so please help me make it better by reporting issues to me here or on our Github page. I will react to all reported problems as quickly as possible. Documentation is far from complete, so feel free to ask me how to do things not defined in Welcome to Trilleon · disruptorbeam/trilleon Wiki · GitHub. Please note that I have added “CUSTOMIZE:” comments throughout the code that is prime for customization and extension. Much of what is not yet spelled out on the wiki will have some helpful information in these comments to allow you to explore all of the features in Trilleon.
Things that are advertised as part of Trilleon that need special consideration on their current status:
-
Trilleon was primarly developed on MacOS. Because of this, the server scripts were developed for MacOS, and will not work out of the box for a Windows machine! I will get access to the resources necessary to set this up on a Windows server machine, but you will need to make modifications in the mean time. The main modifications will need to center around shell script commands that differ in name and usage between the two operating systems. Also, some of the visual aspects of the Nexus editor window need to be updated to look more appealing on Windows. For example, font sizes need to be different for icons between the two operating systems. I will work on updating that for the next release.
-
Record And Playback: (It Works!) I spent a lot of time trying to make this tool as easy to use, helpful, and functional under as wide an array of game situations as possible. It has worked well for me in my testing, but I do not have a massive library of games to test the tool out on. Given the nature of this tool, I expect it to have some hiccups in situations that I could not test it under, or in contexts that I did not consider. So if you encounter any problems, please let me know and I will be happy to improve the tool! It is also very important to note that if you want this feature to work at all, you will likely need to customize the fields in GameMaster.cs between the region defined as “#region Customize With Company And Project-specific Interactable Components”. This is because Trilleon, by default, only accounts for objects that Unity provides in UnityEngine, such as Button, InputField, Toggle, etc. Any custom scripts that you use to create interactable objects in your game must be explicitly declared in these fields. Once that is done, AutomationListener components will be applied to these objects, allowing for Record and Playback to detect them.
-
Code Stub Generation: (Disabled!) This feature was one of the first support tools that I created for Trilleon. The code is some of the oldest within Trilleon, and due to a major set of updates around Record and Playback, the shared logic involved needs to be refactored and fleshed out further to be fully functional. Beacuse of this, I disabled the code. The code is still there, and can be manually activated, but the resulting dynamic code will not be compile-ready. I have delayed revisiting this because I did not consider it as high of a priority as polishing the Record And Playback tool, along with other needed features. It is on my short list to introduce in the next minor version of the framework.
-
Multiplayer Testing in Trilleon; aka Buddy System: (Exists/Has Worked/Status Currently Unknown) This framework has been utilized and worked well in the past. However, due to my personal situation, I lost access to source code for games that require or utilize this type of gameplay. Thus, I have not tested the feature in many months. With massive updates to the framework, I expect the Buddy System to encounter a few hurdles for those first implementing it into their suite of Trilleon automation tests. Please let me know whatever issues you encounter, workarounds you implement, etc. I will fix whatever I can immediately, and I will endevour to get ahold of applications that I can implement the Buddy System on, so that I can handle and resolve issues myself.
*1 “Spiritually” Framework was inspired by Selenium, but certainly is not related to it, and differs from it in many ways. Certain approaches to wrapping and normalizing common interactions with apps were mimicked as closely as possible to generate an experience that was not so different from Selenium that it is a whole new experience understanding the syntax and approach to automation.
