Trilleon - Full Integration Automation and Unit Test Framework

Community Showcases
1

Trilleon
Built and owned by DisruptorBeam! (www.DisruptorBeam.com)
Further supported by GSN (Game Show Network) Games! (www.worldwinner.com & www.gsn.com)

Download Now: v1.1.15

  • Please DO NOT send any reports of issues to DisruptorBeam or GSN. Send them to me, or post them on the Trilleon Github: Issues · disruptorbeam/trilleon · GitHub.
  • The minimum requirement to begin working with Trilleon is to insert the Automation folder into your Assets folder, and add “TrilleonAutomation.AutomationMaster.Initialize()” to an appropriate bootstrap method that begins loading your game.
  • I will be making some tutorial videos and updating the Trilleon wiki over the near future.

New! (Building) **Trilleon Wiki (**I need to update the wiki with the link to our github, it currently says “coming soon”)

Traditional QA Test Automation Framework AND Unit Test Framework

The Trilleon test automation and unit test framework is a comprehensive library of code built for C# in the Unity engine. Modeled (*1) “spiritually” after Selenium, it is the first comprehensive Unity toolset that allows you to build a complete automation (integration) test framework inside of Unity. Additionally, it allows you to write non-standard unit tests that can test code that cannot be tested at compile time (such as IEnumerators or context-specific logic).

The Short and Sweet List – What do I get with Trilleon?

  • On Devices, Test Execution Both Locally and In the Cloud (Mac, Android, IOS, WebGL - Execution via a server using Windows will currently require you to modify the script, developed on MacOS, to work in the Windows shell)
  • Plays Your Game As Close As Currently Possible To How Any Real User Would
  • Near Plug-and-Play Setup Experience
  • Selenium-inspired, although notable differences exist
  • Actively Sends Test Results and Performance Metrics So You Can Tune Your Game
  • Unit Test/Unit-like Support. (Does Not Run Tests At Compile Time. It Is Effectively a Hybrid of Unit Tests and Integration Automation Tests)
  • Record and Playback (Beta state)
  • Out-Of-The-Box, Jenkins-Ready Server Framework (Appium/Python/Shell Scripts Supplied)
  • Framework Impact On Memory Usage Negligible (Great Even For Mobile Games!)
  • Several Editor Windows For Easy Development and Launching Of Tests
  • Reactive, Visually-Appealing, Informative Test Reports (Html Reports and Simple JUnit Xml). I have formerly done full stack web engineer work, and have used this experience to make all html and css-based logic as appealing as possible.
  • Game-Agnostic Client Code Library (C#)
  • Wide Array Of Options That Allow Complete Control Over Test Interactions And Dependencies
  • Easy To Strip In Production Build Process, But Lightweight If You Choose Not To
  • Approximately ~110 files in framework, 28,000 lines of code, and with a footprint of 8.5mb (not including third party code)
  • Abstraction For Easy Integration Of Developer-Specific Code (Ex: TextMeshProUGUI?)
  • Auto-Generates Test Function Stubs For Interacting With Selected Hierarchy Objects (Soft-disabled in 1.0.0 to receive extensive refactor and upgrades)
  • IL2CPP and Mono2x Ready
  • Does not work with Web Player platform (No longer supported by Unity)
  • Unity 5.x+ Tested; Likely Works With No Or Minimal Modifications 4.x+
  • Works In Both .Net 2.0 and .Net 2.0 (Subset)
  • Considerate Of Earlier Unity Version Foibles (ex. No ForEach Usage - fixed in Unity 5.5p3+)
  • FREE and OPEN SOURCE! The entire core and extension libraries are fully available for your edification. Make any changes you like, anywhere. Submit changes you think others will benefit from in the GitHub repository.
  • Trilleon is designed as a basis for automation in all types of games. But you can design and add your own Starter Kits (must be free and open source), such as modified driver classes that are highly specialized for specific genres of games, such as FPS Shooters, Turn-based Strategy RPG’s, or Mobile Scrollers.
  • Currently Supports Single Player Or Turn-Based Multiplayer Testing (Multiplayer not recently used on my end due to lacking a multiplayer app, and thus this may need several small updates to work properly)
  • You can finally Unit Test your Coroutines, and anything else that cannot be validated at compile time.
  • Offers two default options for server ↔ client communications:
    ----- (default) Can use a simple python ↔ c# socket and threading strategy as a simple pubsub service.
    ----- Can use a free implementation of ©Pubnub as a pubsub service for communicating between a server and client. ©Pubnub is arguably the best pubsub service available for Unity, and has supported the process Trilleon uses to communicate between Jenkins and Unity client applications. Simple, initial implementation is free, and can be scaled to a paid model as needed.
    ----- However, you can implement any service you already use, easily, using the provided wrapper classes.

What Are My Options?

Unity Test Tools is the only other full QA test integration framework available to Unity developers. However, it does not provide a substantial footing for development of traditional test automation suites. Unity Test Tools primarily allows you to build scenarios, and to test those scenarios around situations that approximate the game as a user would experience it. The ultimate result becomes an amalgamation of unit testing and automation testing. Additionally, the ability to run such automation outside of the Unity Editor (on devices, and playable builds), as part of an automatic build process, is limited or non-existent.

For years, people have tried developing traditional automation frameworks for their video games that can interact with any game using the same engine. Some have resorted to broadcasting xy coordinates to an external application through a debug log. The external test driver would then parse, interpret, and click the specified coordinates for an intended target. In this way, your automation only has insight into information you explicitly tell the game code to broadcast; setting such a framework up demands that the game be built around this requirement every step of the way. This isn’t a timely, developer-friendly, or scalable approach. Not to mention that heavy use of logging can have a satistically-significant effect on performance.

Other people have built frameworks around image recognition; programmatically comparing baseline images to what the game is rendering to find coordinates of buttons, or confirm text on screen. This process is incredibly unreliable. Your baseline images must be replaced whenever the interface elements change. These baselines must account for aspect ratio, device resolution, and asset quality variants. Anything dynamic or unexpected will throw off the process. Tests developed in this way cannot be relied on as part of a build process.

Enter Trilleon: With the new Disruptor Beam® framework, you will find the tried and true automation approaches of browser applications and standard native applications have been ported into the world of video game development.

Why Should I Use Trilleon?

Trilleon is modeled syntactically after Selenium, offering a multitude of tools built to interact with every aspect of your game, while remaining independent of the actual game code (with the exception of Page/Test Objects that wrap game code interactions in a single, organized place).

Selenium relies heavily on open source support to develop and maintain driver executables. This means that an additional application needs to be run to broker interaction between a test suite and most browsers or applications.

In web and native application automation, this external driver has access to a DOM consisting of XML formatted in child-parent hierarchical relationships. This simple, but highly-ordered structure is easy for an external process to read through and find the object that it needs to interact with. Both the application and driver develop a mutual understanding of what object is being referenced, and the driver in turn is delivered the information needed to simulate an interaction. Video games are a completely different animal. The volatile and mutative nature of a video game is not conducive to a DOM. Therefore is no easy way to retrieve information on game objects from outside the game. Indeed, if there were, it could be a security hazard.

Because of this “middle man” driver in web and native app automation, it becomes difficult to completely rely on the results of your automation, such that you can integrate your automation suite into a Jenkins build step that marks the build as unstable if any test failures occur. What this ultimately means is that, in very large automation suites, it is almost guaranteed that one or more tests will fail simply because the driver encountered a transient error that is out of your control. Because of this, gated builds relying on automation results are never as reliable as unit tests are.

This is a liability that Trilleon eliminates; there is no true driver for interaction between tests and the game (Although appium is used within Jenkins CI to load and launch the game, and dismiss device level alerts. But it does not execute any test code). With Trilleon, your game tests itself. The only thing that prevents you from investing full confidence in the results of your tests is your confidence in the quality of the test code (and how it interacts with your game).

Trilleon provides a diverse selection of test-manipulation tools; some of which even Selenium does not offer. An example of this supra-Selenium functionality is the Dependency Architecture system. With this option, you can map relationships between tests throughout your framework, or exert complete control over the order tests run within the context of the entire suite, linked test classes, or individual test classes. The framework has several layers of validation that prevents and eliminates circular dependencies; supporting you as you scale your framework from a few dozen to a few hundred tests. You can even view an interactive, graphical representation of these dependencies in the form of a custom editor window. Another functionality example is the BuddySystem Architecture which allows you to test game code that relies on more than one client (multiplayer [Turn-based currently supported, and real time support is a WIP]). More information on these topics can be found below.

Trilleon offers easy setup with cloud device farm integration for your automation suite. The code you will need is all there; simply plug it in and customize as needed. In no time, you will be ready to run your tests in TestDroid, AWS Device Farm, Xamarin, or any other provider that offers appium-python integration options. Just as important, full instructions/code are provided to set up launching of test runs on local devices through a Jenkins build step.

There are several Unity Editor Window tools included in Trilleon. For example, there’s the “Assistant” tool which allows inspection of elements in the hierarchy window of the editor. This displays all automation-relevant details, including links to the objects and scripts that reference it, and allows you to choose an action or assertion before generating code stubs that can be pasted into your tests. This code accounts for references to the selected game object, and provides a direct link to that reference. In cases where no references are found, a generic search script will use the finder functionality to locate it based on the requested attribute.

Additionally, a Record and Playback tool has just been developed that catches automation-relevant actions taken in your editor, generating a full test script that you can paste into a C# file and immediately playback. In this manner, no manual test-writing will be required to make automation tests. Therefore people of any technical skill level can generate complete, flexible, and reliable automation test scripts.

Try it out, and decide for yourself if you are ready to take advantage of the contribution to quality that a scalable automation test suite provides.

Here is the original trailer demo (more videos to come shortly):

Important Info For Initial Release!
Note: Trilleon is in Beta. It is not available yet from the Asset Store. It is initially only available on Github @ Releases · disruptorbeam/trilleon · GitHub. Nothing can be guaranteed flawless under all scenarios, so please help me make it better by reporting issues to me here or on our Github page. I will react to all reported problems as quickly as possible. Documentation is far from complete, so feel free to ask me how to do things not defined in Welcome to Trilleon · disruptorbeam/trilleon Wiki · GitHub. Please note that I have added “CUSTOMIZE:” comments throughout the code that is prime for customization and extension. Much of what is not yet spelled out on the wiki will have some helpful information in these comments to allow you to explore all of the features in Trilleon.

Things that are advertised as part of Trilleon that need special consideration on their current status:

  1. Trilleon was primarly developed on MacOS. Because of this, the server scripts were developed for MacOS, and will not work out of the box for a Windows machine! I will get access to the resources necessary to set this up on a Windows server machine, but you will need to make modifications in the mean time. The main modifications will need to center around shell script commands that differ in name and usage between the two operating systems. Also, some of the visual aspects of the Nexus editor window need to be updated to look more appealing on Windows. For example, font sizes need to be different for icons between the two operating systems. I will work on updating that for the next release.

  2. Record And Playback: (It Works!) I spent a lot of time trying to make this tool as easy to use, helpful, and functional under as wide an array of game situations as possible. It has worked well for me in my testing, but I do not have a massive library of games to test the tool out on. Given the nature of this tool, I expect it to have some hiccups in situations that I could not test it under, or in contexts that I did not consider. So if you encounter any problems, please let me know and I will be happy to improve the tool! It is also very important to note that if you want this feature to work at all, you will likely need to customize the fields in GameMaster.cs between the region defined as “#region Customize With Company And Project-specific Interactable Components”. This is because Trilleon, by default, only accounts for objects that Unity provides in UnityEngine, such as Button, InputField, Toggle, etc. Any custom scripts that you use to create interactable objects in your game must be explicitly declared in these fields. Once that is done, AutomationListener components will be applied to these objects, allowing for Record and Playback to detect them.

  3. Code Stub Generation: (Disabled!) This feature was one of the first support tools that I created for Trilleon. The code is some of the oldest within Trilleon, and due to a major set of updates around Record and Playback, the shared logic involved needs to be refactored and fleshed out further to be fully functional. Beacuse of this, I disabled the code. The code is still there, and can be manually activated, but the resulting dynamic code will not be compile-ready. I have delayed revisiting this because I did not consider it as high of a priority as polishing the Record And Playback tool, along with other needed features. It is on my short list to introduce in the next minor version of the framework.

  4. Multiplayer Testing in Trilleon; aka Buddy System: (Exists/Has Worked/Status Currently Unknown) This framework has been utilized and worked well in the past. However, due to my personal situation, I lost access to source code for games that require or utilize this type of gameplay. Thus, I have not tested the feature in many months. With massive updates to the framework, I expect the Buddy System to encounter a few hurdles for those first implementing it into their suite of Trilleon automation tests. Please let me know whatever issues you encounter, workarounds you implement, etc. I will fix whatever I can immediately, and I will endevour to get ahold of applications that I can implement the Buddy System on, so that I can handle and resolve issues myself.

*1 “Spiritually” Framework was inspired by Selenium, but certainly is not related to it, and differs from it in many ways. Certain approaches to wrapping and normalizing common interactions with apps were mimicked as closely as possible to generate an experience that was not so different from Selenium that it is a whole new experience understanding the syntax and approach to automation.

6 Likes
2

I want to officially announce the introduction of a Socket connection strategy within Trilleon. Previously (if you have followed Trilleon’s development), Pubnub was the only implementation used to facilitate communication between a server (Jenkins/Bamboo etc?) and the client (iPhone/Android/Desktop etc?).

That is ultimately overkill. Pubnub has been excellent, and if you use Pubnub for your game’s chat features (or more), then it makes sense to plug Trilleon into that. But for those that do not already use Pubnub, it is less than convenient. You must create a free account, and set that up. It remains free if you do not have a massive farm of automation devices, so it is not a monetary hurdle for most. But the Pubnub SDK is very large, and is all required. The SDK takes up valuable room, particularly for mobile games.

Because of this, I have focused on developing a local socket communication framework for Trilleon. It is located in the C# assets under SocketConnectionStrategy.cs. But I have not completed the server code logic just yet. It is close, and will be within the first minor update of Trilleon (v1.1.0). As soon as it is fully functional, I will set this strategy as the default one used by Trilleon, but I will keep Pubnub as an option.

This should save MB’s of data on the dependencies, and offer more options. Which is particularly useful if you intend to keep the core Trilleon SDK in your final production app build. It will also negate the need for an internet connection if your game under test does not require one.

While the server-side code is not yet in place, the client-side code is fully implemented, so feel free to plug in your own solution at any time.

3

New release! Socket communication implemented for talking between server and client. It is an alternative to Pubnub, and also the default setup now.
Download Now: v1.1.1

4

A new Trilleon update has been released that fixes several potential bugs: v1.1.2

Additionally, I have created the first demo video for Trilleon that explains all of the Nexus editor window functionality!

5

Hi Ya’ll,

I just wanted to gauge how many people are using the Trilleon framework, to determine how much of my free time I should be devoting to Trilleon. I was recently laid off again, and I do not expect to be working within Unity in my next position.

The response to Trilleon has been incredibly luke-warm. Before release, it seemed that there was a strong interest in the framework, but since then, I have seen almost zero interest in the Trilleon automation framework. Both in the forum threads and via Github.

I would love to know who is using it, and a quick poke on how it has helped you. If there seems to be a reasonable number of people using it, I will continue to develop for it and improve it.

Thanks to any responses.

Regards,
Tim

6

I’m following the development and would like to use it one day, didn’t have a chance to use it yet.

7

I’d like to spend some time in the near future to upgrade Trilleon. Among those that are invested in Trilleon, can anyone give me a heads up on what they would like to see most. It can be anything from better documentation to a specific feature!

8

Hello, I’m currently looking into Trilleon for a project I’m a part of. Now a couple things to keep in mind. First I’m just freshly graduated, and moved into a QA potion on this project. So I’m completely new to using any type of automation testing, or integrating with automated build systems such as Jenkins.

Now the question I have - Is there a way I can call to launch all of the trilleon tests within a debug build of unity? I’ve been looking through the Trilleon source and so far it seems all test launches are done from Editor scripts. How can I get these tests to run and output a log on a build or debug build? Can this only be done with Jenkins? As I saw something on the wiki that referenced running the tests in the Jenkins Pipeline. Is there a static reference to AutomationMaster that can be called to run the LaunchTests from that class?

Thanks.

1 Like
9

Hi Charles,

If you want to run it remotely via the editor, you can simply use the Unity editor command line tools, and point to a static method that will then trigger the entire run. I don’t believe I have create something myself for this just yet, but you can create your own as needed. And then within that static method that you call via command line, you can use the exact same logic in Utilities > CommandConsoleBase.cs

        static string RunTests(List<string> args) {

            Arbiter.LocalRunLaunch = true;
            ConnectionStrategy.ReceiveMessage(string.Format("{{ \"automation_command\": \"rt {0}\" }}", string.Join(string.Empty, args.ToArray())));
            return "Test run command received. Please view the Nexus AutoConsole for further logging updates.";

        }

This will launch the tests without the framework expecting a server to communicate with. All reports will be generated locally on the machine running it within the User folder under the Trilleon folder, as with manual editor runs.

Is this what you are looking for? When you say “debug” build of Unity, I am not sure if you are referring to a build apk, ipa, exe etc. If I am off base on this, let me know and I will try to answer your actual need.

10

So again I’m new to all this automation stuff so I’m still wrapping my head around a few things. I’m guessing that when I get my system setup with Jenkins build server, that the commands you’ve posted in the wiki will run all my tests on the build that Jenkins creates correct?

Also what I’m looking for, and this may be it, is that I can create a local development build (also known as debug build) from unity and have a way to run all my tests inside that build on a single machine, Basically something like

if(Debug.isDebugBuild)
{
    //Run all tests
}

Basically is there a way to run tests in any kind of unity build?

11

I got it figured out with what you sent me previously Apparently somehow my AutomationMaster.Initialzie() was missing that’s why it wasn’t working for me.

12

Ah, yes. The Initialize method is the one required line of code to go in your game code. And you certainly could have it automatically run the automation if you start up the application if it’s a debug build.

Now, the one problem with that is - if you ever want to do anything else with the debug build, you will need to manually disable the code that auto-runs automation. But if that is an acceptable limitation, then you can certainly do that.

You would not need much of the server code I provide with the framework for that, since there is no server communication. Ostensibly, you would need a simple script to launch the game, and then have the CI job wait and wait until the report is generated from the test run. And it times out after X minutes with no report, or it finds the report, pulls it up as an artifact etc.

Also, you could leverage the existing pubsub in Trilleon, and simply send basic heartbeats over the socket connection. These approaches, of course, require a little bit of customization on your end, but the lionshare of the needed logic is there - just needs to be plugged into.

13

Hey,

New Question for you as it was brought up as now we are 100% sure we are using Trilleon for testing on our project after my initial research of it has been complete. I’ve been unable to find any sort of Licensing agreements for the use of Trilleon? I’m assuming it falls under the Open Source Umbrella but if I could get a link or something to the specific licensing I would appreciate it.

14

Hi Charles! The license can be found here: https://github.com/disruptorbeam/trilleon/blob/master/LICENSE

Note: The license blurb also appears on the vast majority of files in the framework. The blurb will be at the top of the file and look like this:

Basically, you are free to do anything you want with the framework except for distribute it as your own work, or sell it. :slight_smile:

/*
+   This file is part of Trilleon.  Trilleon is a client automation framework.
+
+   Copyright (C) 2017 Disruptor Beam
+
+   Trilleon is free software: you can redistribute it and/or modify
+   it under the terms of the GNU Lesser General Public License as published by
+   the Free Software Foundation, either version 3 of the License, or
+   (at your option) any later version.
+
+   This program is distributed in the hope that it will be useful,
+   but WITHOUT ANY WARRANTY; without even the implied warranty of
+   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+   GNU Lesser General Public License for more details.
+
+   You should have received a copy of the GNU Lesser General Public License
+   along with this program.  If not, see <http://www.gnu.org/licenses/>.
*/
15

Greetings,

This is exciting stuff and I am just getting started with setting up a working environment to assess viability. I am working with the v1.1.4 deployment and I have a few questions. Actually, I have a lot of questions but I will start with a few quick high level ones that will unlock more research for me, i’ll defer the other questions until after the research…

  1. Quickstart.rtf point 4 references test examples in …/Assets/Automation/Examples however, I do not find this path. There is no Examples folder. Can this be found elsewhere?
  2. Quickstart.rtf point 5 references documentation on Trilleon Customization, where is this?
  3. Finally, the WIKI appears to have many errors when trying to load pages in the side nav TOC… are these links supposed to be navigating to detail pages or simply navigating lower down to targets in the same page?

Thanks

16

Hi Armado,

  1. The example project can be found here: https://github.com/disruptorbeam/trilleon/tree/master/examples
    And the he example tests can be found here: https://github.com/disruptorbeam/trilleon/tree/master/client/Assets/Automation/Tests
  2. The customization can occur primarily to all files under here https://github.com/disruptorbeam/trilleon/tree/master/client/Assets/Automation/Customizable
    Information on how/why to customize is under the comments in these files.
  3. At the moment of this posting, the website’s database is down (Amazon Web Services). I need to fix that this afternoon. All you can see is the homepage. Other pages will throw errors.
17

Awesome, thank you.

18

I recently moved, and my source control for the trilleon wiki exists solely on my desktop machine. I cannot find the power cord for the machine, and need to order a new one. I should have put it on Github like my other repos, but I sadly didn’t. So I do not have the ability to make the fix to get it working at the moment. I will fix that asap, though.

19

Hey tsibiski,

Firstly hope you were able to retrieve your source code for the wiki, I have a question, and of course a few suggestions at the end of this post.

First the question, is there some section of trilleon that captures exceptions thrown while the game is running? And if so is the garbage collection handled? The problem I ran into, the project I’m working on has about 6 scenes, one of these is a game loading bootstrapper, and in that can load one of 27 games. When I set up a group of tests to make sure all of these games load properly, after about 8-9 games would load, the tests would fail and the result was a critical failure, Garbarge Collection Alloc leak. Now before I decided to post here I wanted to make sure it wasn’t the project causing this error. So I basically wrote a set of tests that would load the games the same way I was with Trilleon, but without actually using trilleon and I didn’t run into any unexpected errors. Now there are quite a few exceptions that are thrown in our games as some things are still being worked on, and the majority of them are null references so nothing game breaking. Any insight on where this might be coming from would be helpful.

As far as suggestions. Considering you’re still working on the wiki, I was unable to figure out how to use the pubnub service with Trilleon as I’ve never used it before, so in order to get the reports in a way that I can read them in my automated build system I had to create my own reporter that sent them out to a text document. I also changed it so that it creates the .html file locally even when running in the build. Maybe having some way of setting these as options as I’m sure like me not everyone will want their reports going to a third party service.

Edit: I found what I was looking for

20

Unfortunately, I am having a heck of a time getting access back to the source control. Does it tell you what file the leak is occuring in? I would think the leak would possibly be occurring in PerformanceTracker.cs.

Trilleon does not dabble much in things that would not be automatically cleaned up by the garbage collector. Listeners are attached and detached.

When you say “the tests would fail and the result was a critical failure, Garbarge Collection Alloc leak”, do you have some specific error message or call stack?