GitLab CI and PlayMode tests?

Hello.
Apparently you can't run playmode tests in batchmode, as batchmode does not support coroutines and async code. However, running Unity in regular mode (non batchmode) via GitLab CI seems to not spawn any test result files.

How do you folks manage to run tests in playmode on your GitLab CI configuration?

Not sure where you got the idea that batchmode doesn't support coroutines or async code: https://docs.unity3d.com/Manual/CLIBatchmodeCoroutines.html


Hey. Thanks for the answer. To be honest I can't find the source but I'm pretty sure that I've read that on a ticket about unit tests in Unity.
Ok so, you confirm that play mode tests are supposed to work even in batch mode?

Yes, absolutely. We use them extensively when testing Unity itself.

3 Likes

Thanks. Another question @superpig (my last question I'd say): do you use it with GitLabCI or know devs who do it?
I just need confirmation that my GitLabCI + play mode tests is possible and that the issue I get comes from a mistake I did and not an incompatibility of the two systems.

@laurentvictorino We use GitLab CI with Playmode tests. We're running a self-hosted GitLab Runner on Windows, executing a PS script. Works mostly fine. Recently we ran into issues about Unity not being able to create an OpenGL context, because GitLab Runner was being installed as a Windows Service, which can cause these kinds of issues. For some reason this only seems to happen on some systems or GPU drivers but not all. For now GitLab Runner is just registered as a startup item instead of a service.

@DrummerB Hey thank you for your answer. I'm looking that way now. I can't really set gitlab-runner as a startup item as it's on a building machine that is not operated - no real log in or session here -. But it gave me an idea and a good lead I guess.
May I ask you if you could share the command line you use in the .gitlab-ci.yml configuration file to start a Unity Build?
Thanks for your help.

Sure. Sorry for the late response, I was on holidays. I'm posting the relevant parts of the GitLab test setup. I was also thinking of writing it all up into a blog post at some point. It takes a bit of tinkering, but you have full control and it works quite well.

This is our current GitLab CI config file.

stages:
  - Verify
  - Build
  - Test

.Job:
  tags:
    - self-hosted, windows, powershell, unity
  variables:
    GIT_CLEAN_FLAGS: -ffdx -e Library/
  before_script:
    - BuildScripts/before_script.ps1

.Verify:
  extends: .Job
  stage: Verify

Verify Commit:
  extends: .Verify
  script:
    - BuildScripts/verify_lfs.ps1

.Build:
  extends: .Job
  stage: Build
  rules:
    - if: '$CI_MERGE_REQUEST_ID == null && $CI_COMMIT_MESSAGE !~ /[skip build]]/'
  script:
    - BuildScripts/build.ps1
  artifacts:
    name: "${env:CI_JOB_NAME}-${env:CI_COMMIT_SHORT_SHA}"
    expire_in: 3 days
    paths:
      - "./Builds"

Redacted-Win64:
  extends: .Build
  variables:
    BUILD_NAME: Redacted
    BUILD_TARGET: StandaloneWindows64

Redacted-Linux64:
  extends: .Build
  variables:
    BUILD_NAME: Redacted
    BUILD_TARGET: StandaloneLinux64

.Test:
  extends: .Job
  stage: Test
  dependencies: []
  script:
    - BuildScripts/build.ps1
  artifacts:
    when: on_failure
    reports:
      junit: TestResults/JUnit*.xml
      cobertura: CodeCoverage/*-cobertura/Cobertura.xml
    paths:
      - TestResults/

Editor Tests:
  extends: .Test

Runtime Tests:
  extends: .Test

And the relevant part of the build script:

$BUILD_PATH = Join-Path $PROJECT_PATH -ChildPath "Builds/$BUILD_TARGET/"
$TEST_PATH  = Join-Path $PROJECT_PATH -ChildPath "TestResults/"
$NUNIT_PATH = Join-Path $TEST_PATH -ChildPath "NUnit.xml"
$JUNIT_PATH = Join-Path $TEST_PATH -ChildPath "JUnit.xml"
$LOG_PATH   = Join-Path $PROJECT_PATH -ChildPath "Logs/Unity.log"
New-Item -Path $LOG_PATH -ItemType "file" -Force | Out-Null

# Prepare the arguments to pass to Unity.
$UNITY_ARGS = @()
$UNITY_ARGS += "-projectPath", """$PROJECT_PATH"""
$UNITY_ARGS += "-batchmode"
$UNITY_ARGS += "-logFile", "-" # This makes Unity print the logs to stdout, which we forward to the console (for GitLab and PowerShell) and a log file
$UNITY_ARGS += "-buildTarget", """$BUILD_TARGET"""
$UNITY_ARGS += "-buildPath", """$BUILD_PATH"""
$UNITY_ARGS += "-buildName", """$BUILD_NAME"""

if ($UNITY_CACHE_SERVER) {
  $UNITY_ARGS += "-CacheServerIPAddress", "$UNITY_CACHE_SERVER"
}

if ($CI_JOB_STAGE -eq "Build") {
  $UNITY_ARGS += "-executeMethod", "Redacted.Editor.BuildScript.PerformBuild"
  $UNITY_ARGS += "-quit"
}

if ($CI_JOB_NAME -eq "Editor Tests") {
  $UNITY_ARGS += "-executeMethod", "Redacted.Tests.TestRunner.RunEditorTests"
  $UNITY_ARGS += "-enableCodeCoverage"
  $UNITY_ARGS += "-coverageOptions", "assemblyFilters:+Redacted"
}

if ($CI_JOB_NAME -eq "Runtime Tests") {
  $UNITY_ARGS += "-executeMethod", "Redacted.Tests.TestRunner.RunRuntimeTests"
}

if ($CI_JOB_NAME -eq "Standalone Tests") {
  $UNITY_ARGS += "-executeMethod", "Redacted.Tests.TestRunner.RunStandaloneTests"
}

if (!(Test-Path $UNITY_EXE)) {
  "Did not find Unity $UNITY_VERSION."
  exit 1
}

# Run Unity
& $UNITY_EXE $UNITY_ARGS | Tee-Object -FilePath $LOG_PATH
$UNITY_EXIT_CODE = $LastExitCode

"--------------------------------------------------------------------------------"
"Finished with exit code: $UNITY_EXIT_CODE"
exit $UNITY_EXIT_CODE

The build method:

 static void PerformBuild()
        {
            Console.WriteLine("Performing build");

            // Use the build target specified on the command line, or default to the active build target otherwise.
            string buildTargetArg = GetArgument("buildTarget");
            if (string.IsNullOrWhiteSpace(buildTargetArg) || !Enum.TryParse(buildTargetArg, out BuildTarget buildTarget))
            {
                buildTarget = EditorUserBuildSettings.activeBuildTarget;
            }

            // If we do not support the selected build target (e.g. corresponding editor module is missing), abort.
            if (!BuildPipeline.IsBuildTargetSupported(BuildTargetGroup.Standalone, buildTarget))
            {
                Console.WriteLine($"Build target {buildTarget.ToString()} not supported. Install the module from Unity Hub.");
                if (Application.isBatchMode)
                {
                    EditorApplication.Exit(1);
                }
                return;
            }

            string buildName = GetArgument("buildName") ?? Application.productName;
            string buildPath = GetArgument("buildPath") ?? Path.Combine("./Builds", buildTarget.ToString());
            string fullPath = Path.Combine(buildPath, buildName);

            if (buildTarget == BuildTarget.StandaloneWindows64 || buildTarget == BuildTarget.StandaloneWindows)
            {
                fullPath += ".exe";
            }

            var options = new BuildPlayerOptions
            {
                scenes = (from scene in EditorBuildSettings.scenes where scene.enabled select scene.path).ToArray(),
                locationPathName = fullPath,
                target = buildTarget,
                options = Application.isBatchMode ? BuildOptions.None : BuildOptions.ShowBuiltPlayer
            };

            var report = BuildPipeline.BuildPlayer(options);
            LogBuildReport(report);

            if (Application.isBatchMode)
                EditorApplication.Exit(report.summary.result == BuildResult.Succeeded ? 0 : 1);
        }

And the test runner:

public class TestRunner : IPrebuildSetup, IPostBuildCleanup, ITestRunCallback
    {
        #if UNITY_EDITOR
        [MenuItem("Test/Run Editor Tests")]
        static void RunEditorTests()
        {
            Console.WriteLine("Running editor tests");

            var filter = new Filter
            {
                testMode = TestMode.EditMode,
                groupNames = new[] {@"^Redacted\.Tests\.Editor\."}
            };
            ScriptableObject.CreateInstance<TestRunnerApi>().Execute(new ExecutionSettings(filter));
        }

        [MenuItem("Test/Run Runtime Tests")]
        static void RunRuntimeTests()
        {
            Console.WriteLine("Running runtime tests");

            var testRunner = ScriptableObject.CreateInstance<TestRunnerApi>();
            var filter = new Filter
            {
                testMode = TestMode.PlayMode,
                groupNames = new[] {@"^Redacted\.Tests\.Runtime\."}
            };
            testRunner.Execute(new ExecutionSettings(filter));
        }

        [MenuItem("Test/Run Standalone Tests")]
        static void RunStandaloneTests()
        {
            Console.WriteLine("Running standalone tests");

            var testRunner = ScriptableObject.CreateInstance<TestRunnerApi>();
            var filter = new Filter
            {
                testMode = TestMode.PlayMode,
                groupNames = new[] {@"^Redacted\.Tests\.Runtime\."},
                targetPlatform = EditorUserBuildSettings.activeBuildTarget
            };
            testRunner.Execute(new ExecutionSettings(filter));
        }
        #endif

        public void RunStarted(ITest tests)
        {
            Debug.Log("TestManager.RunStarted");
        }

        public void RunFinished(ITestResult testResults)
        {
            Debug.Log("TestManager.RunFinished");

            ResultsWriter.WriteResults(testResults);
            CoverageReportGenerator.Generate();

            if (Application.isBatchMode)
            {
                int returnValue = 1;
                switch (testResults.ResultState.Status)
                {
                    case NUnit.Framework.Interfaces.TestStatus.Inconclusive:
                        returnValue = 2;
                        break;
                    case NUnit.Framework.Interfaces.TestStatus.Skipped:
                        returnValue = 0;
                        break;
                    case NUnit.Framework.Interfaces.TestStatus.Passed:
                        returnValue = 0;
                        break;
                    case NUnit.Framework.Interfaces.TestStatus.Failed:
                        returnValue = 1;
                        break;
                }
                Console.WriteLine($"Test Result: {testResults.ResultState}");
                #if UNITY_EDITOR
                EditorApplication.Exit(returnValue);
                #endif
            }
        }
    }
3 Likes

Hey @DrummerB thanks for your reply (I hope holidays were good!).
Thanks for sharing all the good stuff. I've been looking at it, and there is only one thing I'm not sure to get: What does GetArgument() is supposed to do? Where does it come from?
Thanks.

@laurentvictorino It just reads the passed command line arguments.

        static string GetArgument(string name)
        {
            var args = Environment.GetCommandLineArgs();
            for (int i = 0; i < args.Length; i++)
            {
                if (args[i].Contains(name))
                {
                    return args[i + 1];
                }
            }

            return null;
        }

Should probably just parse the arguments once into a hash table and then use that, but this works ok.

@DrummerB apologies for posting on such an old thread, but it was the only one that I could find that was relevant, so I thought it might be useful for future readers.

What you've done looks great! Got most of it working, but would love to get the coverage integrated with GitLab's tracking and highlighting in MRs for which you need cobertura formatted output, which it looks like you have.

From what I can tell, the default Unity coverage output is in a custom format. Did you write a custom tool to that reformatting, or is that an option in Unity that I'm missing?

Thanks!

Unity's coverage package uses the third-party ReportGenerator project to generate its coverage output. So if your project includes the coverage package, you have access to the ReportGenerator and can use it to output any format you want. Not the most elegant solution, but this is what we do:

using System;
using System.IO;
using System.Linq;
using System.Xml;
using Palmmedia.ReportGenerator.Core;
using Palmmedia.ReportGenerator.Core.CodeAnalysis;
using UnityEditor;
using UnityEngine;

/// <summary>
/// This is used by the TestRunner after a test run was finished to convert the generated
/// OpenCover formatted code coverage results into other report formats.
/// The CodeCoverage package from Unity currently (v0.3-preview) only supports generating
/// html reports. GitLab however requires Cobertura format, which we generate here.
/// </summary>
public static class CoverageReportGenerator
{
    public static bool Generate()
    {
        // For config details, see https://github.com/danielpalme/ReportGenerator#usage--command-line-parameters
        var projectPath = Directory.GetCurrentDirectory();
        var coveragePath = Path.Combine(projectPath, "CodeCoverage");
        var reportFilePatterns = new[] {Path.Combine(coveragePath, "**/TestCoverageResults_????.xml")};
        var targetDirectory = coveragePath;
        var sourceDirectories = new string[]{};
        string historyDirectory = null;
        var reportTypes = new[]{"Cobertura", "TextSummary", "Html"};
        var plugins = new string[]{};
        var assemblyFilters = new[]{"+*"};
        var classFilters = new[]{"+*"};
        var fileFilters = new[]{"+*"};

        string verbosityLevel = null;
        string tag = null;

        var config = new ReportConfiguration(
            reportFilePatterns,
            targetDirectory,
            sourceDirectories,
            historyDirectory,
            reportTypes,
            plugins,
            assemblyFilters,
            classFilters,
            fileFilters,
            verbosityLevel,
            tag);

        var settings = new Settings();
        var thresholds = new RiskHotspotsAnalysisThresholds();
        string coberturaPath = Path.Combine(targetDirectory, "Cobertura.xml");

        if (!Application.isBatchMode)
            EditorUtility.DisplayProgressBar("Code Coverage", "Generating Coverage Report", 0.4f);
        else
            Debug.Log("Generating Cobertura Report");

        bool success;
        try
        {
            var generator = new Generator();
            success = generator.GenerateReport(config, settings, thresholds);

            if (!File.Exists(coberturaPath))
            {
                success = false;
                throw new FileNotFoundException(coberturaPath);
            }

            // Replace absolute file paths in the cobertura XML with relative paths (to the project root).
            // This is required for GitLab to be able to parse the results.
            var document = new XmlDocument();
            document.Load(coberturaPath);
            const string xPath = "/coverage/packages/package/classes/class/@filename";
            var fileNameAttributes = document.DocumentElement?.SelectNodes(xPath);
            if (fileNameAttributes != null)
            {
                foreach (XmlAttribute attribute in fileNameAttributes)
                {
                    // Remove project path prefix, Path.GetRelativePath() requires .NET Standard 2.1
                    if (attribute.Value.StartsWith(projectPath)) {
                        attribute.Value = attribute.Value.Substring(projectPath.Length + 1).Replace("\\", "/");
                    }
                }
            }
            document.Save(coberturaPath);
        }
        finally
        {
            EditorUtility.ClearProgressBar();
        }

        if (success)
            Debug.Log($"Cobertura Code Coverage Report was generated in {targetDirectory}");
        else
            Debug.LogError("Failed to generate Code Coverage Report.");

        // Log the coverage summary to the console.
        var summaryPath = Path.Combine(coveragePath, "Summary.txt");
        if (File.Exists(summaryPath))
        {
            var summary = string.Join("\n", File.ReadLines(summaryPath).Take(11));
            if (Application.isBatchMode)
                Console.WriteLine(summary);
            else
                Debug.Log(summary);
        }

        return success;
    }
}
public class TestRunner
{
    [UsedImplicitly] // by the build script
    static void GenerateCodeCoverageReport()
    {
        Console.WriteLine("Generating code coverage report.");
        bool success = CoverageReportGenerator.Generate();

        #if UNITY_EDITOR
        EditorApplication.Exit(success ? 0 : 1);
        #endif
    }
}

The TestRunner.GenerateCodeCoverageReport method is executed using the Unity CLI in a separate GitLab job after all test jobs have been completed.

Code Coverage:
  stage: Report
  tags:
    - self-hosted, windows, powershell, unity
  variables:
    GIT_CLEAN_FLAGS: -ffdx -e Library/
  before_script:
    - BuildScripts/before_script.ps1
  script:
    - BuildScripts/build.ps1
  dependencies:
    - Editor Tests
    - Runtime Tests
  rules:
    - if: '$CI_MERGE_REQUEST_ID == null && $CI_COMMIT_MESSAGE !~ /[skip test]]/'
      when: always
  coverage: '/^ *Line coverage: (\d+\.\d+%)$/'
  artifacts:
    reports:
      cobertura: CodeCoverage/Cobertura.xml
    paths:
      - CodeCoverage/
4 Likes