Performance Testing package doesn't produce a test result when running on a device

Unity Engine
,
1

Hello,

I got 2 issues with test reports.

  1. There is no data in the Test Report window when running tests on a target device via Test Runner in the editor.
    This post tells that at least result.xml should be in the streaming assets folder, but it is not there either.
    So I tried to run it via the command line as this post suggests and got another issue.

  2. There is no test result saved at the given path when I am running tests using -testPlatform StandaloneWindows64 -buildTarget StandaloneWindows64 or -testPlatform Android -buildTarget Android even though I can see that tests are finished successfully both on my android device and PC respectively.
    Here is the full command I use:

./Unity.exe -runTests -projectPath D:/Development/Unity/<projectDirectory> 
-testPlatform Android -buildTarget Android -mtRendering -scriptingbackend=il2cpp 
-testResults D:/Development/Unity/<projectDirectory>/TestReport/TestReport.xml 
-logfile D:/Development/Unity/<projectDirectory>/TestLogs/TestLog.txt

However, if I replace -testPlatform with -testPlatform PlayMode then the test result is produced at the given path, but obviously, this argument makes the run be executed in the editor instead of a target platform, therefore results are kinda irrelevant.
So without TestReport.xml I cannot even use Unity Performance Benchmark Reporter to produce a report and got no way to see performance results after testing on a target device.

I tested it on:
Performance Testing versions:
2.7.0-preview,
2.8.0-preview
(2.5.0 from the docs is not working for me at all, as a package couldn’t be found)

Unity versions:
2021.3.0f1,
2022.1.0b8.

The project where issues reproduce in 100% cases:
GitHub

What is the way to get a test report after running a performance test suite on a real device?

2

Well, JFYI I tried GitHub - Unity-Technologies/XRAutomatedTests: XRAutomatedTests is where you can find functional, graphics, performance, and other types of automated tests for your XR Unity development., particularly NoXR.
Using 2019.3.3f1 and Performance Testing 1.1.31.

A run on WindowsStandalone64 via the command line produced a test report file.

So I tried to downgrade the Performance Testing package to the same version in my project, but the run still doesn’t produce a test report. Looks like the next step should be downgrading the Unity version, going to create a sample project since the one I am using requires a new API that was introduced in 2020.

3

I created an empty project, added the Performance Testing package, and checked running tests on a device using different unity versions.
Here are the results:

  • Unity 2019.3:

  • Performance Testing 1.1.31: a report is produced when running tests on a device via the terminal. A report is also available inside the editor via the Test Report window when running tests using the Run All Tests (Android) button inside the Test Runner window.

  • Performance Testing 2.8.0: same as the version 1.1.31. Reports from a device run are produced both using the terminal and the editor.

  • Unity 2020.3/2021.3/2022.1

  • Performance Testing 1.1.31: a report is not created when running tests on a device via the terminal. After running tests on a device via the editor the Test Report window doesn’t show the results (the data is updated only when running tests in the editor, not on a device).

  • Performance Testing 2.8.0: same as the version 1.1.31. Nothing is available after the test run on a device.

Unity Technologies please fix.

1 Like
4

Somehow I was able to get the report in the editor after running performance tests on an Android device via the test runner window using the same project I linked in the first message and Unity 2021.3.

Out of around 30 runs, I got 2 in a row, which results were available in the Test Report window.
And I noticed in both runs that instead of showing a Performance Testing message “All test(s) succeeded” in the end, the app was automatically closed and after that I could see data in the test report window.
But when I am closing the app via the close button there are no results available.

So don’t ask me how, but I found the following sequence of actions that allows me to get a report every time:

  1. Run tests on a target platform via the Test Runner window
  2. When a test run is finished manually connect Unity Profiler to your device
  3. In a few seconds the app is closed and the result is available in the Test Report window.

Sounds crazy, but works for me.

1 Like