Performance testing results on builds / in player

Unity Engine
1

Short question:
How can I get performance tests results when run on builds / in player?

Long question:
I’m trying to run some performance tests. Results in editor gives me weird results (i.e. it seems like the JIT C# compiler isn’t able to inline even the simplest methods) and I read all over forums that profiling editor gives unreliable results which shouldn’t be taken into account much and that I should measure release builds.

So I created some performance tests using Unity performance module. Problem is, I cannot figure out how to get tests results when run in build. The xml file is not generated (or I couldn’t find it) and Test Report window display nothing.

So what’s the correct way to execute build/player performance tests in order to get the reports?

What I’m I doing:

  • Follow guide on https://docs.unity3d.com/Packages/com.unity.test-framework.performance@0.1/manual/index.html:

  • Add performance module to packages’ manifest.json file.

  • Create assembly for performance tests.

  • Implement test using PerformanceTest attribute. Measuring is done with Measure.Method(...).WarmupCount(5).MeasurementCount(15).Run();

  • How I’m executing tests:

  • Variant a) (test runner in editor)

  • Window / General / Test Runner

  • Select play mode

  • Select my tests

  • Click “Run selected”

  • This works and I have results, but it’s executed in editor, so according to what I read around the internet, it’s unreliable. Results are available:

  • In the Test Runner

  • In Window / Analysis / Performance Test Report

  • Variant b) (test runner in build/player)

  • Window / General / Test Runner

  • Select play mode

  • Click “Run all in player (StandaloneWindows64)”

  • The player is built and executed. All tests pass.

  • Results are missing:

  • In Test Runner

  • In Performance Test Report window

  • In form of xml file

  • Variant c) (test runner from commandline)

  • Sources:

  • Performance benchmarking in Unity: How to get started

  • https://docs.unity3d.com/Manual/PlaymodeTestFramework.html

  • Execute command:

  • <unity.exe> -projectPath -testPlatform StandaloneWindows64 -runTests -testResults <path.xml>

  • This appears to work exactly as variant b; builds, runs, all tests pass, no results.

What kind of problem I’m trying to solve:
I have lots of data (tens of megabytes) which I need to process. I already used profiler to find out what the bottleneck is and I found out that most of my assumptions (based on claims like compiler being smarter then programmer) are wrong - for example I needed to copy some continuous data (simple structure, with only blittable members). When I implemented variant which uses pointer magic I gained ~90% performance boost (certain calculation went from ~2000ms to ~200ms) when compared with simple loop and ~30% when compared with System.Array.Copy.

My current problem is that I found out that another of my assumptions - about inlining even the simplest functions - is wrong, too, slowing down something about 25%.

So now I want to measure the most critical parts and I want to find out whether C# JIT in Unity is so ineffective or my code is slow or just editor is ineffective.

If you’re interested, example project with my current tests are included. (In this case, I noticed that BitArray is quite slower when compared with using int[ ] directly.)

3917548–334066–PerformanceTests.zip (21.4 KB)

2

Hi, have you found any solution for this? My Test Report window seems to show nothing regardless of the flow (in Unity 2018.3.10)

1 Like
3

No, I spent too much time without progress so I gave up. Instead, I’m just doing basic profiling in editor, some parts I have to optimize blindly and measure how it affects fps. It’s not that effective, but at least I made needed progress.

4

So, I figured this out.

It’s because if you are on a version before 2019.1, PerformanceTestRunSaver.cs is saving the test data to the Application.streamingAssetsPath instead of the Application.persistentDataPath where the Test Report window is looking for it.

I changed the code in TestReportWindow.cs to look for it at the streamingAssetsPath instead.

private void LoadData()
{
     string filePath = Path.Combine(Application.streamingAssetsPath, "PerformanceTestResults.json");
     ...
}
1 Like
5

Thank you!