Introducing System Metrics Mali package

Unity Engine
, ,
1

System Metrics Mali Package (SMM)

Welcome to the System Metrics Mali discussion thread. You can use this thread to ask for help, share feedback, and have discussions about the experimental System Metrics Mali package.

Introduction
System Metrics Mali provides a simple and extensible interface for reading GPU hardware counters on Arm GPUs. Hardware counters provide a low-level view of application performance. These metrics can provide insight into what impact your changes produce on the hardware level. System Metrics Mali package exposes a set of hardware metrics via Unity Profiler window and Unity ProfilerRecorder API.

Disclaimer: This package is still under development and anything could change on the API side.

High-level features summary
It allows access to and monitoring of GPU low-level counters. Sampling of the counters is performed on a per-frame level, so you can monitor changes in hardware system metrics along with other Unity Profiler frame stats.

Resources
For more information, please have a look at:

Requirements

  • The minimum required Unity version is 2021.2
  • Supports Samsung Exynos, HiSilicon Kirin, or MediaTek CPUs (as these use Mali GPU’s)
  • Doesn’t support Qualcomm Snapdragon CPUs (as these use Adreno GPU’s)

Getting started

  • Download Unity 2021.2.0a17 or newer through the Hub.

  • Create or open a Unity project.

  • Go to Window > Package Manager.

  • Follow “Adding a registry package by name” instructions and add “com.unity.profiling.systemmetrics.mali”

  • Build and run your application

  • Go to Window > Analysis > Profiler

  • Connect to a device and start capturing

  • Click on Profiler Module > Gear Icon

  • Click Add icon > Add counters from User group

Current state
The experimental System Metrics Mali Package is still under development and only a limited number of counters are currently available. When using this package you can expect that:

  • It should have zero CPU overhead if no hardware counters are being read through ProfilerRecorder API and/or Profiler isn’t attached
  • It has minimal fixed memory overhead of around 5-10Kb

Hardware counters are sampled on a per-frame basis. Currently, whenever an application has CPU spikes that might cause System Metrics Mali to sample a portion of the next frame that looks like a spike up followed by a spike down. This is caused by captured frame interval and GPU frame interval misalignment.

Most values are provided as absolute GPU cycles. This is due to the fact that GPU frequency is unavailable on most phones and we can’t measure relative performance.

Older phones might have outdated implementations, but generally, the plugin should work on any phone released after 2010.

What’s next

  • We plan to significantly improve UI usability by introducing of custom Profiler Module which would allow providing pre-configured Profiler Module and have custom visualization for the counters
  • Replace absolute value counters with relative counters related to specific GPU pipeline stage
  • Add more hardware counters for different GPU pipeline stages to have high-level for each stage which might be an application bottleneck
  • Add support for non-mobile platforms and CPUs

Feedback
In terms of feedback, we’re especially looking for:

  • Are there any workflows that are unclear or missing?
  • Any hardware compatibility or stability issues?
  • Are there any issues or unclear parts in the documentation?

Please feel free to post any feedback in this sub-forum.

How to report bugs
Ideally, we’d like any bugs reported through the built-in bug reporter tool, as that will automatically provide us with some relevant context. When reporting bugs, please:

  • Report exact mobile phone model name and number
  • Attach logcat logs files if available

Once you have submitted a bug report through the bug reporter, please feel free to start a discussion about it in this thread.

Thank you for your interest, we’re looking forward to your feedback!

3 Likes
2

0.2.0 update is available
Changes:

  • Updated HWCP library to the latest version.
  • Added custom Profiler Module “Mali System Metrics” with an improved detailed view,
  • Added more hardware and derivated counters. See “Available metrics” for more details.
  • Improved metrics description documentation

Newly added metrics:

  • GpuShaderCoreUtilization - % Shader Core Utilization
  • GpuNonFragmentUtilization - % Vertex And Compute Utilization
  • GpuFragmentUtilization - % Fragment Utilization
  • GpuShaderFragmentActive - Shader Fragment Cycles
  • GpuShaderComputeActive - Shader Compute Cycles
  • GpuShaderTripipeActive - Shader Tri-Pipe Cycles
  • GpuShaderFragmentUtilization - % Shader Fragment Utilization
  • GpuShaderComputeUtilization - % Shader Compute Utilization
  • GpuShaderTripipeUtilization - % Shader Tri-Pipe Utilization
  • GpuShaderArithmeticUtilization - % Shader Arithmetic Utilization
  • GpuShaderLoadStoreUtilization - % Shader Load/Store Utilization
  • GpuShaderTextureUtilization - % Shader Texture Unit Utilization
  • GpuReadStallRate - % Memory Read Stalled
  • GpuWriteStallRate - % Memory Write Stalled
  • GpuInputPrimitives - Input Primitives
  • GpuCulledPrimitives - Culled Primitives
  • GpuClippedPrimitives - Clipped Primitives
  • GpuVisiblePrimitives - Visible Primitives
  • GpuCulledPrimitivesPercentage - % Culled Primitives
  • GpuClippedPrimitivesPercentage - % Clipped Primitives
  • GpuVisiblePrimitivesPercentage - % Visible Primitives

New custom profiler detailed view:

7263487--876484--image (7).png
7263487--876484--image (7).png976Ă—1110 91.3 KB

3

Thanks very useful package :slight_smile:

Adding GPU Utilisation metrics on graphs and in detailed view will be very appreciated :slight_smile: as much as adding description in tooltip for every metric in detailed view :slight_smile:

1 Like
4

Wish we could use this! Do you know of any way to get these counters from Adreno GPUs?

5

Unfortunately, Qualcomm doesn’t provide API or documentation on how to access Adreno GPU hardware counters.

Although it might be possible through kernel IOCTL. Here is an article on the topic.
Sampling Performance Counters from Mobile GPU Drivers | Lei.Chat()
Haven’t tried it myself yet, but it seems it’s possible while not easy though.

6

New 1.0.0-pre.2 version is now available

The package is Release Candidate now and can be searched and added via Unity Package Manager now.
Changed

  • Significantly reduced plugin binary size.
  • HWCP library is updated to the latest version.
  • Counters were renamed to be named uniformly.

Added

  • Exposed additional counters for relative load measurement.
  • Exposed additional counters for primitives count and GPU shader core activity.
  • Added API documentation.
  • Added tooltips to Profiler Module UI.
  • Added examples and usage guide into documentation.
  • Added detailed description for each hardware counter.
  • Added new play mode verification tests.

Fixed

  • Fixed issue when Pixels counter might have shown large negative value.
  • Fixed issue that it was impossible to add counters before connecting Profiler to an Android device.
  • Fixed crash on some Mali-T880 devices caused by zero data in counters
1 Like
7

New 1.0.0-pre.3 version is now available
Changed

  • Reduced plugin binary size by ~10%
  • HWCP library is updated to the latest version to add Mali-G68 and Mali-G78 support.
8

Thank you for this!
It doesn’t seem to work with my phone (probably it’s the phone’s fault, it’s a Moto e4 and ARM Streamline reports that CONFIG_HW_PERF_EVENTS is not set and/or that the PMU isn’t configured in the target device tree). I’m going to try rebuild the kernel (whenever I learn how to) (Edit: I enabled it and more stuff, but still didn’t work) with CONFIG_HW_PERF_EVENTS set, but my feedback would be for the “Mail System Metrics” tab to report that the phone isn’t compatible (specifying why is probably asking too much, and something that can be solved by the developer anyway) instead of showing all the stats at 0 (which could lead to confusion).

1 Like
9

Good point, we’ll look into how we can do that. Thanks!

1 Like
10

Hey @antonk-unity ,

First of all, thank you for making this package - so much profiling details is something one could only dream before!

But I couldn’t get it working so far with my Mali-G76. More likely I’m missing something obvious, but this is what I see while profiling my Galaxy S10+ (Exynos 9820):

7882495--1002820--upload_2022-2-9_17-0-50.png
7882495--1002820--upload_2022-2-9_17-0-50.png1599Ă—1098 97.6 KB

I have dev build, Frame Timing Stats option enabled and using package Version 1.0.0-pre.2 (pre.3 is not available in the latest public 2022.1.0b7).

I also don’t see the User counters group in the Profiler Module Editor:

7882495--1002823--upload_2022-2-9_17-4-52.png
7882495--1002823--upload_2022-2-9_17-4-52.png1317Ă—983 70 KB

What could I miss while setting things up?..

P.S. tried to manually update up to the pre.3 and pre.5 versions - there were errors in console and no Mali System Metrics profiler module available at all in both cases (might be not compatible with 2022.1.0b7).

11

Hi @codestage

I’m just investigating this for you now, I don’t have the exact same device to hand but I’ve tried on a Galaxy Note 10 (Mali-G76) and it looks to be working for me. Did you happen to have any error messages shown to you when you tried

Thanks

12

Thanks for taking a look, but no, not a single error in the Unity Console.
I’ve also checked adb logs and didn’t find anything related. I’ve attached the adb logs here just in case.

Furthermore, I also tried to profile the same device from an entirely another PC but without luck as well - still nothing for the Mali System Metrics module so far.

Please let me know how can I debug it if possible to get more logs for you, I’ll be glad to help.

7885756–1003438–logcat.txt (44.5 KB)

1 Like
13

I’m from the Arm team who work on the backend counter interface the Unity profiler is using. Thanks for reporting the issue. A couple of questions:

  • Can you confirm the driver version in your device? (Open chrome://gpu and find GL_VERSION string in the table).
  • Can you try the Streamline profiler in Arm Mobile Studio and see if that manages to collect any data?

Also, at the risk of sounding like an IT tech support, have you tried turning the device off and on again? The main causes of failure in the driver for this look related to the kernel driver being unable to allocate memory, so a reset may help (although I doubt it).

Kind regards,
Pete

1 Like
14

Hey Pete,

Thank you for chiming in,

Sure here is my GL_VERSION from the device:
OpenGL ES 3.2 v1.r32p1-01bet2-mbs2v39_0.9832bf74fac3612104002d946600017d

I’ve tried, and it seems to work just fine:
7888309--1003888--upload_2022-2-11_14-11-23.png

While still nothing in the Unity Profiler’s Mali System Metrics…

=D yeah, this is a good one actually, and I know a lot of cases which were fixed with rebooting a device, but not this time unfortunately: rebooting didn’t help - still nothing in the profiler for the Mali module =/

7888309--1003894--upload_2022-2-11_14-13-16.png

15

I just tried to reproduce it with another device on the same SoC (SM-G970F with Exynos 9820 and Mali-G76) and it does reproduce for it as well.

16

Thanks for the extra info. If Streamline is working then the underlying mechanism isn’t broken, so I just need to work out why the driver connection is being rejected for the backend library the Unity profiler is using.

Cheers,
Pete

1 Like
17

Just let me know if I can help you with any further debugging, I’ll be glad to help!

1 Like
18

New 1.0.0 version is now available!
Relative to 1.0.0-pre.3 changes are:
Fixed

  • Fixed counters sampling in Profiler-only capture mode
  • Fixed example script.

Added

  • Added usage guide documentation.

Your issue should be fixed in 1.0 package version

1 Like
19

@antonk-unity

Thank you for releasing a new version!
I’ve just updated the package to 1.0.0 version at Unity 2021.2.0f1 and this is what I see every time when entering Play Mode (I didn’t make a build or connected to the Android device yet after fresh Unity instance start):

SystemMetricsMali: Initialization failed
UnityEngine.Debug:Log (object)
Unity.Profiling.SystemMetrics.SystemMetricsMali:InitializeSystemMetricsMali () (at Library/PackageCache/com.unity.profiling.systemmetrics.mali@1.0.0/Runtime/SystemMetricsMali.cs:307)

It does happen in Unity 2022.1.0b9 too.

Is it expected behavior?

Next thing I did:

  • Opened project with older SMM package in Unity 2022.1.0b9
  • Build and run on device
  • Attach Profiler and start capture (saw empty Mali profiler module)
  • Update SMM package
  • saw numerous warnings in Console like this:
SystemMetrics: Failed to get handle for 'GpuCycles' id, check that native plugin id match
UnityEngine.Debug:LogWarning (object)
Unity.Profiling.SystemMetrics.SystemMetricsLibHWCP:GetProfilerRecorderHandle (string) (at Library/PackageCache/com.unity.profiling.systemmetrics.mali@1.0.0/Runtime/SystemMetricsLibHWCP.cs:40)
Unity.Profiling.SystemMetrics.SystemMetricsMali:.ctor () (at Library/PackageCache/com.unity.profiling.systemmetrics.mali@1.0.0/Runtime/SystemMetricsMali.cs:325)
Unity.Profiling.SystemMetrics.SystemMetricsMali:.cctor () (at Library/PackageCache/com.unity.profiling.systemmetrics.mali@1.0.0/Runtime/SystemMetricsMali.cs:312)
Unity.Profiling.Editor.SystemMetrics.Mali.MaliProfilerModule:.cctor () (at Library/PackageCache/com.unity.profiling.systemmetrics.mali@1.0.0/Editor/ProfilerModule/MaliProfilerModule.cs:10)

closing with these errors:

ArgumentException: ProfilerRecorderHandle is not initialized or is not available
Parameter name: handle
Unity.Profiling.LowLevel.Unsafe.ProfilerRecorderHandle.GetDescription (Unity.Profiling.LowLevel.Unsafe.ProfilerRecorderHandle handle) (at <b2c530e619ab42aeba489c5671c82671>:0)
Unity.Profiling.Editor.SystemMetrics.Mali.MaliProfilerModule.GetDescriptorProfilerCounterHandle (Unity.Profiling.LowLevel.Unsafe.ProfilerRecorderHandle handle) (at Library/PackageCache/com.unity.profiling.systemmetrics.mali@1.0.0/Editor/ProfilerModule/MaliProfilerModule.cs:56)
Unity.Profiling.Editor.SystemMetrics.Mali.MaliProfilerModule..cctor () (at Library/PackageCache/com.unity.profiling.systemmetrics.mali@1.0.0/Editor/ProfilerModule/MaliProfilerModule.cs:10)
Rethrow as TypeInitializationException: The type initializer for 'Unity.Profiling.Editor.SystemMetrics.Mali.MaliProfilerModule' threw an exception.

Unable to create Profiler module of type Unity.Profiling.Editor.SystemMetrics.Mali.MaliProfilerModule. The Profiler module 'Mali System Metrics' cannot have no chart counters.
UnityEditor.ProfilerWindow:OnEnable ()

And Mali module disappear from the Profiler entirely.
Looks like an edge case with updating from the specific version to the 1.0.0 which might reproduce when updating from 1.0.0 to the newer version.

Then I restarted Unity 2022.1.0b9, build&run with Auto-connect Profiler option enabled, and it worked fine, though it didn’t show the graph until the profiler window resize.

Hope this feedback helps!

20

Thanks for the feedback. Unfortunately, that’s a Unity Editor limitation.
You need to restart it, as Unity doesn’t support plugin DLLs unload in Editor.
And while you upgrade the package the DLL is being replaced, but the old one is still loaded.
I’ve tested it locally and Unity Editor reload seems to help.

1 Like