High memory usage on Android without a clear idea why

[BananaBug]

Hello,

I’m trying to get the memory usage down of our game on Android devices, as on “low” memory devices (2GB and below), the game gets killed if the game shells out to a link in the browser.
The first step is obviously trying to understand what exactly is using the memory.

Our game is built with Unity version 2019.4.18, but for getting the memory profiler to work better, I’ve updated to 2019.4.31 for the purpose of making builds for profiling. Builds have been built in development mode, but in release mode in Android Studio. The device being tested on has 4GB ram to make there’s enough space to not get influenced by memory pressure.

Using the memory profiler package, on one specific point, I am seeing 314MB in use and 412MB reserved. However using adb command adb shell dumpsys meminfo, it is showing memory usage is 772MB. That’s quite a massive gap and I would like to understand why that is, or if I am misunderstanding something. Screenshots of both are at the bottom of this post.

As far as I can tell, it looks like the “Private Other”/“Unknown” category of the adb command might not be tracked by the Unity memory profiler, but there’s no easy way to determine what’s in there. (The value remains high on non development builds as well). I checked a competitor game that according to logs looks like it’s built with a similar 2019.4 version, and their “Private Other” is much much lower. So it looks like it should be possible to get that lowered, but we might have third party libraries or code that’s creating buffers that aren’t properly released to the system or something like that. I am not an expert in native Android development and debugging, so maybe I’m missing something obvious.

I would appreciate any advise on tools, techniques, or knowledge on finding out what’s up with the memory use, and then thinking about how to reduce it.

Thanks!

Screenshots
Unity memory profiler:

adb memory usage:

[alexeyzakharov]

I believe in 2019.4 the “Total” memory size we’ve been using meant in fact total system memory usage which included all other app on the device. We’ve fixed that in 2020 and use Pss set to build a breakdown. This can explain the discrepancy between 3.6GB and 790MB

Native Heap, GL mtrack and Unknown is what memory profiler directly and indirectly tracks.

Graphics and Graphics driver should correlate with GL mtrack - 79.8MB vs 126MB.
Native Heap + Unknown should correlate with Managed Heap + Audio + Other + Profiler - ~400MB vs 476MB.

We definitely have gaps and some are quite tricky to cover (e.g. native plugins or gl driver allocations), but we are actively working on improving the tool and our tracking systems at the moment

In your case the “Other” part is definitely important and I would recommend trying Detailed view of Memory Profiler module - Unity - Manual: Memory Profiler module . That would allow to see some Unity internal subsystem in more details - I believe those are responsible for large portion of unknown.
(We are working on integration of this information into the Memory Profiler package too)

[BananaBug]

Thank you for proving these insights, they are very helpful to help look into this. I have however a few follow up questions around this if you don’t mind.

@alexeyzakharov

I believe in 2019.4 the “Total” memory size we’ve been using meant in fact total system memory usage which included all other app on the device. We’ve fixed that in 2020 and use Pss set to build a breakdown. This can explain the discrepancy between 3.6GB and 790MB

That makes sense, I thought it was intentionally showing all the rest of the memory not used by our game, maybe to indicate memory pressure, but I see now the untracked memory is supposed to show the missing amount to get to 790MB.

@alexeyzakharov

Native Heap + Unknown should correlate with Managed Heap + Audio + Other + Profiler - ~400MB vs 476MB.

Did you mean ~332MB VS 476MB? I assume the Graphics and Graphics Driver should not be included in it? Just checking to make sure I’m not making wrong assumptions and our textures aren’t ending up in two places.

@alexeyzakharov

In your case the “Other” part is definitely important and I would recommend trying Detailed view of Memory Profiler module - https://docs.unity3d.com/Manual/ProfilerMemory.html . That would allow to see some Unity internal subsystem in more details - I believe those are responsible for large portion of unknown.
(We are working on integration of this information into the Memory Profiler package too)

I had a look here, and it’s showing even less memory in there (in the detailed view, screenshot below). The assets part I can map to the the memory profiler package easily, and the “other” section in there is mostly the profiler. and there’s not much else in there.
One thing however I noticed, is that there’s ~100MB of “Total GC Allocated”. Am I right in understanding this matches the “Managed Memory” section of the memory profiler package? Is this allocations we make in c# code which is then potentially freed by us but not returned to the system?

Assuming I’m not mistaken there, and this managed memory is already account for in the memory profiler package snapshot, the main question remains, where is the difference between the 400+MB tracked memory, and the nearly 800MB adb is reporting. What kind of allocations are there that would not be tracked at all? Are we guaranteed that everything we allocate in c# will be tracked or are there known classes that we know for sure won’t be?

At the moment I’ve taken out all third party libraries (ads, facebook, analytics, google play services, etc) and that had an impact in adb on the “code” section, taking it from ~150 MB to ~50MB. (We’re also experimenting with upping the code stripping to medium to lower than even with third party libraries).

So there’s still a big chunk of memory unaccounted for. If we know the code section is my original screenshots were not tracked, it would come down to 790 - 143 = 647 - 412 = 235MB that’s untracked.
I’ve been testing without internet connection so there’s no additional downloads. What the game really is left doing during startup is the following:

• Load and show initial loading scene
• Instantiate systems and load data, which involves loading some TextAssets from Resources (async) and deserialising json (in a different thread), loading in scriptable objects which might have some prefabs linked (async), loading in json data from persistent data (some might be encrypted)
• (For Json we’re using newtonsoft json in case that matters)
• Writing json data in encrypted format to persistent data
• Load in main scene (async)
• Load in some textures and prefabs via resources, though this is a minority, most are linked in prefabs in scene or in a loaded scriptable object.

I’ve been doing all sorts of other experiments to try and see if particular actions trigger the increase in memory (or decrease by disabling certain systems), but so far, I’ve not found anything that makes any big difference.

Thanks for your time.

Profiler screenshot:

[BananaBug]

I did a bit more experimenting, and have broken up the initialisation flow to go step by step by me pressing a button, so I could see what specific parts allocate memory that might be unexpected. I’ve one example in screenshot below, which isn’t a lot of memory, but is an example of adb showing 3 times more memory being increased than the increase shown in Unity’s memory profiler package.

In screenshot below the top part is before a step, and the bottom part is right after that step. This step is asynchronously loading in a scriptableobject from resources (Resources.LoadAsync and yielding null and looping the coroutine until it is done). The scriptable object contains one public List (from generic collections) of a struct. The struct code is simple and is pasted below in a code block.

This scriptable object that’s loaded in from resources has 52 prefabs linked in that list. Each prefab has a few images (to be used in a canvas), but nothing particular in terms of scripts of components. As you can see from the screenshot, after this scriptable object is loaded in, the Unity memory profiler shows increase of about 10MB, pretty much all in graphics. this makes sense as all those linked prefabs have unique sprites, and this is acceptable to be in memory. However looking at the ADB printing on the left side, the increase is over 30MB. There’s about 10 in graphics, but also increase in Native Heap, Code, Private Other, and System. So for loading in a scriptable object from resources in an async matter, which links to a bunch of prefabs (that themselves are not in resources), what other memory is this allocating behind the scenes. I feel that maybe if I could figure out where that other memory is going, that could be a clue to other places where we do similar logic (other scriptable objects, and other linked prefabs etc).

PS: It should be noted that the adb memory consumption goes up when capturing a snapshot, so the screenshot below shows adb values after the first capture but before loading the scriptable object, and the second adb values are right after, before capturing the second snapshot. This should as much as possible the “net difference” between the two states.

The struct used in the List inside the scriptable object:

[Serializable]
    public struct MyStruct : IEquatable<MyStruct>
    {
        public string name;
        public GameObject prefab;

        public override bool Equals(object obj)
        {
            return obj is MyStruct objectiveView && Equals(objectiveView);
        }

        public override int GetHashCode()
        {
            return name.GetHashCode();
        }

        public bool Equals(MyStruct other)
        {
            return string.Equals(name, other.name) && (prefab == other.prefab);
        }
           
        public static bool operator ==(MyStruct lhs, MyStruct rhs)
        {
            return lhs.Equals(rhs);
        }

        public static bool operator !=(MyStruct lhs, MyStruct rhs)
        {
            return !(lhs == rhs);
        }
    }

Memory screenshot (left is adb values, right is unity profiler package; top is before loading scriptable object, bottom is after).

[alexeyzakharov]

@BananaBug

Did you mean ~332MB VS 476MB? I assume the Graphics and Graphics Driver should not be included in it? Just checking to make sure I’m not making wrong assumptions and our textures aren’t ending up in two places.

thanks for the clarification! you are totally right - gfx driver should not be included in Native Heap + Unknown, so ~332 VS 476MB

@BananaBug

What kind of allocations are there that would not be tracked at all? Are we guaranteed that everything we allocate in c# will be tracked or are there known classes that we know for sure won’t be?

All C# allocations which are part of the heap are tracked - those are e.g. new List, new Object. However right now we don’t track substantial portion of utility allocations made by Virtual Machine layer - Mono or IL2CPP. Such allocations are e.g. type metadata, generics metadata. That can use some memory in case you use generics or reflection with a pattern such as “iterate assemblies, iterate types, find specific type”

[BananaBug]

@alexeyzakharov

All C# allocations which are part of the heap are tracked - those are e.g. new List, new Object. However right now we don’t track substantial portion of utility allocations made by Virtual Machine layer - Mono or IL2CPP. Such allocations are e.g. type metadata, generics metadata. That can use some memory in case you use generics or reflection with a pattern such as “iterate assemblies, iterate types, find specific type”

Thanks for the extra insights @alexeyzakharov
We do use quite a bit of generics and have some reflection usage for some of our systems, especially where we’re deserializing json to the correct type based on a string identifier. Do you have any rough idea what kind of impact that could have in terms of size per type/class?

Have you had a chance to look at my last post? I’ve not had time to experiment more why loading a single scriptable object from resources with a bunch of linked prefabs was causing a large overhead of non-tracked allocations. I’ve been wondering if maybe the overhead can be avoided if I don’t make use of a scriptable object, or if I use addressables to load that object instead of loading it from resources, of if maybe I need to load the specific prefabs via addressables instead of having them all linked in 1 scriptable object. It’s not a quick and easy change to make, so if there was known memory overhead with any of this, that might help refactor parts of our system knowing it will have some impact.

[alexeyzakharov]

@BananaBug

Do you have any rough idea what kind of impact that could have in terms of size per type/class?

it very much depends on amount of types in the project - could be 10-20MB.

@BananaBug

Have you had a chance to look at my last post?

yes, however, not sure if I have an answer
The difference of Resources folder vs AssetBundles should be minimal tbh. From 2018 IIRC all players use same AssetBundle archive packaging mechanism. The main difference of Android platform is apk zip compression which might create additional 1MB buffer for decompression if you choose Default player compression. But that doesn’t explain 10MB. And if you use LZ4 already that doesn’t explain it at all.

[BananaBug]

@alexeyzakharov

it very much depends on amount of types in the project - could be 10-20MB.

yes, however, not sure if I have an answer
The difference of Resources folder vs AssetBundles should be minimal tbh. From 2018 IIRC all players use same AssetBundle archive packaging mechanism. The main difference of Android platform is apk zip compression which might create additional 1MB buffer for decompression if you choose Default player compression. But that doesn’t explain 10MB. And if you use LZ4 already that doesn’t explain it at all.

Thanks, it’s good to know if certain things are known to have a big impact or not. It sadly means there’s not quite a smoking gun we’ve found yet. We are currently using default player compression, so it might be worthwhile to run some experiments with that as well.

I’ll be off work for a while and have some colleagues look into this with fresh eyes. I’ve forwarded them the info you’ve provided and hopefully they’ll be able to find more clues that I’ve overlooked so far. Thanks again!

[Johanhan]

Hello,

@BananaBug

I did a bit more experimenting, and have broken up the initialisation flow to go step by step by me pressing a button, so I could see what specific parts allocate memory that might be unexpected. I’ve one example in screenshot below, which isn’t a lot of memory, but is an example of adb showing 3 times more memory being increased than the increase shown in Unity’s memory profiler package.

did you ever found the source of the additional memory loaded, when referencing a prefab in scene? Because in our project we have the same problem and can’t solve it.