Debug Builds on Android Busting Memory Limits, Crashing

I have a fairly massive game that uses AssetBundles. Recently hit an issue where the debug mono builds are crashing during gameplay when I load some of my larger assets. They were unchanged recently, so I must’ve pushed memory over somewhere else and it was the tipping point. But it lead me down a rabbit hole observing Unity behavior in my project.

For reference, I’m on Unity 2019.4.3f1

My initial scene when booting up the game that leads to a UI menu shows this in the profiler. Now, this profile is the first frame that shows up in the Unity Profiler after waiting to attach the debugger on Android:

Used Total: 377.1 MB   Unity: 134.0 MB   Mono: 58.3 MB   GfxDriver: 145.9 MB   Audio: 3.5 MB   Video: 0 B   Profiler: 35.3 MB  
Reserved Total: 429.3 MB   Unity: 171.0 MB   Mono: 73.4 MB   GfxDriver: 145.9 MB   Audio: 3.5 MB   Video: 0 B   Profiler: 35.5 MB  
Total System Memory Usage: 3.46 GB 

Textures: 251 / 133.8 MB
Meshes: 57 / 20.4 MB
Materials: 163 / 477.0 KB
AnimationClips: 48 / 0.8 MB
AudioClips: 8 / 2.7 MB
Assets: 52014
GameObjects in Scene: 134
Total Objects in Scene: 570
Total Object Count: 52584
GC Allocations per Frame: 2040 / 197.0 KB

Here is another profile but this one is Standalone windows, again, first frame that shows up:

Used Total: 423.0 MB   Unity: 181.7 MB   Mono: 76.0 MB   GfxDriver: 154.9 MB   Audio: 3.9 MB   Video: 0 B   Profiler: 6.4 MB 
Reserved Total: 507.1 MB   Unity: 258.0 MB   Mono: 82.3 MB   GfxDriver: 154.9 MB   Audio: 3.9 MB   Video: 0 B   Profiler: 8.0 MB 
Total System Memory Usage: 0.76 GB 


Textures: 246 / 142.6 MB
Meshes: 65 / 20.5 MB
Materials: 153 / 0.6 MB
AnimationClips: 56 / 0.8 MB
AudioClips: 8 / 2.7 MB
Assets: 52819
GameObjects in Scene: 146
Total Objects in Scene: 648
Total Object Count: 53467
GC Allocations per Frame: 11 / 0 B

The Reserved Total is concerning me the most. It seems quite high especially on the GfxDriver no?

So what is it showing when I crash? Well, the profiler is showing much much worse:

Used Total: 1.14 GB   Unity: 0.60 GB   Mono: 182.4 MB   GfxDriver: 347.1 MB   Audio: 4.0 MB   Video: 0 B   Profiler: 18.8 MB 
Reserved Total: 2.00 GB   Unity: 1.44 GB   Mono: 204.8 MB   GfxDriver: 347.1 MB   Audio: 4.0 MB   Video: 0 B   Profiler: 21.0 MB 
Total System Memory Usage: 3.46 GB 


Textures: 848 / 367.8 MB
Meshes: 71 / 25.3 MB
Materials: 301 / 503.0 KB
AnimationClips: 50 / 4.1 MB
AudioClips: 8 / 2.8 MB
Assets: 58073
GameObjects in Scene: 125
Total Objects in Scene: 867
Total Object Count: 58940
GC Allocations per Frame: 491327 / 33.5 MB

Am I crazy? Why are the Unity numbers so pumped at launch and super high during my gameplay? I presume the “unity” number is encompassing the textures and other numbers. I’ve tried to take memory snapshots using the Memory Profiler and the tree really shows me that during gameplay right before crashing:

Used Total: 3.43 GB   Unity: 1.50 GB   Mono: 321.3 MB   GfxDriver: 1.03 GB   Audio: 4.0 MB   Video: 0 B   Profiler: 0.58 GB  
Reserved Total: 3.83 GB   Unity: 1.78 GB   Mono: 424.4 MB   GfxDriver: 1.03 GB   Audio: 4.0 MB   Video: 0 B   Profiler: 0.60 GB  
Total System Memory Usage: 5.77 GB  

Memory Profiler Tree view:
Textures: 1GB
Animation: 260MB
Mesh: 190MB
RenderTexture: 100MB
Font: 60MB
Shader: 60MB
MonoScript: 66MB
Particle System: 38MB

Everything else is magnitudes smaller. The above is 1.7 to 1.8GB at most.

Let’s say my game is targeting 4GB RAM devices like a Pixel 2.

  • I’ve observed the OS can sometimes take up to 1GB of memory! Let’s assume that’s worst case. A fresh reboot of a Pixel 2 knocks it down to like 200-300MB and this sometimes has me crashing less.
  • Debug Mono & Unity are taking up to 500MB on boot, but as I progress into game that reserved number runs up to about 1GB+. At crash, I saw Reserved Total: 2.00GB!!
  • That leaves my game only 1-2GB~ RAM for textures and other stuff which apparently is insufficient

What should I be doing? Obviously, I need to knock down texture memory in general to support a healthy build. But, even if I eliminate textures entirely I’m going to keep hitting crashes if I can’t figure out how to wrangle the Unity player memory in.

I noticed that even if I’m on a device with more than 4GB of ram, if I hit 4GB of used memory, it will crash. 32bit problem?

Bump

Bump

Hi
First some questions:

  • Are you using any Native Plugins?

  • Are you using reflection code during runtime?

  • Are you using any Unity or Asset Store packages that deal with post effects, textures, video, camera, etc?

  • Have you checked the Memory Profiler Package’s Memory Map (also in a diff from start to end or between level loads)?
    Pay attention to the blue sections there (Managed Memory). You could be hitting fragmentation issues. This post of mine should help getting some clarity on that view while I’m fleshing out the documentation write-up. Also generally that Profiler Previews subforum might contain some more helpful information for you.

  • QUOTE=“OmegaNemesis28, post: 6129632, member: 160569”]GC Allocations per Frame: 491327 / 33.5 MB[/QUOTE]
    Is this happening because of a LevelLoad or could this be the cause of your crash? because 33.5MB allocs in any normal frame sounds pretty mad to me.

  • If you are observing the Profiler Window’s Memory Profiler Module while playing: does the memory usage steadily grow or are there particular moments in which it bumps up? What are those and have you taken Memory Snapshots with the Memory Profiler Package to diff and compare before and after?

  • Is this a 32 bit or 64 bit build

Also, what does your Player log say after the crash? could be those last allocs are a huge bunch of Debug Logs?

  • Native plugins? I use a wide array of stuff from Unity.Mathematics to Entities to the preview transport layer if that’s what you mean. I can get a list of those packages.

  • Reflection code: Not during gameplay no.

  • Packages with post effects and textures or camera? I think the only thing used during gameplay for UI is DOTween. Project is URP though worth mentioning.

  • Yes, the memory map is what I was referring to when talking about the memory profiler (sorry). At most it was just displaying what the tree was - 1GB of textures, and the rest nothing bigger than 260MB. The profiler doesn’t give insight to Unity’s reserved memory or Mono as far as I can see.

  • This is long after LevelLoad. Specifically, I’m spawning characters through assetbundles when the crash was happening which is probably why the allocations are so large that frame. But it’s just that frame. I intend to make async loading soon so that 33MB is spread across multiple frames.

  • It bumps up at places you’d expect it to, but app launch does it the most. Memory Profiler does not reveal what it is, I can’t take a profile before app launch. Post launch, it only shows a few textures but nothing massive. So I have no insight to what Unity is reserving or doing.

  • Android is on Mono .Net 4.x ARM v7 targeting Lolipop 5.1. Not sure how to specify 64 bit.

Player log didn’t show anything, it would just end. It’s just a tombstone callstack. Sometimes it would change to something not exactly the same but similar mentioning GfxDeviceWorker. Here’s one example:

07-23 02:29:29.375  5876  5876 E CRASH   : other thread is trapped; signum = 1107-
07-23 02:29:29.391  5876  5876 E MessageQueue-JNI: java.lang.Error: *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
07-23 02:29:29.391  5876  5876 E MessageQueue-JNI: Version '2019.4.3f1 (f880dceab6fe)', Build type 'Development', Scripting Backend 'mono', CPU 'armeabi-v7a'
07-23 02:29:29.391  5876  5876 E MessageQueue-JNI: Build fingerprint: 'samsung/star2qltecs/star2qltecs:10/QP1A.190711.020/G965WVLS7DTE1:user/release-keys'
07-23 02:29:29.391  5876  5876 E MessageQueue-JNI: Revision: '14'
07-23 02:29:29.391  5876  5876 E MessageQueue-JNI: ABI: 'arm'
07-23 02:29:29.391  5876  5876 E MessageQueue-JNI: Timestamp: 2020-07-23 02:27:49-0400
07-23 02:29:29.391  5876  5876 E MessageQueue-JNI: pid: 5876, tid: 6481, name: UnityGfxDeviceW  >>> com.game.game <<<
07-23 02:29:29.391  5876  5876 E MessageQueue-JNI: uid: 10473
07-23 02:29:29.391  5876  5876 E MessageQueue-JNI: signal 11 (SIGSEGV), code 1 (SEGV_MAPERR), fault addr 0x0
07-23 02:29:29.391  5876  5876 E MessageQueue-JNI: Cause: null pointer dereference
07-23 02:29:29.391  5876  5876 E MessageQueue-JNI:     r0  00000000  r1  00000000  r2  0004c2b8  r3  00000000
07-23 02:29:29.391  5876  5876 E MessageQueue-JNI:     r4  ee40dd20  r5  000014c9  r6  00000002  r7  00008f37
07-23 02:29:29.391  5876  5876 E MessageQueue-JNI:     r8  be4d38b0  r9  00000000  r10 c01e1a38  r11 0004c338
07-23 02:29:29.391  5876  5876 E MessageQueue-JNI:     ip  00000000  sp  b80e9ed0  lr  bf4d40a8  pc  ef7abdf8
07-23 02:29:29.391  5876  5876 E MessageQueue-JNI:
07-23 02:29:29.391  5876  5876 E MessageQueue-JNI: backtrace:
07-23 02:29:29.391  5876  5876 E MessageQueue-JNI:       #00 pc 0005ddf8  /apex/com.android.runtime/lib/bionic/libc.so (memset_a7+48) (BuildId: f7fcf1e3a2e994c617c9b9b888e91865)
07-23 02:29:29.391  5876  5876 E MessageQueue-JNI:       #01 pc 009760a4  /data/app/com.game.game-_byFZYbTU3o6znQmpTvgJw==/lib/arm/libunity.so (ApiGLES::ClearBufferSubData(unsigned int, gl::BufferTarget, long, long)+148) (BuildId: 6bb88d5fd94d81c332ebfbe5a37085a2c7a40ad0)
07-23 02:29:29.391  5876  5876 E MessageQueue-JNI:       #02 pc 009518dc  /data/app/com.game.game-_byFZYbTU3o6znQmpTvgJw==/lib/arm/libunity.so (DataBufferGLES::smile:ataBufferGLES(BufferManagerGLES&, unsigned int, DataBufferGLES::BufferUsage, bool)+288) (BuildId: 6bb88d5fd94d81c332ebfbe5a37085a2c7a40ad0)
07-23 02:29:29.391  5876  5876 E MessageQueue-JNI:       #03 pc 009523f8  /data/app/com.game.game-_byFZYbTU3o6znQmpTvgJw==/lib/arm/libunity.so (BufferManagerGLES::AcquireBuffer(unsigned int, DataBufferGLES::BufferUsage, bool)+276) (BuildId: 6bb88d5fd94d81c332ebfbe5a37085a2c7a40ad0)
07-23 02:29:29.391  5876  5876 E MessageQueue-JNI:       #04 pc 00951748  /data/app/com.game.game-_byFZYbTU3o6znQmpTvgJw==/lib/arm/libunity.so (BufferGLES::EnsureBuffer(DataBufferGLES*&, unsigned int, DataBufferGLES::BufferUsage)+116) (BuildId: 6bb88d5fd94d81c332ebfbe5a37085a2c7a40ad0)
07-23 02:29:29.391  5876  5876 E MessageQueue-JNI:       #05 pc 00951294  /data/app/com.game.game-_byFZYbTU3o6znQmpTvgJw==/lib/arm/libunity.so (BufferGLES::Initialize(void const*)+32) (BuildId: 6bb88d5fd94d81c332ebfbe5a37085a2c7a40ad0)
07-23 02:29:29.391  5876  5876 E MessageQueue-JNI:       #06 pc 0095fa90  /data/app/com.game.game-_byFZYbTU3o6znQmpTvgJw==/lib/arm/libunity.so (GfxDeviceGLES::InitializeBufferInternal(GfxBuffer*, void const*, GfxUpdateBufferFlags)+28) (BuildId: 6bb88d5fd94d81c332ebfbe5a37085a2c7a40ad0)
07-23 02:29:29.391  5876  5876 E MessageQueue-JNI:       #07 pc 003715ff  /data/app/com.game.game-_byFZYbTU3o6znQmpTvgJw==/lib/arm/libunity.so (GfxDeviceWorker::RunCommand(ThreadedStreamBuffer&)+25830) (BuildId: 6bb88d5fd94d81c332ebfbe5a37085a2c7a40ad0)
07-23 02:29:29.391  5876  5876 E MessageQueue-JNI:       #08 pc 00371ccb  /data/app/com.game.game-_byFZYbTU3o6znQmpTvgJw==/lib/arm/libunity.so (GfxDeviceWorker::RunExt(ThreadedStreamBuffer&)+26) (BuildId: 6bb88d5fd94d81c332ebfbe5a37085a2c7a40ad0)
07-23 02:29:29.391  5876  5876 E MessageQueue-JNI:       #09 pc 00371c91  /data/app/com.game.game-_byFZYbTU3o6znQmpTvgJw==/lib/arm/libunity.so (GfxDeviceWorker::Run()+96) (BuildId: 6bb88d5fd94d81c332ebfbe5a37085a2c7a40ad0)
07-23 02:29:29.391  5876  5876 E MessageQueue-JNI:       #10 pc 0036aed3  /data/app/com.game.game-_byFZYbTU3o6znQmpTvgJw==/lib/arm/libunity.so (GfxDeviceWorker::RunGfxDeviceWorker(void*)+2) (BuildId: 6bb88d5fd94d81c332ebfbe5a37085a2c7a40ad0)
07-23 02:29:29.391  5876  5876 E MessageQueue-JNI:       #11 pc 007f2e57  /data/app/com.game.game-_byFZYbTU3o6znQmpTvgJw==/lib/arm/libunity.so (Thread::RunThreadWrapper(void*)+342) (BuildId: 6bb88d5fd94d81c332ebfbe5a37085a2c7a40ad0)
07-23 02:29:29.391  5876  5876 E MessageQueue-JNI:       #12 pc 000abcbb  /apex/com.android.runtime/lib/bionic/libc.so (__pthread_start(void*)+20) (BuildId: f7fcf1e3a2e994c617c9b9b888e91865)
07-23 02:29:29.391  5876  5876 E MessageQueue-JNI:       #13 pc 00062dc3  /apex/com.android.runtime/lib/bionic/libc.so (__start_thread+30) (BuildId: f7fcf1e3a2e994c617c9b9b888e91865)
07-23 02:29:29.391  5876  5876 E MessageQueue-JNI:
07-23 02:29:29.391  5876  5876 E MessageQueue-JNI:     at libc.memset_a7(memset_a7:48)
07-23 02:29:29.391  5876  5876 E MessageQueue-JNI:     at libunity.ApiGLES::ClearBufferSubData(unsigned int, gl::BufferTarget, long, long)(ClearBufferSubData:148)
07-23 02:29:29.391  5876  5876 E MessageQueue-JNI:     at libunity.DataBufferGLES::smile:ataBufferGLES(BufferManagerGLES&, unsigned int, DataBufferGLES::BufferUsage, bool)(DataBufferGLES:288)
07-23 02:29:29.391  5876  5876 E MessageQueue-JNI:     at libunity.BufferManagerGLES::AcquireBuffer(unsigned int, DataBufferGLES::BufferUsage, bool)(AcquireBuffe
07-23 02:29:29.392  5876  5876 D AndroidRuntime: Shutting down VM
07-23 02:29:29.398  5876  5876 E AndroidRuntime: FATAL EXCEPTION: main
07-23 02:29:29.398  5876  5876 E AndroidRuntime: Process: com.game.game, PID: 5876
07-23 02:29:29.398  5876  5876 E AndroidRuntime: java.lang.Error: *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
07-23 02:29:29.398  5876  5876 E AndroidRuntime: Version '2019.4.3f1 (f880dceab6fe)', Build type 'Development', Scripting Backend 'mono', CPU 'armeabi-v7a'
07-23 02:29:29.398  5876  5876 E AndroidRuntime: Build fingerprint: 'samsung/star2qltecs/star2qltecs:10/QP1A.190711.020/G965WVLS7DTE1:user/release-keys'
07-23 02:29:29.398  5876  5876 E AndroidRuntime: Revision: '14'
07-23 02:29:29.398  5876  5876 E AndroidRuntime: ABI: 'arm'
07-23 02:29:29.398  5876  5876 E AndroidRuntime: Timestamp: 2020-07-23 02:27:49-0400
07-23 02:29:29.398  5876  5876 E AndroidRuntime: pid: 5876, tid: 6481, name: UnityGfxDeviceW  >>> com.game.game <<<
07-23 02:29:29.398  5876  5876 E AndroidRuntime: uid: 10473
07-23 02:29:29.398  5876  5876 E AndroidRuntime: signal 11 (SIGSEGV), code 1 (SEGV_MAPERR), fault addr 0x0
07-23 02:29:29.398  5876  5876 E AndroidRuntime: Cause: null pointer dereference
07-23 02:29:29.398  5876  5876 E AndroidRuntime:     r0  00000000  r1  00000000  r2  0004c2b8  r3  00000000
07-23 02:29:29.398  5876  5876 E AndroidRuntime:     r4  ee40dd20  r5  000014c9  r6  00000002  r7  00008f37
07-23 02:29:29.398  5876  5876 E AndroidRuntime:     r8  be4d38b0  r9  00000000  r10 c01e1a38  r11 0004c338
07-23 02:29:29.398  5876  5876 E AndroidRuntime:     ip  00000000  sp  b80e9ed0  lr  bf4d40a8  pc  ef7abdf8

Once again, sorry for the delay in my answer. I might’ve tried to juggle to many ongoing threads again

No, I meant native libraries in your Plugins folder or the Plugins folder of any package. I assume most of these packages would be purely C# code and therefore all their memory usage is fully captured in the snapshot.

Even using Reflection during startup can bloat memory because type meta data needs to be loaded in. That shouldn’t lead to memory growth over time though. Still if you use it in between of scenes or something, it could still grow that Mono memory footprint. Beyond that, it might be down to fragmentation in Mono.

Not as a summary but all the dark blue regions in the Memory Map are Mono Reserved memory regions, aka reserved memory.

Also, yeah that crash looks like an OutOfMemory crash…

It could be something to do with your Asset bundle setup?

Also, FYI, that Total System Memory Usage value is currently bugged on Android and showing overall memory used on the device, not just the current app. That fix is in the process of landing/shipping and we’ll adjust the Manual for the Memory Profiler Module to explain that on Android it will show Residual Shared memory from that fix onward.

Textures yes but not Audio, Video and Profiler. The Manual update should clarify this too.