Unity 2022.3.15f1.
Tried using the Memory Profiler recently on our development and editor builds with very poor results - Only a couple times of 10+ attempts did I successfully open a snapshot. All others would sit on Crawling GC Handles for 30+ mins before I quit.
I downgraded to v1.0 and it opens the snapshots within a couple of minutes.
Hi Scott,
Could you do me a favor please and check if you modify version 1.1 by adding
if(i >= 1000) break;
here (i.e. ManagDataCrawler.cs line 642)?
If that solves the issue, then it’s this bug that we’re aware of and for which I’m currently working on a fix (spoiler alert, the above isn’t really one). The issue there is that we previously did not correctly scan the fields of structs/value types for potential pointers to managed objects, possibly missing out on references to some objects or even missing entire managed objects on the heap. 1.1. fixed that but we somehow didn’t expect some worst case scenarios here. I.e. I’ve seen how having arrays of structs with over a billion entries can congest in the code linked to above. There is a question of whether or not such code should use NativeArrays or other Native Collections instead but the Memory Profiler should obviously still be able to deal with that.
If this hotfix does not help, I’d like to ask you to file a bug report, attach a snapshot that won’t open, and ping me just the issue ID as soon as you get it, so that I can look at this asap 
Yep that “fixes” it.
Note that we don’t use any Native Arrays or Collections if you mean ones of the Unity kind. Though we do use a lot of big lists for things as we’ve written our own ECS system, but its all C# stuff. Certainly we’d have a few list that are too big (one of the reasons we’re trying to use the tool!), but surely not billions big.
Can you provide an ETA on a new preview build with your fix in it at this point?
Thanks for confirming the “fix”, that could be a workaround until we push a fix version.
I’d recommend Native Collections over managed ones. The Boehm GC treats every pointer-sized field as potential pointer to parse for garbage collection, so when the Memory Profiler starts getting perf troubles like this, the GC is likely suffering as well, without providing any benefits. Larger continuous managed memory allocations used for these will also contribute to Managed Memory Fragmentation.
No ETA but I’ll hurry the fix along. We need to have a bit if a huddle on the release and version strategy for the next versions but I’m hopeful we’ll have something up soon.
Our ECS has specifically been built to allocate contiguous large arrays for itself, duplicated per CPU thread for multi-threading. Its been in production for years now and we’re in the last few months prior to release, so unlikely we’ll be able to look into whether Unity Native Arrays are worth a genuine look at this stage unfortunately.
That said there could be memory issues we’re experiencing - That’s why I’m diving into the Memory Profiler right now.
Case in point - We’re currently experiencing either a memory leak or a massive GC thrashing that we’re trying to resolve. We’re experiencing massive memory bloat of 200-300MB added per reload into the same level from main menu multiple times.
GC.GetTotalMemory(false) reports for example 2.5GB when the memory used as reported by Task Manager is 5GB. Even after the below GC Collect. What could be a cause of this? Fragmented free heap memory? Though a compaction (below code) doesn’t seem to improve it.Update: Running the following GC Collect does not really affect the total memory usage as reported by Task Manager:
Debug.Log($"Performing forced GC Collect. Current Total Memory: {GC.GetTotalMemory(false) / 1000000f}MB.");
GCSettings.LargeObjectHeapCompactionMode = GCLargeObjectHeapCompactionMode.CompactOnce;
GC.Collect(GC.MaxGeneration, GCCollectionMode.Forced, true, true);
Debug.Log($"Finished forced GC Collect. New Total Memory: {GC.GetTotalMemory(false) / 1000000f}MB.");
Your assuming the Boehm GC version shipping with Unity cares about any of those settings. It doesn’t. It’s way to old for that. It’s non-generational, non-compacting/non-moving/non-defragmenting.
The one extra call that could make a difference is GC.WaitForPendingFinalizers, though the pure usage of Finalizers is, in my eyes, an anti-pattern in Unity that will bloat and fragment your memory even harder, especially if you revive objects in a Finalizer.
You’re hitting pretty much what I’d expect here: Memory fragmentation because of way too large allocations. I’d seriously recommend switching to Native collections. There’s a reason DOTS and Unity’s ECS use them over managed arrays.
Also, the Memory Profiler can currently not help you see what in the empty fragmented heap space would be objects that are ready to collect, vs those already collected. That is because it doesn’t have data on the free lists and buckets, and through that crawler code you had to modify, just finds objects that are rooted and those referenced by rooted objects (or x levels down the line). Non-referenced objects are just not found, and neither are those only referenced by thread stacks.
Further, unused pages of heap memory are freed during every 6th GC Cycle (warning, implementation detail and therefore could be subject to change) on a page by page basis. Calling GC.Collect 6 times could tell you if those pages are actually empty for debugging purposes, but that, as well as any manual call to GC.Collect is nothing I’d encourage as it’s just as well liable to fragment the heap harder. Because it would be triggered automatically when there’s not enough empty heap space for an allocation. And once the pages are unloaded, you loose that empty contiguous heap space that could’ve been used for the next big allocation.
The incremental GC is neat for small, non-trivially avoidable allocations in moments where a synchronous GC spike would be bad. But it also means, when it triggers (because there’s not enough space) a new heap section needs to be allocated for the new object right now even if the triggered incremental GC might free enough space a few frames down the line. This further fragments your memory.
Version 0.7 of the package still has the fragmentation page that can be somewhat useful, if armed with enough knowledge of how the Boehm GC works, to analyze this all a bit, but it’s also lacking some improvements to the crawler. Particularly: if any of those arrays of structs contained reference to managed objects, or even just an 8 byte (on 64 bit platforms) value that happens to match the address of an object, that memory isn’t technically empty (at least to the eyes of the Boehm GC), but with that stopgap solution, just not found by the package’s crawler.
I do have some good news though, I’ve come up with an approach to speed up that array of structs processing that hopefully speeds it up enough (and allows for async multithreading it) that we might not have to cut our losses after x entries in an array and left to guess if Boehm might’ve misinterpreted the second to last element in it as a pointer.
Firstly, that’s a fucking great amount of information mate. Thanks for taking the time.
Unfortunately GC.Collect() 6 times doesn’t do much.
The other thing that worries me is that snapshot B has +400,000 managed objects. The 2 snapshots should really be near-identical. Just to be sure I’m reading it right - That’s certainly our code leaking objects that are somehow still referenced somewhere right?
I’ll take a look at the v0.7 fragmentation page to help validate the theory. How does one actually download previous packages these days?
Apologies for the delay:
You can try turning it off entirely, or turn it of temporarily from code while you churn over (allocate or reallocate) big arrays.
That’s 2.5 GB of MANAGED memory, I assume your app also uses some GameObjects and other Unity subsystems (Native Memory) Textures (Native and Graphics Memory), DLLs (Native though likely non-resident memory) and any kind of C# Type data (Managed Virtual Machine Memory, which is Native in IL2CPP and “Managed” in Mono), the latter doesn’t need reflection, just using code, though Reflection might balloon this usage up unnecessarily.
As I explained over in this thread , you could also use ProfilerRecorders to monitor GC Used Memory and GC Reserved Memory or check the Memory Profiler Module’s manual page for more counters to check that would include the Native and Graphics memory, though the latter only in Development builds. GC Used Memory would likely be the same as what the GC.GetGetTotalMemory() API returns and as that documentation page also says, this excludes fragmentation. Fragmentation would be included in the Reserved counter.
Yes, the References panel should tell you what is keeping them around, but yeah, I’d assume it’s most likely something in your code.
Page size is likely 4 KB on your platform, so any blue strips that survived 6 GCs and are entirely empty, those have something pointing to them that the memory Profiler missed. Either something stack local, or in those arrays that weren’t fully processed.
All 4KB blocks that are mostly empty are fragmented and will stick around because of the few objects still in them.
Hi there,
I’m one of Scott’s work mates and the main author of our ECS, which is based by its nature on large struct arrays.
So we look forward to a fix for this case.
I was trying to use the memory profile quite a bit, but I still struggle to really understand the output it creates and concrete documentation about comparing memory snapshots is quite scarce.
So it wasn’t very helpful yet to find our memory leak(s), since there are many unknowns to make sense of the output of the snapshot comparison of “All Of Memory”.
The leaks occur due to loading and quitting the same level again and again, so I did snapshots after each level load for some loads.
To avoid any effects of lazy loading first time operations, I tried to compare the snapshot after second load with a later one.
What we see is a huge increase in the sub category “Untracked*”, a smaller increase of “Reserved”, and a small increase in “Managed” and the latter is even negative at times. As of yet we are unsure what this means.
Description shown when clicking on “Untracked*”:
Can you tell us more about this “Untracked*” type and what this what we see here actually means?
How can our code create so much untracked memory?
Obviously we tried to make use of the details viewed in the “Managed” category, but it doesn’t make much sense or maybe is even incorrect.
For example it found a “diff” on an array used in our terrain map between second and fourth level load.
If I understand the output correctly, then the tool tells us there wasn’t a terrain map in snapshot A(second load), but there is now one in the snapshot B (fourth load).
But this can not be true, since there will be always a terrain map or there wouldn’t be a game.
Almost all the diffs are based on such arrays popping out of nothing not recognized in the first snapshot but present in the second, while other disappear but should be there (or the game wouldn’t work).
So what is this “Diff” actually based on and how can it help in any way to find a leak?
We have an hard time to leverage the tool to help with finding leaks.
By just looking at the basic Profiler’s memory section instead, we could figure the leak could have to do with save game serialization.
The Memory Profiler - if it worked correctly, or it would be better understood by us - respectively, it seems it could be potentially a way better tool to find the leak, but so far we fail to interpret what it is actually telling us.
Any pointers on these questions would be much appreciated.
That’s not what it is trying to tell you. It’s only telling you something about the SplarseIntegetSet object. The Map object is just a direct referee to this object. SplarseIntegetSet is a managed object and has (at least) two different instances at different addresses in memory since it’s getting created new for the reentry. The Memory Profiler has no chance to know your mental concept of “this is the same object”, because it isn’t and there are no hints it could use to make that determination.
You could see if an object of the same type has been delete between the snapshots and make the mapping in your head. If there is none, maybe it’s kept and you might want to turn on the option to “Show Unchanged” in order to see that you’ve leaked it, and then check the references to it to see why it was leaked.
You could also find that Map Object (search by type name was added in 1.1) and check the value of the field holding the SplarseIntegetSet and see how that changed from the version in A to the one in B.
I have plans to make it a bit more intuitive that could help in this instance as it’d be using the strongest binding root to structure the data, sort of providing you with that “understanding” that your level contains a map. Though that’s a bit further off than the perf regression fix. I got sidetracked on something else for a bit but am back on this and hoping to be able to speed up processing so much that this super stopgap workaround won’t be necessary, but have yet to get it to run and see if that hypothesis holds water.
I’ve noted your comment about the documentation, but really, it’d make more sense to get to that improved workflow on the double rather than document it better, as it’d have to be rewritten and currently would need quite some more complex explanations that’d no longer be necessary then.
So I’d rather revisit the documentation after that work is done.
For Unity versions before 2022.2 we’ve put that list into the tool there and in the first page of the package documentation for the versions up until 0.7:
As of 2022.2+ the 2nd and 3rd element have been taken off that list.
The first one can be avoided if the native Plugin uses the Native Plugin uses our Memory Manager API to allocate it’s memory.
Thanks for this list.
In our case it shows by far the largest positive diff between loads.
As far as I’m aware, the stack wouldn’t grow each new fresh load, there would be no reason for it. We do not use currently any Marshal.Alloc… calls in our project by ourselves.
With 2nd and 3rd off the List “Native Plugin allocations” would be the only one left.
So does that mean we are using a native plugin that leaks? And eventually not destroyed GOs dangling around, which are “DontDestroyOnLoad” (we switch scene between levels)?
I will see tomorrow if I can now make better sense of it.
For now big thanks for taking the time to leave these important pointers.
That or something related to the math of the approximation mentioned in your screenshot. But I don’t get the DontDestroyOnLoad comment, as that would be tracked.
@MartinTilo we discovered a leak during our scene reloads and plugged it, however we still have a colossal amount (1.6GB) of memory sitting in “untracked memory” and no strong ideas on how to determine what it is or how to reduce it. We don’t have any native plugins, this is just a few minutes into gameplay on a development PC build (not IL2CPP).
The only lead I have is that we currently spawn some 150,000 gameobjects, with a lot of that for cached UI RectTransforms and components. I do wonder if that would outlay a lot of native Unity buffers. But I don’t know.
If you have any further investigation paths we could take that would be amazing! Or could this be a red herring?
My colleague went a bit into the details of 1.1.0 + 2023.1+ and how that lists details for Untracked in his United talk, maybe that helps?
As for the opening time bug / ManagedDataCrawler speedup work, that is still ongoing. It’s gotten a bit more complex than I’d have hoped but the basic structure looks solid now, so I’ll hopefully “just” have to debug out the bugs in that solution and make sure the data is correct. That said, it should be a nice speedup for the opening times.
Thanks. Yeah we’d already watching the talk - Unfortunately not much new info vs whats in this thread already.
Looking forward to the next version of the Memory Profiler! But yeah the “Untracked Memory” is currently our biggest problem and AFAIK despite all our efforts so far there is no way to further isolate what that is.
Have you tried updating to 2023 to take a capture from that to see what it says untracked is made up off?
I.e. just upgrading a branch for profiling, making a build with 2023 and profiling that
