How can developers diagnose excessive "Untracked Private" ram use?

[colinleet]

Hello, I’ve been monitoring an issue in my VR project (using HDRP) where what appears to be excessive “untracked - private” RAM usage occurs. I’ve reached the limits of my own skills trying to diagnose the cause of this. Specifically, within my main scene—on frame 0, both in builds and when entering Play mode in the Editor—there is a giant allocation of around 3 GB of unmanaged private RAM. This represents about 60% of my game’s total RAM usage, and I have no idea how to control or diagnose it.

My primary concern: In all test scenes and scenarios I’ve tried, “unmanaged - private” RAM usage never falls below about 40% of the total RAM usage, as reported by Task Manager. I believe the total listed under “All of Memory” might include graphics memory on the GPU, but I’m not certain.

I found very little information on what could cause “untracked - private” RAM. This forum post was the most helpful resource, which is why I’m tagging @MartinTilo. Your first response there helped me start narrowing down potential causes in my project.

To summarize, you listed these possible causes:

1. Native Plugin allocations
2. The size of Executable and DLLs
3. Stack Memory
4. VMs using IL2CPP (not applicable, since I’m using Mono)
5. Memory allocated using Marshal.AllocHGlobal (which I’ve never used, so if this were the cause, it’d have to be one of the external DLLs)

This left me with the first three items as possible culprits. I spent a day testing various combinations to see if any might be responsible.

I use three C/C++ precompiled plugins that could add to “untracked private” memory:

• Steam Audio
• SQLite
• Undertone

All tests were run on the latest version of Unity (6.0.31f1) with the latest stable package versions. All numbers reported are from Development Windows builds—Release builds show the same issues.

Next, I included the three native plugins. As expected, total RAM usage doubled to just under 1 GB, in line with what I anticipated from “untracked private” source #2 (the plugins’ uncompressed size is about 500 MB across their DLLs). This suggests that none of them individually caused the specific spike in my main project. What’s odd is that the ratio of “untracked private” stayed consistent at about 40%.

Here are the two above tests under the “compare snapshots view”:

At this point, I was clueless about potential rabbit holes I could even investigate. In my main scene, the “untracked private” RAM is 2.75 GB, while the test scenario with the same set of DLLs uses only 420MB. Both load and run the same 3rd party DLLS, yet the memory usage differs drastically.

In desperation, I tried importing the full demo scene into my main project’s environment. I compiled the XRIK test scene as the first scene in the build list to see if that changed anything. It did—but not favorably. “Unmanaged private” RAM increased to about 700 MB / 1.2 GB (~58%) when loading directly into the XRIK scene (with my project’s content present but not loaded):

For comparison, here is an identical build where my scene was the first in the build list (so it loads first). I mostly used the same post-processing stack in both scenes to see if it mattered—it did not.

In the above comparison, “A” represents the initial build (first photo in this post) loading directly into my main scene (showing the severe issue), while “B” is the snapshot from the previous photo.

I’m truly at a loss for what could be causing this issue. I’m worried it might worsen and become even harder to fix later on. The only approach I can think of now is to try “hail-mary” builds, removing random components from my game to see if anything helps. But since this massive RAM allocation occurs on frame 0, before most of my gameplay systems have time to run (I have most of them initialize a few frames later), making them unlikely culprits. Moreover, my main scene uses over 2 GB more “untracked private” RAM than the demo scene within the exact same project.

If anyone has any ideas, I’d love to hear them. I don’t know how to create a repro case without submitting my entire 5-year-old project as a bug report.

I also don’t know how to test point #3 (stack memory issues). I highly doubt it’s stack-related since the total C# RAM usage doesn’t surpass 200 MB.

In the short-to-medium term, I can live with this problem since most PCs that run VR can handle an extra 3+ GB of unexplained RAM usage, and the amount doesn’t grow over time. However, I’m worried about console releases like PS5, or porting to Quest2 with URP, where total available RAM for apps to use is under 3 GB. In that scenario, this bug alone would break the build.

If anyone has insights, I’d greatly appreciate them.

Best Regards,
Colin

[MartinTilo]

@colinleet

there is a giant allocation of around 3 GB of unmanaged private RAM.

That is the allocated amount. Have you switched the dropdown in the All Of Memory table from Allocated Memory to Allocated and Resident Memory on Device to check how much of that is actually important to the memory usage you see in Task Manager, and how much of that is just mem-mapped files, DLLs and other non-resident memory?

This should be the very first check to see that you are investigating relevant memory usage instead of just focusing on the biggest number. (My apologies if you did, it just isn’t clear to me from what you write and the forum post you link to is from before we did report the resident memory amount on a per-page basis (which we do since Unity 6) so it’s not mentioned in there. Also see this Unite talk for more details on the destinction between Allocated and Resident)

@colinleet

VMs using IL2CPP (not applicable, since I’m using Mono)

This is also no longer relevant because in 2022.3 and Unity 6 both VMs allocate their VM memory using Unity’s native allocators, so the Memory Usage Profiler Module includes their VM memory usage as Native, the Memory Profiler Package UI extracts that info and moves it under Managed.

@colinleet

Memory allocated using Marshal.AllocHGlobal (which I’ve never used, so if this were the cause, it’d have to be one of the external DLLs)

Yes, I’ve previously seen projects where I was told this wasn’t used, yet a quick ILSpy look at their .Net libraries revealed that these were calling Marshal.AllocHGlobal for their memory usage. You could try that to see if that might be the case for your DLLs as well.

@colinleet

Stack Memory

As you can see under Untracked, we group the memory based on broad categories supplied to us by the OS, and Thread Stacks is one of these.

@colinleet

Next, I included the three native plugins. As expected, total RAM usage doubled to just under 1 GB, in line with what I anticipated from “untracked private” source #2 (the plugins’ uncompressed size is about 500 MB across their DLLs). This suggests that none of them individually caused the specific spike in my main project. What’s odd is that the ratio of “untracked private” stayed consistent at about 40%.

The Executables & DLLs (or Executables & Mapped) are ONLY untracked in the runtime counters and the Memory Usage Profiler Module, the Memory Profiler Package UI has these tracked as Executables & Mapped, which did grow about 46.6 MB between your 2 captures. That said, I’m uncertain if that’s just their compressed amount with an uncompressed version mapped into Untracked Private. Once again, check how much the Resident Memory amount grew between the snapshots (sadly that’s currently not possible in compare mode, only in single mode).

@colinleet

At this point, I was clueless about potential rabbit holes I could even investigate. In my main scene, the “untracked private” RAM is 2.75 GB, while the test scenario with the same set of DLLs uses only 420MB. Both load and run the same 3rd party DLLS, yet the memory usage differs drastically.

By “load and run” do you mean you are firing the same kind of initial calls and workloads at these plugins as your main project does? If they are not used like in your project, then your test can’t exclude that it is not them who allocate the remaining unclear Untracked amount. Just having them in the project doesn’t necessarily fire all the code paths under which they could allocate untracked memory. For example, does the Steam Audio plugin pick up on audio assets in the build? Does Undertone initialize by loading an AI model?

Also, since 2022 Unity comes with Memory Manager API for low-level native plug-ins that these plugins could utilize to surface their memory usage in the Unity Memory profiler, making it easier to reason about their impact. You could try reaching out to their maintainers asking them to do so (though that’s probably unlikely to happen in the SQLite case)

Beyond that, for the moment you are probably down to native platform tooling to memory profile the remainder of that Untracked amount…

[MartinTilo]

@colinleet

In the short-to-medium term, I can live with this problem since most PCs that run VR can handle an extra 3+ GB of unexplained RAM usage, and the amount doesn’t grow over time. However, I’m worried about console releases like PS5, or porting to Quest2 with URP, where total available RAM for apps to use is under 3 GB. In that scenario, this bug alone would break the build.

Finally, I’d suggest to make builds for your target platforms and take snapshots from these. Extrapolating the memory usage from one platform to another can be quite misleading.

[colinleet]

Thank you so much for this great, extensive, and thorough response! I’m going to go through all of the learning resources you mentioned here and do a more testing before I give a larger report back.

A few quick answers I can do now though:

Allocated Memory vs Allocated and Resident Memory on Device: I viewing the All Allocated Memory… I think… Not total sure… since it sounded like the most comprehensive one and closely mirrored the full amounts being reported by Windows Task Manager. All of the numbers I reported above I verified as also being what was reported by WTM, but not any other 3rd party memory profiling tools since I’m unfamiliar with them and Unity’s profiler has met all of my needs since GPU reporting was fully filled out for the SRP pipelines 3-4 years ago. I’ll look into learning some of the ones you mentioned too.

The numbers I reported were all from builds on my only current target platform (Windows), I was assuming that the underlying problem would probably continue over to other platforms once I start my ports for them next year. I’ll try not to do that going forward without testing them first.

For the two of the three packages I imported into the test scene, I made sure that they at least loaded the core packages: for 1) Steam Audio I made sure it was set as the audio spatializer under the audio settings, and that the required game objects for it were also instanced in the scene, although I didn’t spawn a large number of audio clips since most of my games audio clips are loaded and or generated on the fly – built in clips probably don’t add up to over over a few minutes of audio total. I’ll make sure to do that as well next time. 2) For Undertone I did have it load up the Whisper AI models to the GPU same as it does for my own scene, but I likewise didn’t test it with recording new audio clips as what I pass into it are likewise audio clips recorded from the mic at playtime in my main app. 3) I just included the SQlite dll w/o running it extensively as it seemed the least likely to be a problem (my code using it has hardly changed since 2021 before this problem ever occurred, and I think I’ve only updated the DLLs for it once again several years before this bug (if it is one) appeared). I’ll add in the whole library’s I wrote to use with it and run the most experience set of operations I do with in my main app (indexing several thousand audio files metadata) for the next report.

ILSpy sounds like a really useful tool here, I’ll look into learning it

[MartinTilo]

Since you mention a change since 2021: in 2022 we changed how we get the total allocated amount on all platforms by using respective platform native APIs to ask the OSs directly for what they attribute the app as using, so if you saw a change going from Unity 2021 to 2022, it might just be one of more precise reporting.

[colinleet]

I wish in my case it was the tools just getting better, and me just suddenly realizing that from doing a major version upgrade, but I’ve always just stayed on the most recent weekly versions of the non LTS branches throughout the whole of my game’s 5 year production. The time ranges I gave above arn’t precise since all of my past memories of using the engine are a blur of the “most recent” builds. This problem in particular of excess “private ram” being allocated might have been happening a bit a year or so ago, but it’s been something within the past 6-8 months that it suddenly jumped from < 1 GB (max) (e.g., too small to notice) to 3+ GB (all of the time) in my case.

Overall 99.5% of time I could not be more happy with the really detailed reports I get from the default profiler. There are only two times I can remember total that I couldn’t immediately isolate my issue at hand through just using it. 1) Was in late 2020/early 2021 when Unity’s didn’t yet supporting SRP GPU profiling at the time (excluding ProfilerRecorders). That getting fixed on all platforms about a year later was such a major improvement that it stuck out like a red flag in snow in my mind. 2) Was just a really simple inability to query the current total available ram (to all apps on the whole OS system level) w/o using platform specific plugins, or system specific API calls to the OS directly. I wouldn’t even quality that as a real problem, just a very rarely needed unsupported feature.

This is the first time ever that I’ve had any problem with RAM reporting of any kind from within Unity, period. I could not be more happy with what you and the rest of the people in your department have provided us all in general =).

Admittedly the most complex profiling work I’ve needed to do RAM wise has mostly just detecting leaks in native allocations (for burst/the job system); and unity’s ability report down to the line number where all of those allocation are made makes those ram leaks really trivial to diagnose in editor.

[MartinTilo]

Hello again,
I’ve just dug a bit deeper into workflows for analyzing Untracked Memory on Windows to confirm a few things, so I can share how a deeper analysis might look like on your side as well.

First off, this is highly platform dependent and this…

@MartinTilo

Beyond that, for the moment you are probably down to native platform tooling to memory profile the remainder of that Untracked amount…

still holds (for a non comprehensive list of tools see this list). But lets dig into one such native platform tool for windows:

Windows Performance Recorder (WPR) and Windows Performance Analyzer (WPA).

First off, you’ll want to install the The Windows Assessment and Deployment Kit (ADK) which includes the latest version of these two tools (a just slightly older version failed to process the captured results for me so make sure you get something recent).

Next up, there is this guide on how to use the tools and e.g. grab the Symbol files for your Player (located at e.g. C:\Program Files\Unity\Hub\Editor\2022.3.46f1\Editor\Data\PlaybackEngines\windowsstandalonesupport\Variations\win64_player_development_mono in my case). It’s from 2018 but still holds up well for this.

In WPR you are mainly going to be interested in the VirtualAlloc usage profile so under the More Options foldout, make sure you’re recording that.

Use the Code Example for the MemoryProfiler.TakeSnapshot API to write yourself a small script that will take a snapshot and save it to a file. Log out the file location so you know where it’ll land.

Build a Player and quit other processes that you don’t want to capture with your analysis (like the Editor for example), start a capture in WPR and start your Player .exe. Get to the point where you want to check your memory usage, trigger a Memory Snapshot and quit the app (the easiest way to later determine around about where you took your snapshot). Stop WPR and safe the results to file, then open them in WPA.

As mentioned before, make sure to follow the guide for setting up the Symbol paths in WPA, then check the VirtualAlloc Commit LifeTimes analysis.

You’ll want to be filtering the data via the funnel icon. My filter query looked like this:
[Process Name]:~="2022" AND [Commit Stack]:~!"MemoryManager" AND [Decommit Time]:>"13,59s" AND [Commit Stack]:~!"UnityPlayer.dll!InitD3D11RenderDepthSurface/UnityPlayer.dll!CreateTextureResource" AND [Commit Stack]:~!"InitD3D11RenderColorSurface/UnityPlayer.dll!CreateTextureResource"
Where

• “2022” was enough to filter the processes down to my Unity Player executable
• “MemoryManager” applied to a symbolicated commit stack will remove all call stacks that involve Unity’s Native Memory Manager. For Unity 2022.3 or newer on relatively recent releases that should remove just about everything but Untracked and Graphics allocations.
• The other two filters are parts of our DirectX11 code where I doublechecked our code base that they do register their external memory with the Memory Manager. This registration happens in a callstack parallel to the actual call stack and is therefore harder to isolate.
• The Decommit Time filter filters out any allocation that is decommitted just before that sharp dip of memory usage at the termination of the Player, meaning only allocations that should still have been alive when taking the snapshot are included in the list.

There is every chance that there are other call-stacks that can be excluded as they might also register their external graphics memory. I just tested an Empty Project so far. The All of Memory table subtracts Graphics allocation sizes from the Untracked memory when in Allocated Memory mode by size, as the Memory Manager doesn’t actually know their address, but if you switch to the Memory Map page (only shown when enabled in the Memory Profiler Preference page) you can see all Untracked allocations by address.

While the Memory Map page does not attempt to do any mapping of tracked graphics resources to untracked allocations, you can try to get the overall amount of non-filteres allocations to about the amount of Untracked you see in the All Of Memory page and then use the Memory Map page to doublecheck the addresses of the remaining allocations to the ones the Memory Profiler lists as Untracked. Since the filter should really have caught any tracked native allocation except for Graphics this is where things get pretty fuzzy but maybe this helps regardless