Documentation on the binary profiler data file format?

Alloc · September 8, 2017, 5:59pm

Hi,

as already pointed out in my post in the profiler improvements thread it would be really cool if we could get a documentation on how the binary file of the profiler is organized. Even the source of the class that reads that file or anything like that would be enough I suppose.

Even though we can read out the data that Unity itself displays in the profiler window by calling into the respective UnityEditor classes that’s still quite limited:

It’s limited to the 300 frames the window shows, even though it looks like an actual profiler file stores more frames when running from a build.
It only returns strings instead of proper values (limiting e.g. the resolution of the allocations values; requiring to parse each of those strings to convert them back to floats; it being slower; causing crashes on the editor if you iterate over all frames + samples of a large profile snapshot)
We have to pass the binary file to Unity to run the conversion to our own data that can then be displayed by our own tools which is especially annoying as the Unity 2017 editor seems to somehow force a connection to a running dev build, i.e. you can’t run the profiler in a dev build as long as the editor is open if you want to profile directly to files (all those profiler files will have zero bytes).

Is there any chance to get any kind of documentation on this file format?

Kind regards,
Chris

Alloc · September 12, 2017, 11:07am

Any word on this from any Unity dev?

Alloc · September 13, 2017, 5:56pm

Biggest issue right now is that Unity crashes most of the time I loop through all samples of a frame of a deep profile due to out of memory for the profiler:

        UnityEditorInternal.ProfilerProperty sample = new UnityEditorInternal.ProfilerProperty();
        sample.SetRoot(_frameIndex, UnityEditorInternal.ProfilerColumn.DontSort, UnityEditorInternal.ProfilerViewType.Hierarchy);
        sample.onlyShowGPUSamples = false;

        while (sample.Next (true)) {
            writeProfilerSample (_paths, _funcs, sample, _bw);
        }

        sample.Cleanup ();
        sample.Dispose ();

(writeProfilerSample just queries the different ProfilerColumns on the sample).

So nothing special in there, think no way to make sure it frees up memory in between, so I can’t get all the data exported

Alloc · September 18, 2017, 11:29am

Alloc · September 25, 2017, 9:34am

pachash · November 2, 2017, 11:50am

Yep, I’m also interested in the internal binary format of the Profiler data. With this information at hands I could convert it into format which compatible with Chrome tracing tools as described here: Chrome Tracing as Profiler Frontend · Aras' website

alexeyzakharov · November 17, 2017, 3:36pm

Hi,
Thanks for the feedback! Please use “profiling” tag in the future to help us find questions

If Autoconnect Profiler build option is on, then, yes, player tries to establish connection to the editor at startup.
If you disable that, there should no interference from a player, overwise file a bug please.

We’ve changed the binary format in 2017.3. And yes, we are planning to open the format in the future, but only when it is more or less settled, and provide C# demo converter.

For now only undocumented ProfilerProperty or ProfilerFrameDataIterator can be used to get the information.
If you experience any crashes, please file a bug - there shouldn’t be any unnecessary allocations when treeview representation is being built.

Alloc · November 17, 2017, 5:25pm

We’re building from our own scripts (with the complexity of our project it’s not really manageable doing all the necessary settings manually and use Unity’s build dialog anymore), so when we do a development build we set BuildOptions.Development and AllowDebugging but we don’t set ConnectWithProfiler (assuming this is the relevant one here).
No idea what I’d file here as a bug report or at least as a project, think this is quite generic.

No idea really how to exactly reproduce this, I just know it often happens with larger (deep profiling) results and doesn’t seem to matter if I use Unity’s Profiler Window and expand certain nodes or use my custom code to iterate all samples.

This is the log, in this case (as it’s a bit easier to get) I used our code to iterate the data but it’s exactly the same (except the trace of course, it’s coming from the Unity GUI normally but still ending in ProfilerProperty:Next(Boolean)):

DynamicHeapAllocator allocation probe 1 failed - Could not get memory for large allocation 1560989232.
DynamicHeapAllocator allocation probe 2 failed - Could not get memory for large allocation 1560989232.
DynamicHeapAllocator allocation probe 3 failed - Could not get memory for large allocation 1560989232.
DynamicHeapAllocator allocation probe 4 failed - Could not get memory for large allocation 1560989232.
DynamicHeapAllocator out of memory - Could not get memory for large allocation 1560989232!
Could not allocate memory: System out of memory!
Trying to allocate: 1560989232B with 16 alignment. MemoryLabel: Profiler
Allocation happend at: Line:78 in C:\buildslave\unity\build\Runtime/Allocator/STLAllocator.h
Memory overview

[ ALLOC_DEFAULT ] used: 1103830295B | peak: 0B | reserved: 1208460447B
[ ALLOC_TEMP_JOB ] used: 0B | peak: 0B | reserved: 3145728B
[ ALLOC_GFX ] used: 1676515942B | peak: 0B | reserved: 1692873606B
[ ALLOC_CACHEOBJECTS ] used: 279790296B | peak: 0B | reserved: 297795584B
[ ALLOC_TYPETREE ] used: 51941896B | peak: 0B | reserved: 69206016B
[ ALLOC_PROFILER ] used: 23910497608B | peak: 0B | reserved: 24079080112B
[ ALLOC_TEMP_THREAD ] used: 32768B | peak: 0B | reserved: 53084160B
Could not allocate memory: System out of memory!
Trying to allocate: 1560989232B with 16 alignment. MemoryLabel: Profiler
Allocation happend at: Line:78 in C:\buildslave\unity\build\Runtime/Allocator/STLAllocator.h
Memory overview

[ ALLOC_DEFAULT ] used: 1103830295B | peak: 0B | reserved: 1208460447B
[ ALLOC_TEMP_JOB ] used: 0B | peak: 0B | reserved: 3145728B
[ ALLOC_GFX ] used: 1676515942B | peak: 0B | reserved: 1692873606B
[ ALLOC_CACHEOBJECTS ] used: 279790296B | peak: 0B | reserved: 297795584B
[ ALLOC_TYPETREE ] used: 51941896B | peak: 0B | reserved: 69206016B
[ ALLOC_PROFILER ] used: 23910497608B | peak: 0B | reserved: 24079080112B
[ ALLOC_TEMP_THREAD ] used: 32768B | peak: 0B | reserved: 53084160B

UnityEditorInternal.ProfilerProperty:Next(Boolean)
ProfilerExport:writeProfilerFrame(DictionaryKeyList`2, DictionaryKeyList`2, Int32, BinaryWriter) (at Assets\Editor\ProfilerExport.cs:116)
ProfilerExport:ExportData() (at Assets\Editor\ProfilerExport.cs:47)

[C:/buildslave/unity/build/Runtime/Allocator/MemoryManager.cpp line 1045]
(Filename: Assets/Editor/ProfilerExport.cs Line: 116)

alexeyzakharov · November 28, 2017, 7:59am

In 2017.3 it is a bug - we’ve also seen it recently when opening saved file with many samples (deep profiler). It is being fixed. Thanks for the report!

Alloc · November 28, 2017, 4:16pm

Ah, great
Is this by any chance this solved issue in b11? “UI: Fixing issue with UI profiler memory leak caused by not releasing all stored data. (955744)”

alexeyzakharov · November 28, 2017, 5:54pm

It might be still useful to sumbit a bug :). Though it looks like the same issue, now that I read log again it might be also related to another one - e.g. if the data is loaded from a different version. Even though ProfilerProperty is undocumented Profiler Window should suffer from the same crash - Timeline view effectively uses the same sample.Next (true)

Yes, 2017.3b11 it is.

JesterGameCraft · December 5, 2017, 10:01pm

Hi,

I’m also interested in the profiler data file format. Alexey, do you have any idea when on the Roadmap exposure of the profiler file format sits? It might be beneficial to provide the specs for what is presently available since design community could provide helpful input/suggestions. Thoughts?

alexeyzakharov · December 8, 2017, 8:05am

Hi!

We didn’t really bind it to the timeline yet, but we definitely would like share initial draft once we have it as we want to get it right with less iterations. Thanks for willing to help!
Besides that, could you please explain your use case and how would you use the binary format specification.
We are also considering exposing C# reader API (callback or iterator based) first - that would allow to walk the data stream and be independent on format changes. Would that work for you or you were thinking more about standalone converter in another language?
Thanks!

JesterGameCraft · December 8, 2017, 2:41pm

Hi,

The use case would be the ability to parse the binary data then feed it to another application that augments the profiler and assists in finding source of issues. I find that using profiler to compare two frames against each other in the Timeline can be very time-consuming. Also ability to record longer traces would be a great add on as well. Could be implemented in one large file or in smaller files that have a chunk of the data, kind of like log files roll over on Linux systems. Then could be parsed all together into one large stream if fed into another app. This is more on the recording side of things and file rollover then binary format specification, but if that ability was present then the numerous files could be parsed and stitched together.

It does not have to be a standalone “another app”, could be an Editor plugin if data is available. I think the C# reader API you mentioned would work in that case. Standalone app could be faster though since it could be specifically optimized to deal with this data and not be tied to Editor architecture.

About the binary format. Some thought I had is, if user only cares about a specific subset of info, for instance only CPU, or only RAM. It would be great to parse the file in such a way so that only the relevant data is retrieved and is colocated inside the file, which I’m assuming you’re probably doing this anyways. Or perhaps even having the ability to select (in Profiler) what info user want’s to store in the binary file, eg. CPU only for instance and only that get’s recorded. Then the binary format could specify each subsection separately (CPU, RAM) and when more info is recored it would just be added to the file, file could state in the header what data is present, etc. Ability to select what to record could help with file size for longer recordings. These are just some ideas, take them with a grain of salt

Those are some of the things I’ve been thinking about…

alexeyzakharov · December 11, 2017, 4:13pm

@JesterGameCraft thanks for the answer!

Yes, that’s non trivial. And this is very useful feature.

We can serialize arbitrary number of samples per file (limited by free disk space) with a typical pace about 4MB/s.
Each frame has a separate event marker so we can scroll pretty fast once we analyzed the file. The limitation is more on the Editor side and this is something we need to address.

Right now we serialize the data in a custom binary format as a sequence of event messages. The format approximately looks like this https://docs.google.com/document/d/1oHTAT9Hm1J7XlnBr_y6Y7l4AuJj3EHdEgZrWfE-RawQ/edit#heading=h.tcjdkqtpvp8a (just a high level overview with a very approximate milestones). It might be changed/aligned with other systems in the future.

If I understand correctly this should be a dynamic per-marker/per-category control from C# API (runtime or editor). That is definitely possible and was kept in mind. However, right now we have a data streamed out as fast as possible without looking back at file header - so scanning would be required to get to the specific frame/event.

IMHO, I think it would be better to expose internal profiler data iterator API (aka ProfilerProperty) - that would give an ability to make custom profiler window with custom hierarchy/timeline views in a short term. And it would work in a long term as it will be supported for built-in Profiler Window.

JesterGameCraft · December 11, 2017, 4:53pm

I understand. Well you definitely don’t want to loose any data and I can see keeping up with the data coming in on the stream would be a challenge. I think exposing the API is a good solution. It would give access to all the data and ability to have custom plugins work with it. The data can always be parsed after its been captured, if need be.

sls_pontusl · March 12, 2018, 3:43pm

Alloc:

Biggest issue right now is that Unity crashes most of the time I loop through all samples of a frame of a deep profile due to out of memory for the profiler:
        UnityEditorInternal.ProfilerProperty sample = new UnityEditorInternal.ProfilerProperty();
        sample.SetRoot(_frameIndex, UnityEditorInternal.ProfilerColumn.DontSort, UnityEditorInternal.ProfilerViewType.Hierarchy);
        sample.onlyShowGPUSamples = false;

        while (sample.Next (true)) {
            writeProfilerSample (_paths, _funcs, sample, _bw);
        }

        sample.Cleanup ();
        sample.Dispose ();
(writeProfilerSample just queries the different ProfilerColumns on the sample).

So nothing special in there, think no way to make sure it frees up memory in between, so I can’t get all the data exported

I am having issues saving frames fast enough. The Game Engine is constantly spewing out:

Skipping profile frame. Receiver can not keep up with the amount of data sent

I do not have this ProfilerColumn.DontSort, Intuitively it seems like that might be a good way to solve my issue.

Do you care to elaborate on how you save the frame data fast enough for storage in your own systems?

txy130 · October 12, 2018, 9:38am

Hi,All,
I want to get memory detail infos,How can I do?
I tried to use ProfilerDriver,but I can’t find API about getMemoryDetail.

jimblackler0 · February 23, 2023, 12:56pm

Can I get an update on this 4+ years on? Is the format documented, or otherwise accessible through stand-alone libraries or command line tools?

Yuto_Harada · June 1, 2023, 6:13am

Unity Japan is analyzing Profile data in this project (GitHub - unity3d-jp/ProfilerReader: The library that read Unity profiler binary log directly). The actual data loading part is dll-ized, and while it may not be a commendable method, you can guess the file format by decompiling it. I am hoping that it will be officially documented as soon as possible.

Topic		Replies	Views
[Official] Profiler Improvements Unity Engine Profiling , Official	35	18220	October 11, 2017
Why is Unity's profiler so slow and unresponsive in a frame that has a lot of data? Unity Engine	59	20222	July 2, 2018
We've started work on a new memory profiler Unity Engine Official	88	31874	October 22, 2020
Profiling background threads Unity Engine Scripting	85	19070	March 23, 2021
Benchy Profiler - A simple benchmarking and profiling tool for Unity Free! Community Showcases Asset-Store-Assets	222	44470	November 25, 2015

Documentation on the binary profiler data file format?

Related topics