Allocating unsafe persistent buffers as a part of static initialization

Unity Engine
,
1

I’ve been using Unity.Collections.LowLevel.Unsafe.UnsafeUtility methods for native allocations in multiple areas outside of DOTS as a way to reduce overhead and speed up operations in non-Job contexts by removing the need for bounds checking or pinning GC-bound managed arrays.

One common case I have actually come across several times is statically allocated persistent lookup/hash tables for compression, hash, and encryption algorithms. For example: LZF/Runtime/CLZF2.cs at master · HouraiTeahouse/LZF · GitHub. For the most part, this has been working, it produces the results I expect. My only question is: is this safe? As far as I am concerned, the actual bounds checking and allocation size isn’t an issue, but none of these buffers are being manually deallocated from the C# side. In the Player, the lifetime of these buffers basically last the entire duration of the program, are allocated only once, and will probably be deallocated along with the shutdown of the process, but do these buffers survive domain reload in the Editor?

Alternatively, is there a way to build some of these static buffers at compile time? This is obviously available to C/C++ contexts, but normally out of bounds for most C# programs. I was wondering if there was away to avoid the upfront computation cost at startup to build these tables.

Tangentially related: access to these functions is genuinely amazing. I’ve been working on some low level networking parts (separate from the DOTS multiplayer/transport layer), and it’s been really useful in speeding up a lot of the performance critical parts of the code, as well as completely removing the need for GC-bound allocations.

(If this is the wrong forum, please move this thread).

1 Like
2

There are few things to note.

Persistent allocations survive Editor domain reload. To avoid memory leaks make sure you deallocate the table on the domain shutdown (there is a callback for that). Also, Persistent allocations are slow if you are planning to do it each frame / so you may use stackalloc / Temp label instead. As for building such tables at compile time - nothing prevents from storing them in a file as a byte stream and loadup into memory on application boot - then simply pass the pointer to the memory block.

I hope that answers your question.
Cheers.

3

Thanks for the advice, I’ll add some hooks into AssemblyReloadEvents to assure they get free’d before a domain reload in the Editor then.

Yes, most of these allocations are being done from a static constructor for a class: mostly often to establish a global readonly buffer that I don’t need to pin to directly work with the bytes and cut out the need for bounds checking where it only serves to slow down the algorithm. This is pretty common in compression/encryption/hashing situations. For example: a CRC32 algorithm that, while I have not tested yet, is probably the fastest of any pure C# implementations: https://github.com/HouraiTeahouse/HouraiCore/blob/master/Runtime/Buffers/CRC32.cs.