I have a nativehashmap<int4, struct> that I want to serialize. I tried copying it over to a managed dictionary and use JSON to save it but it was extremely slow.
Is there a way to serialize it with the jobs system? Do I need to use this? Introduction to Unity Serialization | Serialization | 3.0.0-pre.3
If so is there any tutorials anyone knows of? I tried playing around with it but it’s way over my head. Was getting errors like it needs to be used in an unsafe context, etc etc.
I do love the serialization package. However, to my knowledge, it does not use the Jobs system to perform the serialization/deserialization.
The Json serializer does use Jobs when deserializing, but only to read the json string and tokenize it. The actual deserialization is then performed normally, without Jobs.
However it might be faster than copying to a managed dict. But most likely only if you use the binary serialization approach, but i dont really know.
I can provide some examples to do this, in both Binary & Json.
Json Serialization for that package is done by leaning into the Unity.Properties module. So the serializer, by default, will visit the properties of the provided instance that are visible to Unity.Properties. This is:
So by default it wont be able to serialize a NativeHashMap<T,V>. However you can write and provide Adapters that will be the classes responsible for actually writing&reading your data.
int4 has public members x,y,z,w. So thats ok.TValue, it will only serialize it without adapters if its properties follow the rules described above, if not, you can write a dedicated adapter for your type.The advantage of the Binary serializer, is that it can directly serialize the byte representation of any unmanaged struct. And it also directly supports NativeArray<T>'s. So there is no need to follow the property / field rules, as the native map already restricts its contents to unmanaged structs.
A Binary adapter for the NativeHashMap<T,V> could look like this:
// notice the "unsafe" keyword. You can only use "unsafe"
// when compiling within an AssemblyDefinition with the flag "allow unsafe code" checked.
// Unsafe only means that you might want to use pointers in the code.
unsafe class NativeHashmapBinaryAdapter<TKey, TValue> : IBinaryAdapter<NativeHashMap<TKey, TValue>>
where TKey : unmanaged, IEquatable<TKey>
where TValue : unmanaged
{
public void Serialize(
in BinarySerializationContext<NativeHashMap<TKey, TValue>> context,
NativeHashMap<TKey, TValue> value
)
{
// here im getting a copy to demonstrate that it can directly serialize any NativeArray<T>,
// but really there is no need, you could just avoid copying and write the Length as an integer,
// and then, write each key & value sequentally.
// On the Deserialize method, you would do the same in reverse
// (i leave this to you)
var copy = value.GetKeyValueArrays(Allocator.Temp);
try
{
// notice that "Writer" is really a pointer. To call
// its methods you should use ->Method() insteado of .Method()
context.Writer->Add(copy.Keys);
context.Writer->Add(copy.Values);
}
finally { copy.Dispose(); }
}
public NativeHashMap<TKey, TValue> Deserialize(in BinaryDeserializationContext<NativeHashMap<TKey, TValue>> context)
{
NativeArray<TKey> keys = default;
NativeArray<TValue> values = default;
try
{
context.Reader->ReadNext<TKey>(out keys, Allocator.Temp);
context.Reader->ReadNext<TValue>(out values, Allocator.Temp);
// notice that im initializing the instance directly in the serializer with Allocator.Temp
// this might be dangerous with any other allocator if the deserialization for another item
// fails and throws an exception. I guess there could be better ways to approach this
var map = new NativeHashMap<TKey, TValue>(keys.Length, Allocator.Temp);
for (int index = 0; index < keys.Length; index++)
map.Add(keys[index], values[index]);
return map;
}
finally
{
if (keys.IsCreated)
keys.Dispose();
if (values.IsCreated)
values.Dispose();
}
}
}
A JsonAdapter, could look like this:
class NativeHashmapJsonAdapter<TKey, TValue> : IJsonAdapter<NativeHashMap<TKey, TValue>>
where TKey : unmanaged, IEquatable<TKey>
where TValue : unmanaged
{
public void Serialize(
in JsonSerializationContext<NativeHashMap<TKey, TValue>> context,
NativeHashMap<TKey, TValue> value
)
{
using (context.Writer.WriteArrayScope())
foreach (var item in value)
using (context.Writer.WriteObjectScope())
{
// this wont work propperly if TKey or TValue dont have properties / fields
// visible for Unity.Properties.
context.SerializeValue("Key", item.Key);
context.SerializeValue("Value", item.Value);
}
}
public NativeHashMap<TKey, TValue> Deserialize(in JsonDeserializationContext<NativeHashMap<TKey, TValue>> context)
{
// This is valid only if we are using "FromJsonOverride",
var instance = context.GetInstance();
// if not, you should manually create the NativeHashMash using a desired allocator
// (the same danger exists than with the BinaryAdapter)
if (!instance.IsCreated)
instance = new NativeHashMap<TKey, TValue>(0, Allocator.Temp);
foreach (var item in context.SerializedValue.AsArrayView())
{
// this wont work propperly if TKey or TValue dont have properties / fields
// visible for Unity.Properties.
var key = context.DeserializeValue<TKey>(item["Key"]);
var value = context.DeserializeValue<TValue>(item["Value"]);
instance.Add(key, value);
}
return instance;
}
}
You can test this approach like this:
using System;
using Unity.Collections;
using Unity.Collections.LowLevel.Unsafe;
using Unity.Collections.LowLevel.Unsafe.NotBurstCompatible;
using Unity.Mathematics;
using Unity.Serialization.Binary;
using Unity.Serialization.Json;
using UnityEditor;
using UnityEngine;
public class TestSerializedUnmanaged : EditorWindow
{
public struct SomeUnmanagedStruct
{
public float someFloat;
public int someInt;
}
private static readonly NativeHashmapJsonAdapter<int4, SomeUnmanagedStruct> s_jsonAdapter = new();
private static readonly NativeHashmapBinaryAdapter<int4, SomeUnmanagedStruct> s_binaryAdapter = new();
private static readonly JsonSerializationParameters s_jsonParams = new()
{
UserDefinedAdapters = new() { s_jsonAdapter }
};
private static readonly BinarySerializationParameters s_binaryParams = new()
{
UserDefinedAdapters = new() { s_binaryAdapter }
};
[SerializeField]
private byte[]? _lastSerializedBinary;
[SerializeField]
private string? _lastSerializedJson;
private Vector2 _scrollPosition;
public void OnGUI()
{
using var scroll = new EditorGUILayout.ScrollViewScope(_scrollPosition);
using var serializedSelf = new SerializedObject(this);
using (new EditorGUI.DisabledScope(true))
using (var serializedBinaryProperty = serializedSelf.FindProperty(nameof(_lastSerializedBinary)))
EditorGUILayout.PropertyField(serializedBinaryProperty);
using (var serializedJsonProperty = serializedSelf.FindProperty(nameof(_lastSerializedJson)))
EditorGUILayout.PropertyField(serializedJsonProperty);
serializedSelf.ApplyModifiedProperties();
DoJsonButtons();
DoBinaryButtons();
_scrollPosition = scroll.scrollPosition;
}
// notice again the "unsafe" keyword
private unsafe void DoBinaryButtons()
{
if (GUILayout.Button("Serialize To Binary"))
{
using var data = CreateDummyData();
var buffer = new UnsafeAppendBuffer(0, UnsafeUtility.AlignOf<byte>(), Allocator.Temp);
try
{
// here, the ToBinary method needs a pointer to an UnsafeAppendBuffer.
// We pass a pointer by preceding the variable with the & symbol.
// Directly using pointers when dealing with structs is always allowed.
BinarySerialization.ToBinary(&buffer, data, s_binaryParams);
_lastSerializedBinary = buffer.ToBytesNBC();
}
finally { buffer.Dispose(); }
}
if (_lastSerializedBinary?.Length > 0 && GUILayout.Button("Deserialize From Binary"))
{
// here, UnsafeAppendBuffer.Reader, needs a void* pointer to be initialized.
// It really wants a pointer to the start of the byte buffer that we are reading.
// However, a "byte[]" (any array) is a managed type, so we can only create a
// pointer for it if we temporarily "fix" the memory location (this is because managed
// types are free to be moved around in memory, at any time, so a pointer could fail if the
// array gets moved). The "fixed" keyword does precisely that, it just "fixes" the memory
// location of the array while inside the fixed scope. This allows to "cast" the array to a void*
// pointer without raising compiler errors.
fixed (void* buffer = _lastSerializedBinary)
{
var reader = new UnsafeAppendBuffer.Reader(buffer, _lastSerializedBinary.Length);
using var map =
BinarySerialization.FromBinary<NativeHashMap<int4, SomeUnmanagedStruct>>(&reader, s_binaryParams);
LogDeserializedMap(map);
}
}
}
private void DoJsonButtons()
{
if (GUILayout.Button("Serialize To Json"))
{
using var data = CreateDummyData();
_lastSerializedJson = JsonSerialization.ToJson(data, s_jsonParams);
}
if (!string.IsNullOrEmpty(_lastSerializedJson) && GUILayout.Button("Deserialize From Json"))
{
var dummyToOverride = new NativeHashMap<int4, SomeUnmanagedStruct>(0, Allocator.Temp);
try
{
JsonSerialization.FromJsonOverride(_lastSerializedJson, ref dummyToOverride, s_jsonParams);
LogDeserializedMap(dummyToOverride);
}
finally { dummyToOverride.Dispose(); }
}
}
private static void LogDeserializedMap(NativeHashMap<int4, SomeUnmanagedStruct> map)
{
string log = $"Hashmap with {map.Count} items deserialized.\n";
foreach (var item in map)
{
log += $"{item.Key} => SomeFloat: {item.Value.someFloat}, SomeInt: {item.Value.someInt}\n";
}
Debug.Log(log);
}
private static NativeHashMap<int4, SomeUnmanagedStruct> CreateDummyData()
{
return new NativeHashMap<int4, SomeUnmanagedStruct>(4, Allocator.Temp)
{
{ new int4(0, 0, 0, 0), new SomeUnmanagedStruct() { someFloat = 0, someInt = 0 } },
{ new int4(1, 1, 1, 1), new SomeUnmanagedStruct() { someFloat = 1, someInt = 1 } },
{ new int4(2, 2, 2, 2), new SomeUnmanagedStruct() { someFloat = 2, someInt = 2 } },
{ new int4(3, 3, 3, 3), new SomeUnmanagedStruct() { someFloat = 3, someInt = 3 } }
};
}
[MenuItem("Tools/TestSerializedUnmanaged")]
private static void ShowWindow()
{
GetWindow<TestSerializedUnmanaged>().Show();
}
}
I hope this helps to get you started!
Hey thx Canijo, I’ll play around with it and see what I can get.
Wow man, I believe you typed all that out just for me lol. Thanks so much.
I was able to get it working but there wasn’t any speed increases though. Do you think it is possible to burst it? My struct that I am using is just full of ints 
I’ll keep tinkering.
EDIT: Nvm, I was still using JSON. I saved it directly to data file and it’s saving/loading 130+ MB in under a second compared to 8-9 seconds with JSON. Thx so much Canijo, what’s your paypal? I need to tip you. I never would have figured this out on my own.
Haha no worries, i just remember how “unsafe” and pointers didnt make sense to me, but they really are no biggie once you start to understand.
Happy coding!
On a final note, since you are persisting big files (maybe you’ve already looked into this).
The biggest bottleneck in both approaches is getting back the data into a managed format. That is, for json, converting the output into a string, and for binary, into a byte[]. And the same in reverse, having to read the contents of a file into a string/byte[] and then pass those to the deserializer.
If persisting to/from a file, that most performant way will most likely be dealing directly with the file, instead of reading/writing it to a temporary object
JsonSerialization already has overloads for accepting a Stream, that you can get directly through File.Open(string path, FileMode mode).
For the Binary serialization is a little bit more tricky, so i’ll leave an example that works for me in case someone with the same needs eventually finds this post and find it usefull.
(this could be pasted into the original post EditorWindow)
[SerializeField]
private string? _lastBinaryFilePath;
public void OnGUI()
{
// ...
using (var serializedBinaryPathProperty = serializedSelf.FindProperty(nameof(_lastBinaryFilePath)))
EditorGUILayout.PropertyField(serializedBinaryPathProperty);
// ...
DoBinaryFileButtons();
// ...
}
private unsafe void DoBinaryFileButtons()
{
if (GUILayout.Button("Serialize To Binary File"))
{
using var data = CreateDummyData();
_lastBinaryFilePath = FileUtil.GetUniqueTempPathInProject();
var buffer = new UnsafeAppendBuffer(0, UnsafeUtility.AlignOf<byte>(), Allocator.Temp);
try
{
BinarySerialization.ToBinary(&buffer, data, s_binaryParams);
using var fileWriter = File.Open(_lastBinaryFilePath, FileMode.Create);
ReadOnlySpan<byte> span = new(buffer.Ptr, buffer.Length);
fileWriter.Write(span);
}
finally { buffer.Dispose(); }
}
if (!string.IsNullOrEmpty(_lastBinaryFilePath)
&& GUILayout.Button("Deserialize From Binary And Delete File"))
{
// Im not familiar with this API, but seems to get the work done.
// We are just looking for a way to get a byte* to the file contents
// with the least allocation that we can
using (var file = MemoryMappedFile.CreateFromFile(_lastBinaryFilePath, FileMode.Open))
using (var view = file.CreateViewAccessor())
{
var handle = view.SafeMemoryMappedViewHandle;
ulong length = handle.ByteLength;
if (length > int.MaxValue)
throw new ArgumentException();
byte* ptr = null;
handle.AcquirePointer(ref ptr);
try
{
var reader = new UnsafeAppendBuffer.Reader(ptr, (int)length);
using var map =
BinarySerialization.FromBinary<NativeHashMap<int4, SomeUnmanagedStruct>>(
&reader,
s_binaryParams
);
LogDeserializedMap(map);
}
finally{
handle.ReleasePointer();
}
}
if (FileUtil.DeleteFileOrDirectory(_lastBinaryFilePath))
_lastBinaryFilePath = null;
}