How to (de)serialize an object containing NativeArrays?

Unity Engine
, ,
1

Hi,

I tried to use code like this to convert an object from and to byte[ ]

 private byte[] ObjectToByteArray<T>(T obj)
        {
            var mStream = new MemoryStream();
            _binaryFormatter.Serialize(mStream, obj);

            return mStream.ToArray();
        }

        private T ByteArrayToObject<T>(byte[] arr)
        {
            var mStream = new MemoryStream(arr);
            return (T) _binaryFormatter.Deserialize(mStream);
        }

But my object contains NativeArray which is not marked as serializable :frowning:

how would you do this? I need to store a class in a db and i thought that storing as blob would be the easiest.

2

Push here…

I saw that we have a Serialization Package for Dots, but inside the Readme there is a “more documentation is coming soon” under the Binary part … this is exactly what i was looking for :frowning:

Can anyone help me?

3

Hi,

We are still working on the high level API for the binary serialization of the package which is why the documentation is missing.

While the package does not explicitly support NativeArray types yet (this is something we plan to add moving forward). You should be able to work around by using adapters. The binary ones have the same API as the JSON version.

Unfortunately there is support for generic type adapters yet. So you will need to have one explicitly for the types you need.

Here is an example for an int array:

class NativeArrayAdapter : IBinaryAdapter<NativeArray<int>>
{
    public unsafe void Serialize(UnsafeAppendBuffer* writer, NativeArray<int> value)
    {
        writer->Add(value.Length);

        for (var i = 0; i < value.Length; i++)
            writer->Add(value[i]);
    }

    public unsafe NativeArray<int> Deserialize(UnsafeAppendBuffer.Reader* reader)
    {
        var length = reader->ReadNext<int>();

        var value = new NativeArray<int>(length, Allocator.Persistent);

        for (var i = 0; i < length; i++)
            value[i] = reader->ReadNext<int>();

        return value;
    }
}

byte[] ObjectToByteArray<T>(T obj)
{
    using (var stream = new UnsafeAppendBuffer(16, 8, Allocator.Temp))
    {
        unsafe
        {
            BinarySerialization.ToBinary(&stream, obj, new BinarySerializationParameters
            {
                UserDefinedAdapters = new List<IBinaryAdapter>
                {
                    new NativeArrayAdapter()
                }
            });
        }

        return stream.ToBytes();
    }
}

T ByteArrayToObject<T>(byte[] arr)
{
    unsafe
    {
        fixed (byte* ptr = arr)
        {
            var reader = new UnsafeAppendBuffer.Reader(ptr, arr.Length);
          
            return BinarySerialization.FromBinary<T>(&reader, new BinarySerializationParameters
            {
                UserDefinedAdapters = new List<IBinaryAdapter>
                {
                    new NativeArrayAdapter()
                }
            });
        }
    }
}
4 Likes
4

thanks for the answer, i will look into it. No problem with workarounds currently if the code isnt ready

5

So i finally tried it out and it is suprisingly easy to do with the code snippet and a little debugging skills.

The only question i have now is how i would handle “Memory Leaks”.

Example code is below. So i have these native arrays stored inside a class and i need them to do some lookups. I am tracking and cleaning them up by myself, but unity thinks i “forgot” them and is throwing lots of errors. how should i do it?

A Native Collection has not been disposed, resulting in a memory leak. Allocated from:
Unity.Collections.NativeArray`1:.ctor(Int32, Allocator, NativeArrayOptions)
Assets.Modules.Data.Classes.Adapters.NativeArrayTranslationAdapter:smile:eserialize(Reader*) (at Assets\Modules\Data\Classes\Adapters\NativeArrayTranslationAdapter.cs:22)

public unsafe NativeArray<Translation> Deserialize(UnsafeAppendBuffer.Reader* reader)
        {
            var length = reader->ReadNext<int>();

            var value = new NativeArray<Translation>(length, Allocator.Persistent); //this is line 22

            for (var i = 0; i < length; i++) value[i] = new Translation() {Value = reader->ReadNext<float3>()};

            return value;
        }
6

Fast-forward three years. I’m not sure if this is now supported but I was able to serialize and deserialize a NativeList without having to write a concrete Adapter for every T. Though I had to write two separate adapters for NativeList and UnsafeList but was able to condense them to remove code duplication as much as possible (see spoilers at bottom).

This works for me:

public class NativeListMax65kOfT<T> : IBinaryAdapter<NativeList<T>> where T : unmanaged
{
    public unsafe void Serialize(in BinarySerializationContext<NativeList<T>> context, NativeList<T> value)
    {
        var itemCount = value.Length;
        if (itemCount > UInt16.MaxValue)
            throw new ArgumentOutOfRangeException($"List too long: max. {UInt16.MaxValue} length allowed");

        var writer = context.Writer;
        writer->Add((UInt16)itemCount);

        for (var i = 0; i < itemCount; i++)
            context.SerializeValue(value[i]);
    }

    public unsafe NativeList<T> Deserialize(in BinaryDeserializationContext<NativeList<T>> context)
    {
        var reader = context.Reader;
        var itemCount = (Int32)reader->ReadNext<UInt16>();
        var value = new NativeList<T>(itemCount, Allocator.Temp);

        for (var i = 0; i < itemCount; i++)
            value.Add(context.DeserializeValue<T>());

        return value;
    }
}

Notes:

  • list length is limited to 65k (UInt16) to conserve some memory
  • list Allocator.Temp is used so I don’t need to worry about disposing for unit tests

The unit test code for this:

[Test] public void CanSerializeAndDeserializeNativeListOfLinearTileDataStruct()
{
    var linearData = new LinearTileData(2, TileFlags.DirectionSouth);
    var list = new NativeList<LinearTileData>(1, Allocator.Temp);
    list.Add(linearData);

    var adapters = new List<IBinaryAdapter> { new BinaryAdapters.NativeListMax65kOfT<LinearTileData>() };
    var bytes = BinarySerializer.Serialize(list, adapters);
    var deserialList = BinarySerializer.Deserialize<NativeList<LinearTileData>>(bytes, adapters);

    Debug.Log($"{bytes.Length} Bytes: {bytes.AsString()}");
    Assert.That(deserialList.Length, Is.EqualTo(1));
    Assert.That(deserialList[0], Is.EqualTo(linearData));
}

For completeness sake the LinearTileData struct

    [StructLayout(LayoutKind.Explicit)]
    public struct LinearTileData : ILinearTileData, IEquatable<LinearTileData>
    {
        [FieldOffset(0)] [CreateProperty] private UInt32 m_TileIndexFlags;
        [FieldOffset(0)] private UInt16 m_TileIndex;
        [FieldOffset(2)] private TileFlags m_TileFlags;

        public UInt16 TileIndex { get => m_TileIndex; set => m_TileIndex = value; }
        public TileFlags TileFlags { get => m_TileFlags; set => m_TileFlags = value; }
        public UInt32 TileIndexFlags { get => m_TileIndexFlags; set => m_TileIndexFlags = value; }

        public LinearTileData(UInt32 tileIndexFlags)
        {
            m_TileIndex = 0;
            m_TileFlags = 0;
            m_TileIndexFlags = tileIndexFlags;
        }

        public LinearTileData(UInt16 tileIndex, TileFlags tileFlags)
        {
            m_TileIndexFlags = 0;
            m_TileIndex = tileIndex;
            m_TileFlags = tileFlags;
        }

        public Boolean Equals(LinearTileData other) => m_TileIndexFlags == other.m_TileIndexFlags;
        public override Boolean Equals(Object obj) => obj is LinearTileData other && Equals(other);
        public override Int32 GetHashCode() => (Int32)m_TileIndexFlags;
        public static Boolean operator ==(LinearTileData left, LinearTileData right) => left.Equals(right);
        public static Boolean operator !=(LinearTileData left, LinearTileData right) => !left.Equals(right);
    }

And the serialized buffer is 6 bytes: “102040” (length == 1, tile index == 2, flags == 4)

EDIT
You can extend that to NativeList<UnsafeList> …

You need to have two adapters, one for NativeList and another for UnsafeList. I did that and refactored the common parts into a static helper class.

Here’s the final solution (some names were refactored so it won’t work out of the box with the code above) as a gist with tests and adapters.

Output of the NativeList<UnsafeList> test code (brackets mine):
14 Bytes: 20<10<2040>><10<3080>>
2 items in the enclosing NativeList
1 item in each UnsafeList
values 2 + 4 for linearData1
values 3 + 8 for linearData2

1 Like
7

I’m now trying to do the same list serialization/deserialization for Json … oh boy, Json serialization is FAR HARDER than I expected! Binary is trivial compared to Json … understandably though, because binary has no structure besides the code, so the pay-off (or: payback) is in adapting code changes through versioning and fallback deserializers.

After three days of trying to figure out how the Json API expects me to work with it, I could figure out a working solution but I’m not satisfied with it because it won’t work with generics:

public NativeList<T> Deserialize(in JsonDeserializationContext<NativeList<T>> context)
{
    var itemCount = context.SerializedValue["Length"].AsInt32();
    var list = CreateResizedNativeList(itemCount, m_Allocator);
    var array = context.SerializedValue["Items"].AsArrayView().ToArray();
    var genericType = typeof(T);
    switch (genericType)
    {
        case Type _ when genericType == typeof(Int32):
            for (var i = 0; i < itemCount; i++)
                list[i] = (T)(Object)array[i].AsInt32();
            break;
        case Type _ when genericType == typeof(Int64):
            for (var i = 0; i < itemCount; i++)
                list[i] = (T)(Object)array[i].AsInt64();
            break;
        default:
            throw new ArgumentException($"unhandled generic type: {typeof(T).Name}");
    }
    return list;
}

I actually have to switch over the generic type and cast it back to (T) by going through (object) boxing. Is that the only solution, besides implementing concrete type adapters?
(eg NativeListInt32Adapter + NativeListInt64Adapter instead of NativeListAdapter)

I get the feeling that this is what was actually meant by “not supporting generic type adapters yet”. The generic T adapters work for binary, but not for Json. :frowning:

1 Like
8

Is there any update on this yet?