A way to exactly replicate/serialize a world (into file, over network)

Hello

Does anyone know if there are plans to support bit-exact world duplication to quickly store the world somewhere or replicate it across the network without having entities become re-indexed or otherwise modifying the state?

For example, this would be rather useful to initially synchronize the world when joining a multiplayer game.

For a test I hacked the source of the entities package to make it work, but having it supported officially would be make it more reliable and feasible.

I attached my hack if somebody is interested. In my opinion it’s also much easier to use than the built in serialization (never figured out how to serialize shared components since 0.2).

To use it:

using System.IO;
using System.IO.Compression;
using Unity.Collections.LowLevel.Unsafe;
using Unity.Entities;
using System.Collections.Generic;

namespace Game
{
    class Application
    {
        static public void Save(EntityManager entityMgr, System.IO.Stream stream, bool leaveStreamOpen)
        {
            using (var writer = new Unsafe.SaveBinaryWriter(stream, leaveStreamOpen))
            {
                writer.WriteWorld(entityMgr);

                List<SomeSharedComp> registeredSomeSharedComps = new List<SomeSharedComp>();
                List<int> someSharedCompIndices = new List<int>();
                entityMgr.GetAllUniqueSharedComponentData(registeredSomeSharedComps, someSharedCompIndices);

                writer.Write(someSharedCompIndices.Count - 1);
                for (int i = 1; i != someSharedCompIndices.Count; i++)
                {
                    writer.Write(someSharedCompIndices[i]);
                    registeredSomeSharedComps[i].Serialize(writer);
                }
            }
        }

        static internal void Load(EntityManager entityMgr, System.IO.Stream stream, bool leaveStreamOpen)
        {
            using (var reader = new Unsafe.SaveBinaryReader(stream, leaveStreamOpen))
            {
                reader.ReadWorld(entityMgr);

                int numSomeSharedComps = reader.ReadInt32();
                for (int i = 0; i != numSomeSharedComps; i++)
                {
                    int someSharedCompIndex = reader.ReadInt32();
                    SomeSharedComp someSharedComp = new SomeSharedComp();
                    someSharedComp.Deserialize(someSharedCompIndex, reader);
                    Unity.Entities.Serialization.SerializeUtility.AddSharedComponentAtIndex(entityMgr, someSharedCompIndex, someSharedComp);
                }
            }
        }
    }
}

namespace Game.Unsafe
{
    public unsafe class SaveBinaryWriter : BinaryWriter, Unity.Entities.Serialization.BinaryWriter
    {
        int worldPos;
        byte[] worldArr;

        public SaveBinaryWriter(Stream stream, bool leaveOpen)
        {
            OutStream = new DeflateStream(stream, CompressionMode.Compress, leaveOpen);
        }

        public void WriteWorld(EntityManager entityMgr)
        {
            worldArr = new byte[1024];
            Unity.Entities.Serialization.SerializeUtility.SerializeWorldState(entityMgr, this);
            if (worldPos != 0) { OutStream.Write(worldArr, 0, worldPos); worldPos = 0; }
            worldArr = null;
        }

        void Unity.Entities.Serialization.BinaryWriter.WriteBytes(void* data, int len)
        {
            if (worldPos + len > worldArr.Length)
            {
                OutStream.Write(worldArr, 0, worldPos);
                worldPos = 0;
            }
            if (len > worldArr.Length)
            {
                worldArr = new byte[len + 64];
            }
            fixed (void* ptr = worldArr) UnsafeUtility.MemCpy((byte*)ptr+worldPos, data, len);
            worldPos += len;
        }
    }

    public unsafe class SaveBinaryReader : BinaryReader, Unity.Entities.Serialization.BinaryReader
    {
        byte[] worldArr;

        public SaveBinaryReader(Stream stream, bool leaveOpen) : base(new DeflateStream(stream, CompressionMode.Decompress, leaveOpen))
        {
        }

        public void ReadWorld(EntityManager entityMgr)
        {
            worldArr = new byte[1024];
            Unity.Entities.Serialization.SerializeUtility.DeserializeWorldState(entityMgr.BeginExclusiveEntityTransaction(), this);
            entityMgr.EndExclusiveEntityTransaction();
        }

        void Unity.Entities.Serialization.BinaryReader.ReadBytes(void* data, int len)
        {
            fixed (void* ptr = worldArr)
            {
                for (int i = 0; i < len; i += 1024)
                {
                    int block = UnityEngine.Mathf.Min(len - i, 1024);
                    Read(worldArr, 0, block);
                    UnsafeUtility.MemCpy((byte*)data + i, ptr, block);
                }
            }
        }
    }
}

Hoping that someone besides me is interested in this :slight_smile:

5248754–524276–DeterministicSerialization.cs (12.6 KB)

3 Likes

I am also quite interested in this. Not for gameplay, but to allow bit-exact game replays for development and debugging.

It seems like over time, more and more managed and pointer-referenced state has crept into ECS components, which makes it challenging to actually correctly serialize the world perfectly. Is anyone doing this?

I have also found that the SerializeUtility really doesn’t like Entity references in Shared Components, but I don’t know why. It seems very odd.