With the new binary format for ECS being announced I was wondering if I could get some clarification on what could actually be done with it. Is the format purely for serialising scene and object data or could we also use it to serialise our component data into binary?
I have huge amounts of simulation data I want to stream in from an external source. At the moment I’m using protobuf which gives me the fastest possible read/write speeds compared to anything else I’ve tested. We run our simulation on seperate software and create the binary file there, then our prebuilt app points to it and loads the data into memory. The issue is that protobuf uses reflection which prevents us from building with IL2CPP and as a result, loses us runtime performance.
Can our work flow be supported by the new ecs binary format and if so, is it worth switching from protobuf to what unity are offering?
1 Like
To clarify further, this data is in the form of int, float, short, byte and vector3 arrays. So all of our data is compatible with DOTS.
If you are using protobuf use protobuf.net, it has a precompiler that can generate AOT friendly dll’s. Although it’s kind of a pain to setup but once you do it just works.
Protobuf is not going to be good at loading large amounts of data though. It’s actually quite slow because how it encodes/decodes. It was designed for space not time efficiency and small messages.
If you want fast then something like Cap’n proto or flatbuffers would work a lot better. They weren’t designed for big data either but they lay out data much more efficiently if speed is what you put a priority on.
Sorry I actually meant to say protobuf.net. Even with protobuf.net the use of reflection when serialising and deserialising it simply doesn’t work on with IL2CPP.
Thankfully none of our data sets are small. At the file sizes we’re looking at (which is up to a gigabyte when saved using protobuf.net) I haven’t come across another serialising API that manages to read files that large as fast as our current method does. But with the introduction of ECS and our dire need to increase runtime performance with the hybrid renderer, it would be ideal if even our serialisation method could be provided by Unity so that everything is guaranteed to work on IL2CPP. Joachime Ante described it in one of his talks as a “runtime ready binary format” which is exactly what we want.
If using the ECS binary format for our needs isn’t possible then we’ll stick with protobuf.net. It’s just frustrating knowing I won’t be getting the absolute best performance possible D;
Yes it does work on IL2CPP I’ve been doing it for years. You need to use the protobuf precompiler.
The binary format serializes all IComponentData and IBufferElement. So it’s a good fit for efficiently storing large simulation data. The format currently does not provide data backwards compatibility, so you want to treat it as a cache you can rebuild in your build process. In the case of megacity we do that by rebuilding from the original game object based scenes & prefabs.
2 Likes
MsgPack and Protobuf both have precompile APIs to avoid reflection. I haven’t looked at the state of anything else in a while.