It’s such a hot day today I could not code. Instead, I started writing down how to work with the ProTiler grid/map data model since that is already very well fleshed out and supports any kind of (value-type) data in chunked maps, laid out in either linear collections (one instance for each coord) or sparse collections (only instances for coords that you set data to).
Let me know what you think of the first draft, any questions or any feedback is very welcome! 
(some of the text is missing context, like there’s more to be said about serialization)
((the entire doc is available online which is both my tech design, old and new, and starts with some completely unnecessary ProTiler history that I wrote so I won’t forget))
=============== Manual Draft Excerpt ================
ProTiler Data Storage
ProTiler’s data model allows storing any value-type for any grid coordinate, where the map is divided into custom-sized chunks.
Storing Value-Type Data
That means any simple type (int, float, bool, byte, enum) and any struct containing simple types. Even native collections are possible, such as BitArray.
Example data struct:
public struct MyData {
public int MyValue;
public byte MyFlags;
}
Serializing Data
Note that you do NOT need to flag the struct as [Serializable] nor do you need to attribute the fields.
What gets serialized, and how, is determined solely by the IBinarySerializable interface methods Serialize() and Deserialize().
You do not normally need to implement those for simple value types or structs containing simple value types such as Vector3 or int4. But you can use it to, for example, serialize an int as ushort if you know that it will never exceed 65k.
Most importantly, you can use the IBinarySerializable interface to serialize collection types that aren’t supported by ProTiler (yet), such as UnsafeBitArray or UnsafeHashSet. I do aim to support all collection types in the future by writing IBinaryAdapter implementations.
Here’s a complete example how you would write serialization code manually. As you can see, it is far from difficult:
public struct TestData : IBinarySerializable
{
public int3 Coord;
public UInt16 Index;
public unsafe void Serialize(UnsafeAppendBuffer* writer)
{
writer->Add(Coord);
writer->Add(Index);
}
public unsafe void Deserialize(UnsafeAppendBuffer.Reader* reader, Byte dataVersion)
{
switch (dataVersion)
{
case 0:
Coord = reader->ReadNext<int3>();
Index = reader->ReadNext<UInt16>();
break;
default:
throw new SerializationVersionException(dataVersion);
}
}
}
Note the use of the unsafe keyword due to the use of the UnsafeAppendBuffer* pointers. The script must be in an assembly definition that has the “allow unsafe code” checkbox enabled and the method must include the unsafe keywords in its definition.
Also take note of the dataVersion byte. When serializing, you will be able to pass in the “current” version. When you do make a breaking change but you want to keep the old data still deserializable you can use the dataVersion byte to deserialize previous formats so that the designers don’t have to trash their worlds and start from scratch.
A breaking change is when you modify the struct by:
Then, when reading the data and the version is not the current-most version, you would:
-
read the data of a removed field, but discard its value
-
read the data of a field using its previous Type, then convert it to its current Type
-
compute/assign a non-default value to the new field that must not have a “default” value
The next time this data is serialized it will be serialized with the current dataVersion and can then be considered as “upgraded”.
Since supporting many previous versions of data will become cumbersome you should remove previous version support from time to time. For example, you could read and write all data once to ensure it is up to date, then remove all previous version support and reset the current version to zero.
Do not worry about the byte not supporting more than 256 binary versions. Rather, be happy about not having to test, debug and support this many backward versions. If you still feel this is restrictive, you can always introduce an entire new data map with a separate struct. You may want to do so anyway because data should be tightly coupled by use. Thus it’s perfectly fine to model your data with only maps of simple types like int, float, Vector3 but you lose expressiveness in the code: What was that Vector3 data chunk about again?
Data Chunks
The grid map is divided into chunks of X/Z dimension with the minimum chunk size 2x2 (*) and no limit on the upper bounds. Chunk sizes need not be square. Data chunks are structs for burstability and jobbifiability (yes, those are actual words but you are right, I made them up) reasons and thus cannot be subclassed and may only contain value types themselves.
*: The minimum size of 2x2 is to prevent collisions of the chunk coord hash function.
There are two kinds of data chunks:
Linear Data Chunks
Linear chunks always keep one value-type item for each coordinate in the chunk. Chunk data can be accessed and iterated in a linear fashion like an array.
Data is laid out in X/Z layers, one per Y. Initially, a chunk reserves no memory until data is set to it. Then it will expand its size to include the topmost Y layer such that setting data to grid coord (0,3,0) will allocate 4 layers of data (array length: 4XZ).
Linear data is most useful for data that needs to be iterated over in a linear fashion, possibly in parallel. For example, voxels in a voxel map will benefit from a linear data alignment.
Sparse Data Chunks
Sparse chunks store data per coordinate. This is meant for data that is sparse, eg randomly and rather rarely set to some of the coordinates in a chunk. For example, flagging some coordinates as spawn points or pathfinding vertices.
Internally, a hash map is used to look up whether there is data for a given coordinate.
Custom Data Chunks
For advanced uses, creating custom data chunks is possible. Any unsafe Unity.Collections collection can be used to store data, and they can be nested too.
An example custom data chunk could be a specially optimized linear chunk for a voxel map where it is beneficial to performance to lay out tiles in vertical stripes rather than in XZ layers. An example usage could be a “falling sand” game where most of the time “pixels” are moving along incrementing or decrementing Y axis and the algorithms that work with that game mechanic will more often look up grid data above or below the current coord.
Custom data chunks require writing a custom binary serialization adapter however, so it is recommended to familiarize yourself with Unity.Serialization and binary serialization concerns such as versioning data. The provided (Linear/Sparse)DataMap(Chunk)BinaryAdapter implementations will provide implementation guidance.
Data Maps
A data map contains a collection of chunks of a given kind. So there are two main DataMap types:
All data map classes must inherit from the abstract base class DataMapBase. You can of course make a CustomDataMap for your CustomDataMapChunk type by following the example of one of the above two classes. Again, this is for advanced users and requires custom serialization code, but it isn’t too hard and I will support you in writing that.
The Grid
Lastly, the GridBase class stores LinearDataMap and SparseDataMap instances in separate lists. That is, even if you only use one of the two data maps you will have lists for both but the unused list will not allocate memory of course.
Naturally, when creating a CustomDataMap you will want to subclass GridBase and follow its template in extending it to also support CustomDataMap instances.
GridBase also handles all serialization concerns and has a list of binary serialization adapters. By default, you will not need to look into this, but if you want to create custom chunks, custom maps or otherwise have needs to customize serialization, this is the class to add your custom IBinaryAdapter classes to.
In normal operation, a user would simply create a concrete subclass of GridBase and add whatever linear or sparse data maps are needed to store grid data. Initially, ProTiler will ship with Tilemap3D and VoxelMap concrete GridBase subclasses that handle rendering of 3D tilesets and colored / textured cubes.
UML never alerts you to compile errors that might invalidate the whole design - you know, stupid but understandable oversights such as a design built on structs with an “abstract base struct”.
