How to create a generic K-V storage in netcode?

Unity Engine
, , , ,
1

I hope to create a generic K-V storage service to sync rather complex data of my seprate logic module to all clients. I hope this to work like cluster mode of redis, that is, only server set values and clients only sync values and read-only.

But in netcode, everything is wrapped with NetworkVariableBase and I’m thinking about how to create this.

For instance, Maybe I need a NetworkBehavior implementing this interface

public interface IStorageProvider
    {
        public T Get<T>(string key) where T : unmanaged, IEquatable<T>;

        public void Set<T>(string key, T value, TimeSpan expiresIn = default) where T : unmanaged, IEquatable<T>;

        public void Remove(string key);

        public bool Contains(string key);
    }

So in NGO, do I have to manually use a NetworkVairableBase of NativeHashMap or something? and how could I properly use serializations of values? Cound anyone give me a simple example

2

At the most basic level all you need to implement is INetworkSerializable.

HOWEVER this would mean you may be synchronizing the entire dictionary every time a value changes. Therefore it’s highly recommended to code it so that you keep a separate set of “dirty” values and synchronize only these. You also need to encode what the dirty state is (value changed, added, removed).

You could also quite simply use two NetworkList, one for keys and one for values. Wrap them both in a generic class and you got yourself a networked dictionary.

But do keep in mind the synchronization issue and how much traffic this may generate. If your storage could grow to several dozen KB and a single value change would trigger a sync of the entire dictionary this would quickly become unviable, especially if value changes might happen frequently eg every second or even more often. NetworkList etc already have a way to only sync their delta state. If you don’t use them you’ll have to implement this yourself.

Since you mention redis, are you aware that you could either use that service or firebase or Unity’s Cloud Save service?

2 Likes
3

yeah creating a class implementing INetworkSerializable and my IStorageProvider and manually manage dirty values and update events so that syncing deltas only, or use two NetworkList are both good idea. But now my problem is how to efficiently sync value of unknown type. See my value can be any unmanaged type restricted by IStorageProvider T like int, float or some custom struct, typically wrapping it with NetworkVariable<> source generator would generate the class with known generic type. But now I dont quite understand how to declare the type of the storage container in the lowest level so that I data can be stored uniformly.

For example if I try to implement INetworkSerializable should I declare a Dictionary<string, byte[]> or a NetworkVariable<NativeHashMap<FixedString64Bytes, FixedBytes64Bytes>> in the class or something else? If so what Marshalling api should I use? MemotryMarshal, ManagedMarshal which is provided by C# or some api provided by FastBufferWriter/Reader ? I hope this to be same implemented as Netcode internal and cause as less performance & latency waste as possible.

And yes I considered unity’s multiplayer service, but in my case I need rather high real-time so unfortunately I did not look deep into them.

4

The unknown types have to implement INetworkSerializable.

I made this just yesterday. I have a spawn message class (I use custom named messages) which contains two indexes and a ActorTransform which is another struct implementing INetworkSerializable in and of itself. In the spawn message INetworkSerializable implementation I simply call writer.WriteNetworkSerializable(transform) to pass the serialization further down.

Then I had to make a message to spawn the initial, already-spawned actors for late-joining clients. It would be highly inefficient to send 100 individual spawn messages. The new “spawn many” class simply wraps the spawn message and serializes it into a single writer using the same principle as above.

Personally I would use fields that you can work with. For instance my ActorTransform contains a Vector3 and a Quaternion (position, rotation). But I may not be sending these, although the BytePacker.WriteValue(position) (and rotation likewise) are quite efficiently packing these values without having to resort to half (16bit float) conversion. But the option exists, and this decision should be handled solely by the serialize writer and reader implementation rather than as the struct’s fields.

Most definitely not those.

For best efficiency without resorting to a) lossy compression (eg half floats) or b) restricting value ranges (eg uint indexes rather than int) or c) compression (BrotliStream) you should use the BytePacker and ByteUnpacker methods.

If there’s a type these don’t support you should be able to write an extension method that handles your type but most of the time this won’t even be necessary. Consider the Guid struct. Internally I believe it uses 4 integers (or long not sure), so the extension method to handle Guid serialization would only wrap four BytePacker calls.

But there’s another option: var data = new ForceNetworkSerializeByMemcpy<Guid>(new Guid());
Example is in custom messages page.
This forces a non-INetworkSerializable struct to be network serializable. I don’t know if this has any restrictions or what they are.

Be sure to use the RuntimeNetworkStatsMonitor while testing to quickly assess the amount of traffic.

Whether BytePacker, lossy or Brotli compression or sensibly using smaller types (eg byte as index for small collections) produce the least amount of traffic can highly depend on the values being sent and their most common average. For example, if position of objects is likely to remain below 240 units the BytePacker can pack the positions quite efficiently. Also, if you know you have an angle between 0 and +180 this would neatly fit into a byte if you can afford the loss in precision (1 degree steps or a bit more if you normalize it to 0-255 range).

What I mean is: Traffic optimization requires knowing your most common value ranges and experimenting with different options in sending them. This could include sending data at different rates, eg I will update my enemy’s target transforms only 5 times per second (default is 20 times) since interpolation will easily handle the steps in-between and possible slight deviations for clients doesn’t really matter here.

5

I guess in my case I’m struggling seeking for a too generic K-V storage provider, quite like implementing a distributed cache system on top of Netcode transport, then I’m stuck in how to manage and declare low-level data storage, because I had to use a unified data type to storage data for generic types, meanwhite avoid marshalling/unmarshalling data on every get calls.

However maybe actually I should take a look at Roslyn source generator and find some way to generate every network data sync providers for every dedicated types… I will look more into these two ways,

6

What exactly is the goal you wish to achieve?

You keep mentioning marshalling, together with other statements it sounds like you wish to be able to store literally “everything”. Any Unity and .NET Framework type possibly? I can tell you that this is near impossible to achieve. It’s always going to force you to select only those properties that you wish to reconstruct.

For instance, if you want to store a Sprite instance, you cannot serialize this type without custom code that serializes the sprite’s texture, a reference to the material or the material itself, and some of the sprite properties.

On the other hand, if you create a game that intends to store a user’s sprites, all you really need to store is the source image (png, jpg) because the game declares the requirements (eg min/max dimensions, aspect ratio) and ensures they are adhered to. The material is also a specific one set up for this purpose.

Also consider that databases generally avoid storing objects (blobs), most data in a database is comprised of primitive types (bool, int, float, etc) and a few common, general purpose types (string, date, time, guid, blob or byte array). These are really all you need to implement!

You can derive everything else from these kind of data types. Databases are commonly used together with an application layer like Hibernate to map the data to and from instances. This transformation layer runs on the clients, not on the database.

Taken to the extreme are common json-based web request APIs including Unity’s services. Keys and Values are both strings. Conversion of a string value to the actual type happens on client-side.

1 Like