I would like to show you my progress in creating a high level api based on Unity ECS on top of the ENet network library.
The test was performed with 2 standalone client builds (right side, 2500 connections each). The server processes approximately 320000 messages per second with a tick-rate of 64 ticks per second. Each of the clients sends its position to the server.
Current project is heavily utilizes techniques such as:
The goal of my test was to demonstrate you the power of processing the messages in Unity ECS. However the most consuming part is serialization / deserialization that is performed in C# Logic thread via BitBuffer.
As for the techniques, It does not use any delta compression, because storing a world snapshots history for each of 5000 clients would consume a lot of memory. However, you, might look forward to http://www.gamasutra.com/view/feature/129854/ this article explains how to synchronize entities under bandwidth constraints. Recently I’ve got good results with 6 bytes for Position and 4 bytes for Rotation, here you can see some statistics of receiving the data of 500 clients (~20kb / s) with a tickrate of 20 packets per second:
That means that a server requires throughput of 80Mbps in order to allow all of 500 clients to be in one zone of their interests.
80Mbps, Is quite size, but I think that quite average this days, for typical Joe.
Of course I mean, if that is for the client.
For a server that is quite decent.
Surely with added prediction and ensuring determinism, that could increase number of concurrent moving instances.
Either way, nice presentation so far.
PS. I check gamasutra link about sync, when I got suitable time.
With authoritative server and such number of clients, for deterministic system with prediction, rather negligible, since you only send user commands, and receive back character syncing anyway.
Well, I’d not say this is true to be honest. In a fully deterministic system absolutely this would hold, but for a non-deterministic game like an FPS accounting for all player actions that can happen in the world can be very expensive in terms of bandwidth.
How are you transfering the data from the ECS/game thread to the logic thread (and vice versa i suppose)? It’s possible to setup a jobified serializer in the unity ecs without too much effort, allowing you to easily multi-thread the serialization.
This seems really low? 20kb/s for 500 clients with 20 updates per second? Just simple math of:
500 (client count) * 10 (position and rotation byte size) * 20 (packets per second)
Would put you at almost 100kb/s per client?
Edit: Really great work, but I also want to add that be careful of measuring how many clients the system can handle when using local host or local lan connections, as they are much more stable and performant than what you will see online.
RinBuffer of IntPtr. Single consumer single producer. Trust me, it’s faster than schedule a job.
The updates are not coming all at one tick. Each of the objects sends its position in a different time, however with the same interval. So you’re ending up with less than that. + It’s not exactly 20. The closest entities are getting sync at this tick-rate. The one that is far send twice less or even more less updates.
The system would be stable the same as the LAN demo. The reason for that is it does not use any reliable messages for position updates. So the server would not waiting for anything if there would be any packet loss.
You serialize with Span when you produce, deserialize with ReadOnlySpan when you consume. All of that allows C# threads to talk to each other without blocking. For Network messages I use BitBuffer that later on converts to IntPtr of deserialized struct and then just passed to an Entity as NetworkPacket component. Later on I can call UnsafeUtility.CopyPtrToStructure either in Burst/NonBurst Jobs to get the packet data.
Sure, but wouldn’t you get a much higher throughput by not having to pass things to/from a single background thread and simply run the serializer as a parallel job? This would allow you to make use of many more threads. Maybe you are already using a multi threaded serializer and I missunderstood it.
I’ve done a lot of testing around this myself, and been able to reach staggering amounts of serialization throughput on my 32 thread CPU, like hundreds of thousands of entities.
The logic thread performs serialization / deserialization of network messages. While a game thread is free of it as well as job threads. With bit buffer Serialize / Deserialize are quiet fast operations (faster than any High Level Serializers (MessagePack, Protobuf, etc). But yes, I see your idea. It still possible to have jobified serializer / derserializer, however I think it would perform better only in a case when you have to serialize ~10.000 messages at one frame.
As far as I remember it’s just an IJobProcessComponentData with ReadOnly attribute of NetworkEvent component with a switch statement of enum event type: OnConnected, OnDisconnected, OnTimeout; inside the job.
Well, you usually have a set CPU budget for each game instance on a server, so it doesn’t really matter how you split the code over the threads.
I would never dream of using something like a high level serializer for network packets
Not sure I follow here, I ran my tests with the target of having 1000 live entities in a world with 100 players, where you have 100*1000 candidate pairs for prioritization, serialization, etc. Of course all entities can not be written to each client, so there’s a lot more logic going on than just writing packet data, and being able to do this within the job system works really well.
I’ve just decided to pick a regular C# threading approach but apply Unity’s Jobs for actually processing of the data and performing of the game logic. Thank you for your feedback.
I am looking at enet based upon feedback from @nxrighthere performance GitHub project and assessment on new Unity networking stack. Any chance you are sharing your test code? Seems like you have found a great combination.
