I want to use NetCode for MMORPG-scale games, but I’m not sure if It’s a good fit for this purpose. My task is to make sure that there can be from 1000 to 3000 players on the same server at the same time, but according to the description of NetCode, at the moment It can handle about 80 connections. In the future, as I understand, they promise to process 80+ connections, but how much more than 80 will it be? Will it be enough for my purposes, or should I use my own network solution?
MMORPG networking is very specific. In my experience (which is very near to none in practice, as I mostly did research about it), you generally want to have ver lightweight algorithm in order for your server to be able to process those 3000 players without blowing up. From what I know, the current NetCode is oriented toward the “MOBA” game-style, and the last time I checked, they talked more about 16 players than 80, but my info are probably outdated.
Their goal is to provide a network solution with authoritative server, client prediction, reconciliation, rollback etc… This is something very important for say, FPS or actual MOBA, but you most probably want to avoid that for an MMO architecture.
For example, again from what I know (did that research about 8 years ago), World Of Warcraft movement system uses simple “commands” for movement where the client send messages to the server like “I start to move in that direction”
“I stopped”, and the server simply replicate this to the players around.
This means there’s is no “world state” being sent (full or delta) at specific time.
Of course there are probably some “sync point” or some information sent to ensure simulations doesn’t get too offseted (this is why you can sometime see your mate “teleport” around when the connection is bad). In addition, the server probably also check some kind of validity for the movement (when the game launched, you could cheat and teleport in the world by sending false info to the server).
I think the NetCode is absolutely not built toward the idea of an MMORPG, and I believe there’s something about an MMO architecture on their (long time) roadmap.
If I had to make an MMO today using Unity, I would probably try out their “Transport” package with a very simple “command” system like Wow did in their early days, adding complexity and security on the way.
I hope not to be too wrong.
6 Likes
Thanks for the answer! I will look for solutions.
The question is not if Netcode can handle it but how much CPU stress a rollback netcode/actual simulation puts on the server. (it’s a lot with 2-3k players) Which means, lots of server costs and thus, not advisable.
MMO netcodes don’t revolve around precision and even anti hacking as much as rollback so the approach is very different. Most MMOs, even AAA, you just send a position and the server believes you. You can see with that alone how much they try to keep the cost and simulation time low. Sometimes there’s not even a real simulation.
Currently I think you’ll have more success for MMOs with frameworks like Unet/Mirror, etc… Keep it simple and scale from there.
1 Like
For a networking stack to be MMO ready there are some stuff needed
1- the net lib should be able to handle lots of connections and messages per second
2- it should be able to do prioritization based on area of interest and other mechanisms so each client should only receive the data related to people around him and not more
3- the server should not implement memory and/or CPU heavy techniques like rollbacks.
Take a look at big world and hero engine and their docs as good examples. The new transport however can be tested and if it works then you might be able to start from there but I doubt that they tested it in MMO scale yet, this said however they are building an open world shooter to test these things I guess so unity will build at minimum a fortnite like multiplayer game internally with say 50k objects and 100 players and will try to make it work. if that happens then probably using transport layer or a heavily modified netcode will be the way to go. keep the heavily modified with a grain of salt since I don’t know how customizable and modular the netcode will be. if it becomes possible to disable rollbacks and change the way snapshots are generated then you might be able to use a really good version of it without much modifications
this is a very good question, anyone got info on using it to sync swarms of entities to a smaller amount of player ? (like < 30 players) any idea what would be the expected capacity in the current state of the system ?
1 Like
Unity transport in theory should be fine. Networking itself is mostly IO bound. But that they aren’t testing it for larger scale means it probably has issues. And they closed off the source, so not really a great option for using in an mmo at this point.
But you won’t be running thousands of players per server on Unity, for other reasons. Most mmo’s use regions/zones, so you have a natural partition that reduces the number of players to at most several hundred. And Unity can only run a single scene/map at any one time. If you are thinking open world/float origin, think again there also. Shifting as most understand it in the context of Unity isn’t an approach that will work on the server, and other approaches that can work are complex to say the least. Realistically not doable for a small team with no domain experience.
So realistically you are probably looking at around 200 or so max players in a region, maybe half that in npc’s.
So all told you are probably better off using something like this that also has reliable messaging:
Or take a C/C++ networking library and pair it with something like this.
As a point of reference we use .net core server side with our own C networking and the above mentioned reliableio library. Our networking and reliable message handling is single threaded and can handle around 5k concurrent connections pushing 60 messages per second each. Which is roughly double what our design peak is per server.
6 Likes
- How do you handle Physics & Collisions on Server side if you are using .net core and not the Unity itself?
- In case you are using a different physics framework.Are you getting a deterministic result ?
- using Unity Headless builds isn’t enough ? Why?
- I think the DOTs will fix all these problems when it will be more stable.
Eg: Tiny package will be the perfect way of having Headless, Costless and Fast Servers.
@artyomik14
5) Personally i never tested the Transport Package at a high scale (thousands of connections) but i think with good Technics of separating players and environment Entities using regions to limit the data sent to each entity this package can handle thousands of players. (just my opinion) at this stage i think only Unity can really answer this question correctly.
Hi, bringing my unexperienced self on the mater,
I think DOTS will handle quite well separation of player, you can use ISharedComponentData for region/Map and if need be you could probably wire up something more convoluted with several worlds.
As for the fact that unity only run one scene map at a time, first I’m not sure it’s accurate whe nusing dontdestroyonload it’s represented as a scene in the hierarchy, hinting me to think it can handle multiple scene at a time just fine. And event if not hte case, the subscenes will certainly help move toward more flexibility.
You also seem to imply that the source code for the transport layer is not publicly available but the transport package is available through the package manager, you should be able to see the sources just fine.
If I miss understood and you were refering to sealing classes and private attributes, well you have the source so nothing stops you from cahnging what you need. you can even give feedback to unity as to why you need those cahnges. If they see a real use case, I’m sure they’ll think about it. An if by any chance you are doing something wrong, you’l probably get some explainations as to why you should do that.
1 Like
The C source is closed. It was initially open then they stopped providing it it. This is an issue if you want to integrate anything else in C with it like reliable messaging. The reliableio library I mentioned for example uses callbacks for sending the message to the network.
Reliable messaging is best handled all in C otherwise you cross the native/managed boundary a bunch unnecessarily.
One of the advantages of reliable.io is full control over retransmissions, the library itself is basically a flexible NACK-based notification layer, so buffering of reliable messages is up to you, works best in scenarios where packets are shuttle across endpoints continuously, not very suitable for turn-based games.
As for scalability in the Unity, just as Chris said, you have to distribute workload across small nodes/instances, monolithic architectures is where Unity falls apart because of its constrained environment. The data-oriented stack is not helping here because it aims fine-grained processing which is an opposite thing to coarse-grained server systems.
When you are building something truly scalable, you are bound to network I/O anyway, so you have to step over it. As one of the options, you can use a high-performance platform-specific network API such as Registered I/O on Windows and io_uring on Linux. These APIs not requires specific NIC, but you still need good hardware to benefit from it. Depending on how these APIs are approached, you can achieve a great throughput, here’s one of my implementations that allows to fully utilize 1Gbps Intel NIC transmitting more than a million packets per second over the network using UDP. This should be a good starting point.
1 Like
Note that game-specific high-level abstractions play a very important role as well in terms of flexibility, efficiency, and performance. There’s nothing off the shelf in Unity, including 3rd-parties that provides this for truly scalable virtual worlds.
1 Like
Can you briefly elaborate why do you think DOTS approach is not suited for networking? I never really wrote robust networking game layer, but from this point of view, it seam as beneficiary that you get to select only subset of data in quick and easy manner and pack it for transmission. ECS is sort of based on DB querys from my point of view.
1 Like
This is a quite complex topic, so long story short: server architectures (and games in general) is not about moving 100,000 boids in parallel in the most efficient way, it’s about a balance and harmony of all systems with different granularity. When you are obsessed with only performance, blindly chasing the idea of the high speed, you are by definition building a foundation that doesn’t meet the above requirements. When the way of how you are approaching data, the concurrency model, and other aspects are shaped to solve very specific problems, you are building yourself into a corner.
1 Like
You could try https://improbable.io/multiplayer-networking out of box sample runs 200 first person shooter bots/players. I imagine it could handle a lot more simple mmo players.
Not sure why most of the people think “…oh, parallel” when it comes to DOTS. But I don’t see parallelism as most beneficial advantage of DOTS. You could have offloaded native data calculations on another thread even in the past, it was just not this easy. IMO, main thing is ECS and the fact that you don’t need to cache instructor pointer and function params while jumping far and wide across the RAM, just for the code part, plus handle cache misses because you have lots of useless data in it. Which you get if you are hard-core OOP guy, which I am, don’t get me wrong.
That’s why I see DOTS as advantage when it comes to packing data for transfer, and only the actual data you need. The problem here is that you need to have the same kind of approach on server. Again, never wrote a line of code for game networking, I might have more credits to comment about this in a couple of months when I get there.
There’s a ton of expertise needed for a MMORPG server but I think we can agree, when the simulation runs more optimized that’s a big plus and that was something Unity wasn’t able to do previously. If you wanted high concurrency with somewhat of a simulation you had to make your own server and now we slowly start to get the option to write a scalable simulation server directly in Unity.
Of course you’re right when you actually build a monolithic approach in Unity but why would you? And when talking about monolithic, sometimes you can’t break your architecture apart any further, mostly the simulation servers, so sure, you can build all those scalable services like login, chat, matchmaking, lobbies. I suppose you could even make these services in Unity but there’s better tech for that.
Anyway, I just want to say for MMORPG simulation servers, DOTS feels a 100% fitting. It’ll lead to an eventual Fortnite clone that runs on a Unity simulation server. There’s no denying there are more puzzle pieces on the table than just the sim server but that’s more the nature of the beast when it comes to MMOs or something that has 100s of players in general.
Unity is not a scalable environment for a server DOTS or no DOTS.
Game engines are based on a game loop, and DOTS is based on having a single synchronization thread. That is quite the opposite of what you need to scale on the server.
Server scaling is focused around being able to respond to requests in a timely fashion. Serving potentially tens of thousands of requests per second or more.
At scale what probably matters the most is what is your standard deviation. The patterns you need to use to scale at low standard deviation over many cores is different then what DOTS does. DOTS uses the same core principles any good concurrent system does, ie single writer. But from there it uses approaches that work great for a client or small scale server stuff, but not so good for any real server scale.
In a vertically scaled server environment your networking would either be single threaded or possibly a number of pinned threads. Those basically just send batches of requests to other threads. Several ways to structure the flow but basically you synchronize at the batch level. Global sync points are the bane of server scaling. You have to synchronize at multiple points on different threads. The key is using strategies that reduce how often as well as partitioning the work so you synchronize as few threads as possible.
Now that said you don’t need to vertically scale server side. Most games don’t and that’s smart big picture. You can’t afford to engineer every game for what will work best and be most efficient in production. It doesn’t work like that in the real world. What you can afford pre launch is an entirely different thing then what you can afford post launch with some success. Your engineering cost as a percentage is completely different. I’ve worked on games where we easily paid high 5 figures per month that better engineering would have reduced by a significant amount. Quite a few games have scenarios like that.
3 Likes
Forget about it.
There is a couple things we have planned to solve this:
- Dots Runtime can be used as a super lightweight server environment that runs on top of .NET Core. You can use the tiny package to export to this at the moment. We don’t yet have this hooked into our own demos with DotsShooter & Netcode samples because some some of our dependent packages like animation system doesn’t yet have support for Dots Runtime due to a couple uses of reflection. But we intend to solve that end to end, before the end of the year.
Essentially our aim with Dots Shooter is that the server fully runs on a .NET core Dots runtime server (Without using any of the existing standalone player hybrid functionality)
Dots Runtime gives you full control over the main thread. You can write your own main.cs and thus you are fully in charge of how things are ticked updated etc.
- We will add support for running worlds on a manually scheduled thread. Sometimes you just want 8 cores each with their own simulation pinned to a thread and configured to not actually schedule jobs but run everything on core (We already have configuration in the job system exposed to C# where you can simply set the number cores the job queue uses to zero)
These kind of things aren’t built in one day. But we are definately working our way towards making sure that our foundational features support building an incredibly efficient MMO server architecture on top of.
12 Likes