Unity MMO capability questions

Unity Engine
1

Introduction
Hello. I have a few detailed questions about networking. I don’t expect them all to be answered in one post. Answering just one question, or sub-question, is fine. I may update this post if anything else that concerns me comes to mind. If I do update it, I’ll be sure to write a timestamp next to it.

Situation
These questions are in relation to Unity 5.3.3 Pro (I say Pro because some resources may come in handy here) and a Linux operating system - preferably Debian Wheezy.

Questions (all questions answered - new ideas posted here)

  1. Should I split the chat server and game server to minimize lag and possible bottlenecking?

  2. Will Unity3D properly make use of multiple cores naturally? Is there anything I need to do to aid Unity3D in utilizing difference cores for different operations? If the answer to both of these questions is no, then is it safe to assume that getting multiple servers with single cores, higher frequencies, and better cache is the better route to go?

  3. Scenario - If the game were to, hypothetically speaking, achieve 10,000 players and the maps were big enough to spread them around so you don’t have 100 players in one spot:
    A) Should I separate the map in to “network”-chunks and have a server for each “network”-chunk (visually the map will load in chunks on the client, but I’m referring to the network aspect at the moment)? To be more clear, rather than handling calculations and network transfer for every player on a single server, should I figure out some way to break up the work between different servers? If this is the case, what would be a theoretical method to achieve switching between servers seamlessly. I say “theoretical” because I can do research myself, I just need the general idea on how to go about doing it. Such as should the two server boxes have the same IP, different port, and then the client just connects to the second port when they enter the new chunk.

B) If I were to give up the server-authoritative scheme and let the client handle calculations (ultimately offloading processing power from the server), should I still separate the map in to “network”-chunks or is it safe to assume that a single server can handle simply sending packets (providing that the network allows it)?

timestamp: 6:30pm EST - 2/25/2016
I talked to Kennux for a bit and came to a conclusion about my server model. As an example, I have 10 instances of my game to break up the processing between 10 chunks. These instances go on one server box. In addition to that, I have 2 server boxes totaling 20 chunks / instances. In crontab, or whatever I may choose to use for startup commands, I tell it to run the game’s file 10 times. Every instance that starts begins to send packets every 10 - 60 seconds. These packets request instructions from a master server. When it finally get instructions from the master server it stops sending these packets. The master server then increments its online chunks parameter. Until there are 20 (max) chunks online, no players can join. The master server holds the server data (IP and port) and tells the player which server to join.

  1. There is something I have never understood how to efficiently accomplish. How would I prevent a player from hosting a “network”-chunk if they were to have a Linux OS? My previous networking project tried to attempt this by having a few checks: It checks that the host IP matches the subdomain of a website & that their is a file present with certain text present. I used PHP on the domain to also get the current IP of the host. This seems like a bit too much. Isn’t there an easier way to go about this? Especially since not all servers may have the same IP.
3 Likes
2

I dont want to be negative, but to make the story short, unity is not mmo capable out of the box, never was and never will be any time soon.

dont beat a dead horse with a stick,
instead, try a really mmo capable engine built for specially to handle such a task, ie:
hero engine, big world engine you cant go wrong with any of these 2 mmo capable engines built from the ground up specially for that.

3

I appreciate your concern for not being negative. Thank you. However, I still feel that some of these questions would be beneficial wherever I go. If you could perhaps give me some insight on on how to tackle organizing the networking that’d be great (basically answering any aforementioned applicable questions).

4

Since the new networking in unity 5 you should be able to implement mmo-like networking without too much trouble using the LLAPI which takes all the networking stuff already away and you just need to implement the highlevel logic for your game.

Anyways i dont think you can use the HLAPI without any modification for creating mmo games.

So, to your questions:

  1. I would seperate them, for 2 simple reasons: Make cross-world/instance chatting possible (example for a friendslist) and it doesnt really need to run in the world server.
  2. Unity does run stuff on different threads, if im not mistaken unity physics, rendering and your game update logic all run in different threads (Physics does for sure: Unity Blog). You could also just host multiple instances of your server that all handle different regions / instances for the game to make use of the other cores.

A) This is depending on how much load to expect on a single instance. You can for example also just seperate world simulation, npc logic and character / login server. But if you are going for a classical mmo with a pretty big world imho the best should be to seperate the world into chunks and let each server handle a single chunk.
You can keep your world streaming in sync with your chunk servers, so when loading a chunk in your world you also connect to this server and start accepting data from it.

B) Just dont do that for an mmo game at all (Cheating / Hacking in MMOs is a very important topic and client authority lets the door for this wide open :p). But if you want to, it again depends on how much load to expect but a NAT punchthrough / Relay server is ways less processing expensive.

5

Good point. I guess that clears that point up.

I guess I’ll experiment with some things. I’ve never tried to closely examine the processor when running Unity because I’ve mainly done single-player games. Thanks for the insight.

Yeah, I briefly thought about the whole cheating thing. The idea to drop server-authorization in trade for less processing on the server’s side spawned from my idea to have player moderators. I was hoping to have the moderators keep rule-breakers in check. I’m still leaning more server-authorization but removing it is on the table for options. If I could somehow minimize how easy it is to hack, that’d be awesome.

The only thought I had to somewhat solve this, and it’s purely a theory, was to handle server-auth over time. In short, rather than handling them on every request, handle them every 10 requests, as an example. Another thought was to have nearby clients handle the authorization (no, there won’t be peer-to-peer). Sort of like a client-server-client authorization. It’d increase data usage but would share processing between everyone. Just thoughts.

Just to be clear, my questions are already answered. But, it’d still be nice to still have a discussion so I can make plans before I start programming. That way I can get everything done without having to fix huge mistakes.

6

Added question 4.

7

How do you envision using the LLAPI to control network traffic server side? Let’s say you want to handle position updates of players with LLAPI. You can either use Update() which is bound to frame rates (which a headless server does not have) or FixedUpdate() which is useless for anything not physics related (http://forum.unity3d.com/threads/the-truth-about-fixedupdate.231637/). So how exactly does one retain control and control whether he send packets to a player at a rate of 20 times per second or 30 times per second?

8

HeroEngine really isn’t even an option any more. They have run into financial trouble and have had to shut down most of their servers and stuff. The biggest Indie MMO being created with it (The Repopulation) have had to switch to Unreal Engine 4 so that they wouldn’t have to deal with the issues that Hero is currently facing. So if you want your game to have a Stable future I wouldn’t go with Hero until they get everything sorted out. BigWorld I don’t know much about since I have never used or been associated with anyone who is using it. (Repopulation did for a short amount of time before they switched to Hero).

9

Do you have any proof behind this “financial trouble”? I’m not saying you’re incorrect, I’d just like to know your source.

10

I don’t know the complete details. I stopped using Hero a couple years ago BUT here is some articles concerning what’s been going on with The Repopulation:

https://community.heroengine.com/forums/index.php/topic,6099.0.html

11

damn, I was not aware of hero engine issues, it has been like almost 4 years since i stopped using it, altho seems they are back in business as per this update:
https://community.heroengine.com/forums/index.php/topic,6104.0.html

also did not know the repop stopped their dev in hero to switch to UE4,

I hope heroengine picks up back again. otherwise they might go open source soon, which is not a bad thing for some people.

I have used both, hero and bigworld and if i were to do a commercial game, my best choice would of been BigWorld engine, they wont go anywhere any time soon, now that it has been bought by the multi million gaming studio wargaming,net with world hit world of tanks game and the others released later by same company.

meaning Bigworld is a proven real life mmo engine tech which i guess noone can not argue with that when i comes to handling real massive ammount of players for a real mmo game of any sort.

so I guess ill can say i would trust BigWorld engine very much because of the proven technology and multi million company behind it, i dont think it will go anywhere anytime soon.

12

BigWorld is proven to be a reliable engine when it comes to smaller Lobby based MMOs Like as you mentioned World of Tanks and it’s spin offs. However it has yet to produce any real Massive MMORPG that wasn’t lobby based. Also their Indie License or at least their cheapest Indie License came with no MySQL Database support. The database was a Flat file system which I guess would be good enough for development but I doubt it would stand up well post release.

13

Anyone making posts looking for guidance on how to go about making a MMO should likely steer away from BigWorld. I really appreciate their candidness and being upfront about costs. They clearly inform people that their licensing is going to costs hundreds of thousands of dollars. If you have that kind of money, you likely don’t need to be asking these kinds of questions. :wink:

14

What did BigWorld do away with their Indie License?

15

I guess so. (Wargaming Website for Applicants)

16

Actually, BW engine has more than 15 mmos titles released under its belt, most of them Chinese or Korean mmo games all released and commercial games outhere, not just lobby games as you mentioned, real mmorpg games making profits even as we speak.

clarification on the indie license, BW engine release BW indie 2.1 on Nov 2012 which can handle now support for MySql database, so even with indie license BW is quite suitable to handle any sort of mmo game and can handle millions of concurrent users at a time.
guys, I am not advertising here, I do know for fact all this since I used to work on a commercial title using BW engine a few years ago. I have use and tried all sort of engine, but so far there is nothing outhere up to date that can compare to BW engine when it comes to networking capabilities as BW engine backend server.

I want to make a clarification, BW server is the best you can get outhere for networked games, it can handle whatever you can throw at it, easy to use, server deployment is easy as it can get, if your game all the sudden gets flooded with users overnight, all you have to do is install server in other hardware node configure settings and fire it up, the server will auto load balance itself all across all the hardware nodes without any human intervention, get more ccu again? deploy other server do the same, its easy as cake, you can do this over and over as your user base grows larger, the server itself will do the dirty job for you.

the only drawback is the BW client, i am saying drawback because compared to the beast of BW backend server, the client could be better, and better in a sense that the graphics are not that high tech, it can only handle DX9 altho BW 2.1 had great upgrade on graphics wise, but dont compare to today’s DX11 capabilities.
I mean is not bad or ugly at all, what you see in World of Warcraft engine graphics, BW can also compare aswell and even a little bit better imo.

anyways, BW client side is not bad at all, specially for an mmo game,besides you dont see a high end graphics game in an mmorpg game nowadays, and the reason is obvious since their focus is massive worlds and concurrent users with a decent graphics for a fast and lag free enjoyable experience with millions of players online.

So, if you really want a true mmo engine and if you are happy with DX9 rendering and if you are comfortable with python game logic scripting BigWorld is what you looking for, I know this for fact too because I do own a license of BW engine indie version 2.1 with mysql support db.
they dont advertise 2.1 since it was a free upgrade from BW 1.9 indie license holders.

cheers

17

They don’t seem to offer Indie at all anymore…period.

18

you right,
they dont support indie licenses anymore, only commercial ones, well at least existing indie licensees can still enjoy it.
glad i am on of them incase i want to do so.

too bad new ones wont be able to do so, unless they buy commercial :slight_smile:
I imagine cost, at least $300k and up fees for commercial licenses. ouch

and I understand why, before BW was owned by wargaming.net they had indie licenses to continue business, but now that multi million dollar wargaming.net acquired it, their main income is not the enigine itself but the titles they made with it. so they dont care to earn a few bucks from indies anymore, they are after the big profitable studios :frowning:

19

After a little bit of a discussion with others, I’ve come to a conclusion that a chunk-based networking system is a terrible idea. Not only does it require there to be multiple servers present, but it also has another drawback other than being expensive. It fails when you have large events. If this is a chunk-based system and you have a large amount of players in 1 location, you run in to issues. You’d have one servers doing all that work. And, let’s just go back to the fact that this chunk-based system still requires multiple servers… meaning multiple payments and management costs.

Instead, I revised my plan and would like to go over some things I changed with the game as a whole. It’s still an MMO. I mean, I plan for it to be able to support a massive amount of people online. So… yeah. Definitely still an MMO. One thing that’s very different though is that I plan for it easily hosted by players as well. This means a player should be able to have a single server and be fine. But, just one server isn’t enough. So, I thought of a way to expand on the processing power. I’ve gone through a lot of revisions. These revisions solved latency issues between hardware (because my first idea was the have each server communicate with each the master server and back, rather than directly to the client) and also over the network. Here is my final thought on how to handle networking on a large scale while also allowing room to support, to the best of my knowledge, an endless amount of additional servers:

Setup
There are three separate files for this project. The Master Server (Linux only), the Server (Linux only), and the Game (Win/Mac/Lin). The Server and the Game are technically the same files. The only difference though is if there is a password.pub file present the Game acts as a Server. Once the Master Server starts up, it runs the Server and also generates a highly encrypted password and password.pub file. The Master Server begins listening for connections from a Server and a Game (note: Game connection requires at least 1 Server authenticated). The Server reads a configuration file to find out the IP of the Master Server. The Server then sends the password in password.pub to the Master Server. The Master Server receives the password and validates it with the private key. If it is not valid then the connection is dropped. If it is valid then the Server is added to the “serverList” variable in the Master Server.

Connections
If a Game tries to connect to the Master Server, the Master Server will first verify that there is at least 1 Server in its “serverList”. If there is, then the Master Server will permit the client’s Game to connect. Once the client’s Game connects to the Master Server, the Master Server will assign the Game to a Server that is handling the least amount of players. The Game will be given this Server’s IP and will make all computational requests to it. The only time the Game would make a request to a different Server is when the Master Server feels that one Server is starting to have a far greater number of players than other Servers. At this point, the Master Server will start to tell the Games of clients that they should change their assigned Server.

Reallocation
When one Server is determined to have a far greater number of players than another server (%-wise) then that Server will be sent a “redirection” variable that contains an IP. Any Game that makes a request to a Server with its redirection variable set will be assigned a new Server. Additionally, the Server will get a “redirectionCount” variable that indicates how many players are to be redirected before discarding its redirection variable. It should be noted that when a Game is assigned a Server, the Master Server increments that Server’s “playerCount” in the “serverList”. It should also decrement the Game’s previous Server’s “playerCount”. This allows for the addition of Servers very easily while the Master Server is running and the game is online. This also allows for Servers that fail or disconnect to redirect Games elsewhere. If a connection times out, the Game will request a new Server from the Master Server. If there are no Servers present, the Master Server will tell all Games to please wait and that it will send the Server IP when there is one present.

I already started working on this with the Transport Layer API (TLAPI). I haven’t done anything fancy with the TLAPI yet. I can definitely scrap the code at anytime. I’m also all ears on suggestions for other solutions (languages/APIs in particular) to this network architecture in the most efficient way possible. I am also open to any suggestions on how the aforementioned idea can be even more efficient, but not drastically changed.

Also, when I said I could scrap the code, I meant it:

using UnityEngine.Networking;
//http://docs.unity3d.com/Manual/UNetUsingTransport.html
public class MasterServer : NetworkManager {

    // Configuration Channels
    int reliableChanId;
    int unreliableChanId;

    // Sockect Configurations
    int socketId;
    int socketPort = 7000;

    // Communication
    int connectionId;

    void Awake() {
        // Initializing the Transport Layer with no arguments (default settings)
        NetworkTransport.Init();

        // Configuration Channels
        ConnectionConfig config = new ConnectionConfig();
        reliableChanId = config.AddChannel(QosType.Reliable);
        unreliableChanId = config.AddChannel(QosType.Unreliable);

        // Max Connections
        int maxConnections = 10;
        HostTopology topology = new HostTopology(config, maxConnections);

        // Socket Configurations
        socketId = NetworkTransport.AddHost(topology, socketPort);
        print("Socket open on port " + socketPort + ": ID #" + socketId);

        // Communication
       
        NetworkTransport.Disconnect(socketId, connectionId, out error);
        NetworkTransport.Send(socketId, connectionId, reliableChanId, buffer, bufferLength, out error);

    }

    void Connect() {
        byte error;

        connectionId = NetworkTransport.Connect(socketId, "localhost", socketPort, 0, out error);
    }
}
20

A Chunk based Server as you describe it is the only way to go for true MMOs. If you want players to be able to host the server as well then it can’t be a true MMO with thousands of players in a single world. For a true MMO in Unity the best way to go is similar to how uPikko from MuchDifferent handles load balancing. Be able to dynamically load balance the servers as the user base changes. Don’t have set Server regions. Use a Quadtree and once the player limit exceeds a giving thresh-hold sub-divide the QuadTree until there are no regions that has more players then a giving number. Then have a way to offload the sub-divided regions to other server instances as needed and automatically reconnect the client to the needed Server instance.

And yes any true MMO is going to require a Cluster of Servers not just a single server which is going to cost money.