Developing and Programming an Efficient Network Architecture

Unity Engine
1

TLDR; Looking for more efficient ideas on my Network Architecture, I’d like to keep the part that is not a WIP for my Network Architecture if possible, and there are code questions down below in bullet point form.

Current Working Code [GitHub - License] (small revisions and questions will go in posts below)

I’m looking for some further insight on the ideas I came up with for handling an expandable user-base at an affordable cost.

I made a post long ago where I asked a few questions concerning the performance of Unity in terms of MMO games and how Unity handles multiple cores. Eventually, the thread started getting a bit off-topic, since my questions were already answered. I’m pretty set with sticking with Unity and have begun using the Transport Layer API to start working on a prototype to match my ideas.

The current network architecture can be found in the README.md for the GitHub repository. I’m more than happy to accept full or partial revisions of the parts labeled under WIP. But I am obstinate when it comes to changes to the rest of the ideas.

Current Code Questions:

  • Send() does not work as soon as Connect() is finished. How should I go about making sure they are connected before Send() goes through? My initial thought was to add a boolean indicating whether or not they are connected. The boolean would be set in Listen(). A question still unanswered though is, how could I make sure the message waits until they are connected? My thought on this part was to add it to a list that serves as a buffer. Then send them all out when ready. This would require a while loop though.
  • What would be an efficient way to relay client updates to other clients on different servers?
  • What is a good way to go about spawning nearby clients?

Solved Code Questions:

  • As an example, say there was a Hello() method. This Hello() method takes a string parameter and prints it to the console of the Master Server. How should I go about having the Send() method on the Server tell the Master Server to run Hello() with the string “World”? The key question here is, how can I minimize the amount of data being sent?
  • There’s a WhoAmI() method and it takes a string parameter. It also returns a string. The return is simply the whatever the parameter is. How should I go about receiving this data? Once again, the key question here is efficiency.
  • I heard that using BinaryFormatter isn’t the fastest way available to send data. If this is true, what would be a better method to take? The serialization code is an example from here as well.
  • Send() does not work as soon as Connect() is finished. How should I go about making sure they are connected before Send() goes through? My initial thought was to add a boolean indicating whether or not they are connected. The boolean would be set in Listen(). A question still unanswered though is, how could I make sure the message waits until they are connected? My thought on this part was to add it to a list that serves as a buffer. Then send them all out when ready. This would require a while loop though.
2

It occurs to me, that your MasterServer and Server code are similar enough, that you should probably incorporate a little DRY principle. Create an abstract base class that encompasses the Listen as a virtual member, have your abstract inherit from Monobehavior, then add a few flags for connects to master and port / ip where necessary rather than rewriting all that code.

Even better, if each server were basically a node / hive member and operated exactly the same, you may have an easier time. The only difference on each node being the port combination they are connected on. You then can control which one your master is by public ip / firewall rules rather than trying to maintain it as a predefined object in code.

3

They are similar for now just for testing purposes (edit: they are two different projects as well - check github). This is my first TLAPI script so I tried to get a working script to grasp how it would would. I suspect that the two servers will look totally different a week from now.

The Master Server will listen for connections (obciously). But, in addition to that it will listen for what type of a connection it is. Is it a Server or is it a Game Client? If it is a Server, and the proper pub key was sent, then add the Server to the “serverList” and set their value to 0 (for 0 players). If it is not, destroy the connection. If it is a Client, and the serverList has a count of at least 1 element, allow them to join. If there are no servers, make them wait until one is available. When one becomes available, the Master Server will pick a Server from the serverList with the lowest player count and assign the player there. That element will get an incremented player value.

The Server will listen for only Master Server instructions and player calculations / relay requests. Some instructions the Master Server might send are reallocations, shutdown/reboot signals, and possibly player data. Some requests a player may send to the Server are movement authentication, inventory requests, or other jargen. Nearby players may also request relay information to get nearby updates.

Client information isn’t even solid enough to talk about. While I have a detailed outline on this game’s features (3 pages of an outline), I feel like if I start the client now I’d waste time. Primarily because the servers code is not figured out yet.

My issue? Well, first I want to know if there is a faster method to go about Serialization other than BinaryFormatter. Secondly, I want to know the most efficient way to invoke methods over the network. I can easily accomplish this on my own. However, I’d have no knowledge with doing such a thing so it would be sloppy and inefficient for the long-run. Thirdly, I want to know if it’s even worth securing the network connection only when sending the pub key to the Master Server. Lastly, what is the best way to invoke a method and get a returned data back? My thought would be to invoke the method and Send() whatever the method returned. But that doesn’t seem good enough.

4

Without fully understanding your overall goals, I think I see where you’re going, but I’d like to proceed without pretending to fully understand it. :slight_smile:

I think your master server is actually more of a proxy server. So this is a well defined network pattern, and works well. It might even be considered a reverse proxy.

Do sub servers need to talk to each other directly, or will everything marshal through the master server when sub servers talk? Do you envision the sub servers being things like A Login Server, different region servers, a physics simulation server, a chat server…etc? Or am I getting the wrong impression?

Binary serialization is the most CPU intensive, but it’s actually really fast, depending on the size of the serialized objects. You’re best bet is to serialize as little as possible. Send operation codes as bytes that both the client and server understand, and you can actually accomplish tasks through that partnership without having to send it over the wire. Imagine the way special forces combat troops talk on radio silence missions. Holding up a fist, two finders to the eyes, open palm and point…these are “data contracts” with implied meaning. You can do the same thing server to server, or server to client.

Byte 0x00 received, means do this thing.
Byte 0x02 received means do this other thing.

That also works for your question about how to invoke remote operations. Send a byte code that’s basically an instruction, include a subcode if needed to add further granularity. If you truly need to invoke remote operations, investigate RPC or .NET remoting. You can also look into serializing Actions or Func<T,V> and invoke operations and send them across the wire, but be VERY carefull this is only server to server, you cant ever trust the client to do that.

5

I like you enthusiasm!

There are certain times when the sub server talks to the master server. For example, the sub server needs to talk to the master server upon the initial connection. This way they get registered. The way they talk in this instance is they simply send their public key and if it matches with the server’s private key it is validated. Otherwise, kick them off. Another instance is when players are assigned a specific server. This server may be given data about this player (I say may because I’m still wondering if I can get away with not having to store data on the servers – I’ll think on that when things start forming).

Sub servers talking to another sub server is something I want to avoid as a whole. It would definitely reduce latency if I could keep connections on a direct level. The relay section of my network architecture shows a player requesting authentication from the server. The player should also send information about the players nearby. I know that it’s written there that the player should tell the nearby players to send a unique ID to the server to request these updates. It has been on my mind since I thought of this plan: “How the hell do you tell the player to do anything if it’s not on your server, it doesn’t take client-client communication, and you don’t have their IP?” So, lately I’ve been thinking about storing all players basic information on a list on each server. All players. And I’ve also been thinking, “Wouldn’t this bring up a large strain on the servers?” And the thing is… I don’t know… The largest list I’ve ever seen was something like a 100x100x100 multidimensional array of bytes that I created out of testing for a voxel terrain system I was making a year ago (I moved away from storing it all in a giant list at once – It’s a lot more efficient now so don’t worry about that :stuck_out_tongue: ).

You can think of the Master Server as an employer. The Sub Server has a job, and so does the Game Client. Neither of these two have jobs until the Master Server hires them. They don’t get hired unless they meet the criteria. The Game Client’s job is to do the dirty work while the Sub Server’s job is to supervise that dirty work to make sure it’s not too dirty (cheaters).

I feel like the only job the sub servers should have is to act as a means to prevent the client from accessing and altering game-breaking data – movements, inventory, currency, and other players’ personal information. Essentially, the server will handle physics, database calls (which a copy is saved on the server that the player is located on for quicker access and less latency), and some other things.

The chat server is separate from all of this. When the master server starts up, it will start up the chat server as well. A single server box will run the chat server, the master server, and a server (optional). It’s optional through a configuration file where a parameter AutoStartServer is set to 1. If it’s set to 0, the Master Server will not start a server locally. Instead, it will sit in silence waiting for an external server to request access. Then it will send all the waiting players to that server.

Thanks for this. I was actually going to do something along the lines of decimal numbers and not hexadecimal. It’s also good to know the pros and cons for the binary serialization. I’ll try my best to keep it to a minimum. I’ll also try and produce a working code to publish here for critiquing pretty soon. I just got home and I’m heading to bed right after I finish writing this post. Haha.

While I was away from the computer, I was reading up on some things. I saw some suggestions to use WCF or RPC. I’d prefer to use neither honestly. I’d like to try and stick with TLAPI as much as possible. I say this because I haven’t made many networking scripts. I don’t know how WCF works. I know a little bit about RPC. I’m here for a reason and that’s to make sure I don’t do things inefficiently. I think using any one of these two would possibly open up a whole new door for trouble. It’s just another thing I have to learn about at a short notice and keep track of too. However, it’s just an assumption. Since I pretty much know nothing when it comes to those two, I can’t give a clear judgment for myself in terms of how much of an impact they’d be in terms of development.

I think I’ll stick with just sending byte codes over the network. I’ll be sure to post my progress once I’ve made some! Hopefully my reply also sparks some further ideas on the matter.

6

Don’t use WCF. I’ve used it extensively in .NET connected apps and it’s a HUGE HUGE pain. It’s got massive overhead, it’s prone to configuration failure or mismatch, and it’s complicated to maintain. The connections themselves are rock solid once made, but you aren’t going to WCF your game backbone…it’s not even an option.

RPC calls are a very valid way of handling server to client communication. It cant go the otherway, and I think you’ll naturally find you’re doing some of that as you get into the actual programing of your game.

So, as far as hex notation, basically that’s just a writing preference. Look at it this way

  [Flags]
    public enum OperationCode : byte
    {
        ListInventory = 0x00,
        MovePlayer = 0x01,
        MoveNPC = 0x02,
        OpenDoor = 0x04,
        SomethingElse = 0x08,
        SomethingElseAswell = 0x10
    }

    [Flags]
    public enum OperationCode : byte
    {
        ListInventory = 0,
        MovePlayer = 1,
        MoveNPC = 2,
        OpenDoor = 4,
        SomethingElse = 8,
        SomethingElseAswell = 16
    }

    [Flags]
    public enum OperationCode : byte
    {
        ListInventory = 0,
        MovePlayer = 1,
        MoveNPC = 1 << 1,
        OpenDoor = 1 << 2,
        SomethingElse = 1 << 3,
        SomethingElseAswell = 1 << 4
    }

Essentially these are all the same. If you resolve one of the enum members to it’s byte reference in decimal it’ll be like this

00000000
00000010
00000100
00001000
00010000
00100000
01000000
10000000

The point is, you create an ordinal reference. Where any single item is unique, but any combination of items is also a unite byte patter. So you can combine them together, and then separate them using binary XOR or XAND

7

I finally got the time to sit down and have a crack at this. The diffs are presented here on github. Hopefully I did it right.

To save some time from reading the entire code, essentially what I did was I disabled serialization with a variable. If that variable is set to true, it will serialize the data.

Server.cs

    public void Send(string msg, int connection, bool serialize = false) {
        byte error;

        // Serialization
        byte[] buffer = new byte[1024];
        int bufferSize = 1024;

        if(serialize) {
            Stream stream = new MemoryStream(buffer);
            BinaryFormatter formatter = new BinaryFormatter();
            formatter.Serialize(stream, msg);
        }

        ...
    }

I added some essential methods that I’d more than likely be using later. I replaced print() with debug() so I can turn off all debug code later without having to actually remove the code. At some point, I will actually remove them though.

    #region Essentials
    bool debugging = true;
    public void debug(string msg) {
        if(debugging)
            print(msg);
    }

    public byte[] StringToByteArray(string str, Encoding encoding) {
        return encoding.GetBytes(str);
    }

    public string ByteArrayToString(byte[] bytes, Encoding encoding) {
        return encoding.GetString(bytes);
    }
    #endregion

The buffer is set without serialization like so:
Server.cs

buffer = StringToByteArray(msg, Encoding.UTF8);

I set up a way to tell the master server to perform a specific function with byte codes.

Server.cs

    #region Actions
    #region Action Variables
    [Flags]
    public enum Actions : byte {
        Auth = 0x00,
        Debug = 0x01 // test -
    }

    /// <summary>
    /// Send the requested auth password to the master server.
    /// </summary>
    /// <param name="key">The public key.</param>
    void Authenticate(string key) {
        Send(Actions.Auth + key, masterServerId);
    }
    #endregion
    #endregion

MasterServer.cs

    #region Action Variables
    [Flags]
    public enum Actions : byte {
        Auth = 0x00,
        Debug = 0x01 // test
    }

    void PerformAction(byte code, string msg) {
        // Unsure if this cast will work. -- Uncommitted comment: It actually works but still unsure if it's proper here.
        switch ((Actions)code) {
            case Actions.Debug:
                debug("Debugging socket: " + msg);
                break;

        }
    }
    #endregion

There’s still some work to be done, of course. But I just want to know if I’m on the right track. With the conversions, substrings, and including text as a parameter and separating them by spaces. That way I can start cleaning things up and removing any redundancies. I was also thinking of wrapping parameters in escaped quotation marks so strings with spaces in them can be told apart and then I can split them by ‘" "’ and add them to a string[ ] param variable.

Edit: For things like inventory I’d Send(Actions.PickUp | Actions.Item1) - I haven’t tested it yet though. Just a thought.

8

TLDR; Passwords aren’t matching up. Scroll down to where “Problem:” is and read up to the next code example about TrimEnd().

So, I started working on the authentication for the Master Server and Server. One thing I changed was PerformAction()'s parameters. It now has a new parameter that is the connection id that sent this command. My reasoning behind this was so that if a incorrect keyphrase (password) were to be entered it would disconnect the remote server.

MasterServer.cs

void PerformAction(byte code, string msg, int id) {
    // Unsure if this cast will work.
    switch ((Actions)code) {
        case Actions.Debug:
            debug("Debugging socket: " + msg);
            break;

        case Actions.Auth:
            Authenticate(msg, id);
            break;
    }
}

Problem: I added an Authenticate() method to the Master Server. Unfortunately I am getting a comparison error.

MasterServer.cs

void Authenticate(string msg, int id) {
    if(password == msg) {
        debug("Server has been authenticated.");
    } else {
        debug("Server has failed authentication. (10 attempts left before added to blacklist)"); // blacklist can be edited easily via text file so the user won't have to go through a lot of trouble to remove a mistaken blacklist

        byte error;
        NetworkTransport.Disconnect(socketId, id, out error);

        if ((NetworkError)error != NetworkError.Ok) {
            debug("Failed to disconnect remote connection because:" + (NetworkError)error);
        }
    }
}

My first thought was that the password could possibly have a carriage return or newline at the end of it. So, I made a minor change to the way PerformAction() was called. I added TrimEnd() to remove the annoying bits. Of course, I will need to manage this in the future in the event that I would actually want to keep those annoying bits.

MasterServer.cs

PerformAction(code, message.Substring(message.IndexOf(' ') + 1).TrimEnd(new char[]{'\r', '\n'}), recConnectionId);

Unfortunately, that did not work. But, moving on to how the passwords are created, I couldn’t really find any simplistic password generations. There was System.Web.Security.Membership.GeneratePassword(), which cannot be used. There was also System.Security.Cryptography, which seemed like a lot for what I actually needed. I cba to check if those are the right namespaces, so forgive me if I remembered them incorrectly. So, I made a random string generation method, CreatePassword(), that included a set of characters of my choosing. There’s also LoadPassword() which will create a new password and save it if one is not present.

MasterServer.cs

bool LoadPassword() {
    string rawPath = Application.dataPath + "/auth/";
    string path = Application.dataPath + "/auth/keyphrase.passwd";

    if(!Directory.Exists(rawPath)) {
        Directory.CreateDirectory(rawPath);
    }
    else {
        if(File.Exists(path)) {
            password = File.ReadAllText(path);
            return true;
        }
    }
       
    password = CreatePassword(256);
    File.WriteAllText(path, password);
    return false;
}

string CreatePassword(int length) {
    const string valid = "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ1234567890!@#$%^&*()_-+=[{}]|:;<>,.?";
    StringBuilder pass = new StringBuilder();

    while (0 < length--) {
        pass.Append(valid[UnityEngine.Random.Range(0, valid.Length)]);
    }

    return pass.ToString();
}

Server.cs

bool LoadPassword() {
    string rawPath = Application.dataPath + "/auth/";
    string path = Application.dataPath + "/auth/keyphrase.passwd";

    if(!Directory.Exists(rawPath)) {
        Directory.CreateDirectory(rawPath);
    }
    else {
        if(File.Exists(path)) {
            password = File.ReadAllText(path);
            return true;
        }
    }

    debug("Error: Missing keyphrase.passwd file.");
    return false;
}

The only difference is that the Server does not have CreatePassword() and if LoadPassword() returns false it closes the server, or pauses it in the editor.

Server.cs

void Start() {
    if (!LoadPassword()) {
        #if UNITY_EDITOR
            Debug.Break();
        #else
            Application.Quit();
        #endif
        return;
    }

    . . .
}
9

I finally found some time to look over my code and pinpoint a few flaws. After some debugging with the code below, I realized something interesting:

debug("[" + password.Length + "/" + msg.Length + "], [" + password.Substring(0,1) + "/" + msg.Substring(0,1) + "], [" + password.Substring(password.Length - 1, 1) + "/" + msg.Substring(msg.Length - 1, 1) + "]");

The length of the password was 256 and the length of the message was 1022 (or 1024 minus the \n and \r)! So, that made me think, what if I did my Listen() wrong? And it turns out I did. I needed to use my dataSize parameter to grab only the important bits of the message that was sent. So, I did the following:

string message = ByteArrayToString(recBuffer, Encoding.UTF8).Substring(0, dataSize);

I knew that this was correct. Unfortunately, I was missing something still. So I debugged dataSize and came to realize that the size was far bigger than the length of the password as well. Which was wrong. So, I headed over to Server.cs and changed the code to send the proper bufferSize:

int bufferSize = msg.Length;

With that being done, everything now works as intended. The password is sent over the network for authentication and the Master Server reads it just fine!