NetworkTransport - maximum hosts cannot exceed {16}

Multiplayer & Networking
1

Hello All

I am creating a Client (Windows) and Dedicated Server (Linux) using LLAPI - NetworkTransport class for a MMO Real Time Space Exploration / shoot em up game.

The Dedicated Server is to handle +1000 connections.

I was trying to setup a set of test tools that do different test like see if the server can handle X Connections all sending random messages.

However when I set the amount of clients connections to start greater than 16 I get the following Error in the debug log window in the test app:

maximum hosts cannot exceed {16}
UnityEngine.Networking.NetworkTransport:AddHost(HostTopology, Int32)

I can set them to create 16 client connections a just run 63 copies of the app, which works, but that is too tiresome and also takes too much resources on the machine running the test tools.

So I was just wondering if there was a way to get past this restriction via a coded route.

*** update ***
I might re-think in doing the load test app a different way like the following:
1: Make bash script to run X amount of instances of the load-test app.
2: The load-test app will be a headless Linux app which only opens up just the one connection.
3: And once connected spits out random messages real fast.
4: After X Time terminate the instance.
I think that’s a more better route to take than what I just did and failed LOL.

Thanks

Paul

2

The dedicated server should have no hosts, it just listens on a address/port

           ConnectionConfig cconfig = new ConnectionConfig();
           NetChannel.System               = cconfig.AddChannel(QosType.ReliableSequenced);
           NetChannel.UserReliable       = cconfig.AddChannel(QosType.Reliable);
           NetChannel.UserUnreliable       = cconfig.AddChannel(QosType.Unreliable);

           // create the host topology
           HostTopology hconfig = new HostTopology(cconfig, maxccu);

           // start the simple server
           _unetserver = new NetworkServerSimple();
           _unetserver.Listen(port, hconfig);

That’s a snippet of my server code

3

Thanks for the reply.

I am using NetworkTransport LLAPI like I already said in the subject and not NetworkServerSimple.

Also I wasn’t having any issues with my server code, that code is fine.

I was having issues trying to write my Test Tools that puts the server under load by connecting as many client connections at possible until it breaks.

Also the Server Side does need to have a host, that is the host the clients connect to.

I am guessing you are getting the different types of servers unity has mixed up.

Also with your code, if you are not using WebSockets, then in the Listen function it calls AddHost(HostTopology, Port, ipAddress) so your Server does have a Host.

This is my current Server Code at its basic form that I am using for a test which works fine:

using UnityEngine;
using UnityEngine.Networking;

public class Server : MonoBehaviour
{
    private int authChannel;
    private int chatChannel;
    private int serverNewsChannel;

    private int hostId;

    private int maxConnections = 1000;
    private int serverPort = 27030;

    void Start()
    {
        GlobalConfig gConfig = new GlobalConfig();

        MaxPacketSize = gConfig.MaxPacketSize;

        NetworkTransport.Init(gConfig);
        ConnectionConfig config = new ConnectionConfig();

        // Setup channels.
        authChannel = config.AddChannel(QosType.ReliableSequenced);
        chatChannel = config.AddChannel(QosType.Reliable);
        serverNewsChannel = config.AddChannel(QosType.Reliable);

        HostTopology topology = new HostTopology(config, maxConnections);

        hostId = NetworkTransport.AddHost(topology, serverPort);
    }

    void Update ()
    {
        int outHostId;
        int outConnectionId;
        int outChannelId;
        byte[] buffer = new byte[1024];
        int bufferSize = 1024;
        int receiveSize;
        byte error;

        NetworkEventType evnt = NetworkTransport.Receive(out outHostId, out outConnectionId, out outChannelId, buffer, bufferSize, out receiveSize, out error);
        switch (evnt)
        {
            case NetworkEventType.Nothing:
                break;

            case NetworkEventType.ConnectEvent:
                Debug.LogFormat("ConnectEvent - Receive({0}, {1}, {2}, {3}, {4}, {5}, {6})", outHostId, outConnectionId, outChannelId, buffer.ToString(), bufferSize, receiveSize, error);
                break;

            case NetworkEventType.DisconnectEvent:
                Debug.LogFormat("DisconnectEvent - Receive({0}, {1}, {2}, {3}, {4}, {5}, {6})", outHostId, outConnectionId, outChannelId, buffer.ToString(), bufferSize, receiveSize, error);
                break;

            case NetworkEventType.DataEvent:
                break;

            case NetworkEventType.BroadcastEvent:
                break;
            default:
                throw new System.Exception(string.Format("Unknown Network Event Type: {0}", evnt));
        }
    }
}

Anyhow I will try my second idea tomorrow.

Paul

4

Definitely keep us posted on this. This did crop up some time ago here: maximum hosts cannot exceed {16} - Unity Engine - Unity Discussions but unity officials claim it has been resolved… I haven’t yet tested with more than 10 peers in my current project so it is concerning if this is still an issue!

Also, don’t forget (call Receive until there is nothing):

    void Update() {
        bool stop = false;
        while (true) {
            //break when the "Nothing" event happens, and sets stop to true
            if (stop) break;

            int outHostId;
            int outConnectionId;
            int outChannelId;
            byte[] buffer = new byte[1024];
            int bufferSize = 1024;
            int receiveSize;
            byte error;

            NetworkEventType evnt = NetworkTransport.Receive(out outHostId, out outConnectionId, out outChannelId, buffer, bufferSize, out receiveSize, out error);
            switch (evnt) {
                case NetworkEventType.Nothing:
                    //set stop to true here
                    stop = true;
                    break;

                case NetworkEventType.ConnectEvent:
                    Debug.LogFormat("ConnectEvent - Receive({0}, {1}, {2}, {3}, {4}, {5}, {6})", outHostId, outConnectionId, outChannelId, buffer.ToString(), bufferSize, receiveSize, error);
                    break;

                case NetworkEventType.DisconnectEvent:
                    Debug.LogFormat("DisconnectEvent - Receive({0}, {1}, {2}, {3}, {4}, {5}, {6})", outHostId, outConnectionId, outChannelId, buffer.ToString(), bufferSize, receiveSize, error);
                    break;

                case NetworkEventType.DataEvent:
                    break;

                case NetworkEventType.BroadcastEvent:
                    break;
                default:
                    throw new System.Exception(string.Format("Unknown Network Event Type: {0}", evnt));
            }
        }
    }

I’m currently having real issues with this using the NetworkClient/NetworkServerSimple stuff, god I miss using the LLAPI!

5

Hey :slight_smile:

Wasn’t that bug to do with not releasing closed connections?

My issue is that I am trying to create too many hosts on the client side of things not the server side.

I will try and connect and then disconnect 17 times to see if its the same bug, it might just be that same bug, but until I test it I won’t know.

But yeah, I will keep you posted on this.

I was able to create over a 100 connections by having the client create 15 hosts and have them all connect to the server, I then run 7 instances of the client giving me 105 connections on the server, so the server side is fine.

I did take a look at the NetworkServerSimple class, it does look nice and does have a lot of features, but I liked the LLAPI NetworkTrasport, I really do love jumping in the deep end LOL

As for the looping until all received messages, that might be an issue with thousands of connections all sending loads of packets where it would be stuck doing network processing until it receives the Nothing Event.

I could pass each received message into a Message Queue and have another Thread handle those messages.

There are loads of ways I could do all this, so I will get to try all the different ways once I get Test Tools done.

Well time for a late breakfast, then the tackling the 16 host limit issue, then the Test Tool.
Oh what fun.

Paul

6

Yep, just thought I would do the 16 host limit test before I go and eat and I could see right from the start it was that bug due to the hostId wasn’t resetting back to 1 when I disconnected.

This was my fault due to when you disconnect a client, you need to call NetworkTransport.RemoveHost(hostId) on the client side to “Closes the opened socket, and closes all connections belonging to that socket.” that resolved the connecting and disconnecting more that 16 times in a row.

That won’t fix the issue the way I was trying to do it first where I have the Test Tool try and create 1000 hosts that would connect to my server and send random data to test how it handles with all those connections and random data.

That will still have the issue, I think Unity Devs “might” of put that in to stop people abusing it and in a way a client wouldn’t in most cases have more than 16 connections on it, a server would but not a client.

But at least I now know where the limitation is.

Right now food, laters.

Paul

1 Like
7

Ahh okay, I actually thought you were talking about the server side, panic over!

Yeah, it’s probably worth mentioning that with the client-server model (which albeit you’re not forced to use in the LLAPI or the HLAPI) does expect the client side to handle only one outgoing connection, while the server side should handle many incoming. What’s your reasoning for having multiple connections on the client-side?

well… the thing is that if you don’t consume the incoming messages quick enough, they will pile up and will often cause terrible lag (and alot of the time disconnect you), so it’s actually a good idea to consume them as quickly as you can (using your while loop). If you find that you just get far too many messages going through (causing the while loop to slow down your update loop too much) then you are absolutely sending far too many messages over the network.

You are currently talking about fairly ambitious numbers… so you’ve got your work cut out for you to keep your message counts low in numbers and in size, but yeah you definitely still want to avoid them piling up.

A quick tip here: combining small messages into one larger message will be a good strategy for you, considering the numbers you are talking about. Each message that you send has it’s own overhead (8 bytes for UDP, plus the few bytes that unity sends internally along with each message), and takes up a certain amount of cpu time to both send and receive. There’s also garbage collection issues with many small messages versus less larger messages. So all-in-all, I’d suggest to think about ways of combining smaller messages into larger ones.

Yeah I think we briefly spoke about that before. You could do that (I think unity already handle the sending and receiving of messages on separated threads), in order to have more control over the natural piling up of messages. I would say though that if you end up needing to do this, then you’re still definitely sending too many messages.

8

I was using Unity (Client Side) as a Stress Tool where it would open loads of connections to see how the server handles X connections.
That was why I needed the client to open that man connections.
But I have decided on creating a Linux Client in Headless mode that opens only the one connection (a single host) and sends random data.
Now on Linux I run multiple instances of that client X times resulting in what I wanted but only using one connection per client.

Yeah, that’s why I want to do loads of tests first to see what fits, so I will try both ways to see how they both perform.

Well I know the server hardware along with connection to the server is capable of handling 5K CCU so its all down to the coding :stuck_out_tongue:
I just picked 1K as a minimum goal to reach, and if I have issues code wise supporting that amount then I will split it up onto multi-servers, a bit like Zone Servers.

Well I did read some place on the Unity Site that if the client sends too often and also small packets it combines them before sending and when it receives the combined packet it automatically separates them up.
The way I see it, if the Test Tools are sending over the top amount of messages and the server starts getting backlogged, I can just scale back the test tools until its stable, then at that point I know what it can handle and then aim for half to 3 quarters of that as a maximum and I should be safe.

Well either too many connections each sending a reasonable amount of messages resulting is loads of messages, or too many messages like you said.

Normally in general in MMO type games the server(s) receive the packets and puts each individual message into a Threaded Read Message Queue.
And in a separate thread it goes through each message in the read queue one at a time and processing them.
And any outgoing messages get created and added to a Threaded Write Message Queue which goes through each message and sends them out one at a time.
Its done that way so that the main part of the code especially the receiving part doesn’t get stuck in a blocking state.

But I will see how it goes.

In general the client won’t be sending that many messages, this was only to be a stress test so I get a rough idea what it can really handle.

I know with real-time movement I can cheat by cutting down the amount of messages and using LERP to interpolate between the two positions.

Thanks

Paul

9

Well I managed to test up to 144 odd connection without sending any messages (apart from the internal ping messages) before it started dropping connections due to timeouts and then reconnecting them and then being fine up to 170 CCU.

The server was only taking up 13% of the CPU, my guesses are that my Desktop PC couldn’t run any more than 170 clients LOL, my PC was at 100% CPU load.

I even tried to extend the timeout to 5 and 10 seconds, but that still didn’t solve it :frowning:

Sadly if the issue is in the server code then 144 to 170 CCU just isn’t viable, maybe it was me just trying to run too many clients on my Desktop PC and it was in fact the clients that couldn’t keep up and timed out due to the server was only at 13% CPU load when connections started to time out.

I also managed to get the server to receive up to around 60K to 70K messages per second with around 40 CCU any more didn’t result in it increasing, also the server was taking 13% CPU usage.

Any other ideas?

Paul

10

Yep, it was client side related.

I run 9 client apps each doing 16 connections (totalling 144 connections) on my PC and got somebody here to also run 9 client apps each doing 16 connections (totalling 144 connections) totalling 288 clients and the server showed 288 connected and didn’t drop a single connection.

It seems that a Unity app for Windows takes 33% (opening 16 connections) on my CPU.
It also seems that same app when run in headless mode and no gfx drops it down to 20% per app doing 16 connections.

Also to note, when the server had 288 CCU it was taking 45% CPU usage (on a single core) I told the server to use only 1 of the 8 cores.
So if my maths does me proud that’s about 5.120K CCU doing only connections and internal PING, so my 1K CCU might be ok if I am smart coding the game :slight_smile:

So overall I am sort of happy so far.

Paul

11

These are some pretty promising results, and give me some additional confidence for my own project, so thanks for that!

Out of interest, were you sending messages over the network to all of your clients? If so, how frequently were the messages being sent, and what were the size of those messages?

My testing so far has shown that the difference in cpu between 1 and 10 clients (all receiving around 30 bytes per second on average) is almost non-existent. I found actually that the slight increase in cpu usage was actually caused by the additional garbage collection required in my current custom serialization of the byte arrays being sent (which I am working on, so the chances are i’ll notice pretty much no change between 1 and 10 clients once I reduce the amount of garbage being created).

With that being said, I definitely can’t get up to the numbers you are looking for. My target is 200, with an additional overflow of 100 (at a slight cost in reduced performance during peak times). This is because of the actual game logic itself though, where each client is in control of a single object, and there are other objects spawned and managed by the server eating up cpu. Granted, the game that I am working on is designed to have many server instances as separated game worlds, so it has been crucially important for me to ensure that an instance of my server-side can run on a very cheap server (currently, i’m using google cloud micro servers at 1 shared cpu, and 1.7gm ram and it’s working well on that), which is unlikely to be what you’ll be using, but you may still want to factor in some cpu time for your game logic to get a better idea of things.

Thanks again for the results, very useful!

12

No worries, while its still promising, I am not exactly happy with the CPU Usage on the server, it should of been lower for what its actually doing.

This is mostly down to when using as a dedicated server in headless mode Unity is bloatware, there is loads of stuff running that doesn’t need to be running.

Also not a fan of everything running in the same Thread, granted some of the stuff that is being done internally might me threaded like the message queues, but its trying to do everything in the update loop which is using the main thread, same with FixedUpdate is all using the main thread.

I did try to see if NetworkTransport could be run and handled in a separate thread, sadly it cannot :frowning:

I wasn’t sending anything, there was network traffic going on, but that was the behind the scenes ping packets that NetworkTransport does internally.

When I did send messages from 48 clients to see if I could get a rough idea on how may messages the Update loop could handle per second before it broke which was 60K to 70K messages per second.

And those messages was just “Hello World and an INT TimeStamp along with DateTime string” so 30 to 40 bytes.

Well I was only receiving on the server and not sending, but that’s not including what NetworkTransport does in the background.

Also the Channel QOSType could impact that as well depending on what type you choose.

Well I have only tested it on a VM Box, but the server it will be on is a 4 GHz i7 CPU and it has I think 32GB Ram, and its network connection is huge, I have a latency to it of around 10 to 18 ms which isn’t too bad.

I am also planning to separate the world server to loads of nodes which handle different parts of the world, like segment of space, when in primary locations, instances etc, which was why I was still aiming for around the 1K ball park.

At a later date if the game takes off, I plan to move the server side stuff to a more ideal cloud location.

Like I said, no worries.

Paul

13

Yeah, i’ve been in a sort of love-hate relationship with unity over this for many years now. In order to come to terms with it, you need to remember that unity’s overall philosophy is around the portability of it (some of the platforms it supports have limited multi threading support, hence why we suffer overall for it) and ease of use (as you probably already know, multi threading can add a whole heap of problems unless you know what you’re doing etc. Unity seem to focus on simplicity more than anything else). Over the years i’ve played with other frameworks like the unreal engine and the source engine and i’ve come to prefer the foundation that unity offers.

Overall though, you should be able to get round issues with worker threads trying to use unity-specific methods, by thinking cleverly about what you put on a seperate thread, and making sure that the end-result ends up back on the main thread to handle calls to any unity-specific methods. Co-routines, as annoying as they are to use, actually really help with blocking issues that heavy processes would normally cause, while avoiding multi threading issues so these can be a good alternative to multi-threading.

My server side also runs a headless linux build, and my master server (which is pretty much a router, doing almost nothing 90% of the time) sits at ~3% cpu. I got it down to this by tweaking some of the project settings such as disabling physics, rendering, audio etc. The main thing that I see though that helps keep cpu down is the target frame rate (accessed via Application.targetFrameRate). By default, this is -1, causing unity to use as much cpu as it can to run the frames as quickly as possible. I set this to 30 on my master server, which does force vertical sync (causing some cpu overhead to slow down frames when needed), and that was the main thing that kept the overall cpu down. My sub servers (where the main server-side game logic is) have the target frame rate set to 60, while my client-side has it set to -1.

Again, some really useful information to know (and rather impressive results!) so thanks for that.