Please HELP with hard decision after many test ans stats - AS or Proxy

Multiplayer & Networking
1

Some background before I start with the questions.

The Game
It is the RTS tactics battles between 2 players, probably in the future will be 3 or more.
Each one have maximum 5 vehicles with unique properties (cars, tanks, artillery, etc…).
Movement restricted to XZ. Battle length usually 3-5 minutes. Game runs on Linux, Windows, Mac, Tablets (redesigned and optimized for touch) and Web.
Playable version with AI or in network using proxy is already works and run fast and smooth.
Game very dynamic and mix between logic and action.
Game fully utilize physics engine, i.e.: Collision of vehicles of different weight, push forces from explosions, etc… Therefore as a solution deterministic model is not something possible to implement.
Maybe soon I will publish my multiplayer playable demo.

Network and Stats
The first prototype run fast and reliable using proxy.
RPC and State Sync (15/sec) for
Player A units sync go from Player A —> Proxy ----> Player B
Player B units sync go from Player B —> Proxy ----> Player A

The VPS server located in Chicago,
in multiple test between Canada, Winnipeg with ping 50 and Israel with 170, the game run smooth.
State Sync is 15/second.
Instead of using Interpolation/Extrapolation I simply RPC the commands move, fire, etc… and allow each player physics engine to do the work for all vehicles, but then fix positions, rotations, etc… using state sync. It work very nice this way. I measure exact positions of all units on both ends and compare to find problems, and there is no. each small change of physics floating points fixed by state sync.

Some Statistics of Sync State Data
1 Byte - BOOL - stream.Write(platform.move)
4 Byte - Float - stream.Write(_myTransform.position.x)
4 Byte - Float - stream.Write(_myTransform.position.z)
4 Byte - Float - stream.Write(_myTransform.rotation.eulerAngles.y)
4 Byte - Float - stream.Write(_turretTransform.rotation.eulerAngles.y)
4 Byte - INT - stream.Write(currentHP) some units regenerate all the time, therefore RPC not an option.

21 Bytes per vehicle, total 10 vehicles and 5 per player = 105 Byte.
105 byte sent to other player. and 105 received = 210 in total.

15 times per second =
TOTAL 3,150 Byte / sec 25,200 Bit / sec
IN 1,575 Byte / sec 12,600 Bit / sec
OUT 1,575 Byte / sec 12,600 Bit / sec

If game runs 10 min
600 sec x 3,150 Byte = 1,890,000 Byte (2 Mbyte per game which is Ok)
I am planning to record each game to allow watch and learn for users later.

Measuring UDP data on a Linux VPS server I got 25 for each direction, 50 in total.

The Questions:
1.
50K/s for one game x 200 battles = 10 MBit / sec all the time which will eventually be
more than 15 Tera /month for only 200 battles.
Is it Ok ? or it is a lot and I need to find other solutions ?

2.
I know that proxy model have few big disadvantages:

  1. Security - to build security on a proxy model is a mouse and cat games, as I need to build some behaviour prediction checks to find hackers(crackers), and it will go and go,
    not something I really want to deal with.
  2. Not super efficient in bandwidth.
  3. Game logic that run on client - works now but no super efficient model.

and as you quest all this bring us to Authoritative Server, which I started to implement.
It solve all this Proxy problems, but create a huge new once.

1. How much more efficient bandwidth will be,
according to my calculations from 50K (25 up, 25 down), I will have just 25 down.
but if I now need to transfer all 10 units and not 5 and 5 it might even be the same.

2. Yes now game logic on a server (super), but the biggest question is how many game server processes the server can handle. I did many checks using all kind of bat file scripts, that run game in all kind of CPU efficient variations (of course in batch mode).

On my MBA 13 (i5) that run Windows 8 Pro - with 100 game server processes - 90% CPU.
But on my desktop i7 - 2600K after 150+ processes I got MONO memory errors.
I tried everything. funny that even with 70 games on 1 CPU core from 8, it runs ok, but when other CPUs got loaded with game process after 150+ all crash.

The question is, how much UT game processes server can handle ?
is 150+ some kind of limit ?
Can Linux solve this problem ? My VPS to slow to check it.

3. Lets say I can run 100-200 game processes in a AS model. it is just 200-300 concurrent online users. For me it sounds that I will just be able to pay for a server $100/m that will run them. And to make a game that will just pay for servers that run it, not a good idea.
From your experience and I know it vary a lot, but for a regular game how much it is 200 CCU. Nothing - $10/m - or thousands a month, of course if monitorization system in the game is build and working.

I know I bring a lot of questions, and I need answers from experienced users, but after a few nights spent in writing and testing, I must finally make this hard decision: Authoritative Server or Proxy that run already in fully playable game prototype.

After 15 years working as Linux Administrator I tend to AS model, but really afraid that eventually I will work to finance the game due to the huge clustering model with Unity Server Processes running in Linux virtual frame buffer.

2

If it’s an RTS you can just remove the state sync part (at least reduce it)
It’s quite easy to do (depending on the game but usually it is) and you can save lots of bandwidth.
Tell us more about you game (in term of gamemecanics)
What justify to set a so high rate of state sync?

3

To try and offer some answers:

  1. In my experience 50K/s is too high for this type of game, although I think you’ve combined both up and down stream to get the 50K/s. I’m guessing you are using the built-in Unity networking which doesn’t support many datatypes to be sent, I would advise you to swap to one of the third party options as they are much better (uLink would be my recommendation for you but I’ll come onto that later on).

You can look at your datatypes to shave some space:

4 Byte - INT - stream.Write(currentHP) -
Quite often health is an integer between 1-100, on an authoritative server even if you have more precision then the clients can simply send an approximation of the other players health between 0-100 which you can represent with 7bits of a byte would be enough for health.

1 Byte - BOOL - stream.Write(platform.move)
Not completely sure what you are using this for but it’s only a bool so 1bit is sufficient, I would combine this with your 7bits needed for health for shipping over the network, therefore no needing a seperate item.

4 Byte - Float - stream.Write(_myTransform.position.x) stream.Write(_myTransform.position.z)
With 2 bytes you can represent a number up to 65536 so we could represent the opponent players position up to a distance of 655.00 (depending on your game 2 decimal places is probably accurate enough for most games). I’m not sure how big your maps are but with only 2 players in game then 655 world units will probably be large enough.

4 Byte - Float - stream.Write(_myTransform.rotation.eulerAngles.y) stream.Write(_turretTransform.rotation.eulerAngles .y)
Using 4 bytes for an angle is way over the top as it gives your around 7 decimal places of precision, as per the position I would approximate this over the network using 2bytes for rotation with 2 decimal places.

Total
Applying all of these optimizations you’d reduce your 21bytes down to 9bytes (13bytes if your maps are larger than 655world units)

Each player has 5 vehicles but I’m not sure how he controls them? is it 1 at a time or by selecting groups and moving together in formation, you may be able to just synch positions of the group over the network or only send state synch when positions have actually moved or various things to reduce traffic.

  1. Running too many unity instances on your server will be another bottleneck as there is an overhead for each instances started up, lots of garbage collection runs for mono, etc.

I would recommend you look at uLink over at http://www.muchdifferent.com it is very similar to Unity Networking and not only has different datatypes supported as mentioned in my answer to number 1 but also has a groups feature so that several of your matches can take place on 1 unity server.

There is an example on their site http://developer.muchdifferent.com/unitypark/Tutorials/uLinkNetworkGroups of using groups to have 20 matches with 2 players in each match running inside 1 server. Note that for collision detection that you require you’ll either use layers on the server or simply duplicate your scene on the Y axis and have game 1 with Y =0, game 2 with Y=200 and so on. I can share more details on how to achieve if needed.

That will allow you to get substantially more games running on a physical machine.

  1. I would really advise against the proxy model regardless, it’s just too easy to hack and for players to ruin great games by trying to cheat. It’s difficult to offer you much of an estimate for a 200CCU model as it’s very dependant on game design, time available to optimize traffic/memory/cpu usage etc. For the parts you’ve mentioned though I would expect you to be somewhere in the $50-80 a month for 200CCU range.

Spend the time and performance tune it then I’d guess reaching 400CCU on a $50-80 should be easily achievable.

4

Bariel, thanks for detailed answer, and good advice for optimizations.

I did my networking as a fast prototype to see how the multiplayer in the game will run.
And it run very good and smooth, even with huge latency (response time).
Now I will start to decrease the size of the data transferred, according to your advice.

In the game you control each vehicle independently, you give set of commands,
Multiple move commands and fire, and dynamically change them during the game.

Switching to Authoritative Server will not help in bandwidth:
In proxy I am transferring 5 units IN 5 units OUT for each player total 20 for a 2 player battle.
In AS it will be 10 out from server for each player, also 20 units per 2 players battle.

I can reduce the rate from 15 to 10 per seconds, but this is something that I can optimize always, and because my game is very physics dependent, I found that with 15 I have less errors in vehicles positions.

The only benefits in AS will be

  1. Latency (response time), as it is twice less distance to travel.
  2. Security – no hacking.

I started to experiment with uLink and uZone in implementing Authoritative Server.
I trying to make it with minimum changes from the client version, to use the same unity project, as it make it easier to maintain the code. I just strip the textures and meshes from all objects in server scene, and leave green collision boxes only. many classes are the same with if (server) statements.

You probably right that running multisession in one process will be more efficient than single session per processes.
Specially for my game with only 2-3 players.
But how much. This change require to code server in completely different way than a client.
Almost two separate Unity Projects for server and client, something I prefer to avoid.
And the logic in create/manage games, sessions, will be much more complicated.

Let’s say I will run 5 games in the same game server instance,
How much of those I will be able to run now on one server?
Probably max 80 then it is 400 battles = 800 CCU in the game that run on a $100 server.

If I understand you correctly, with 200 CCU I should have around $50-80 / month.
And with 800 it will be $200-320 / month per one server that cost me $90/month.
But with proxy I can run unlimited number of games.

You think it is better to code AS with much more complicated and time consuming logic,
Plus costs of $90 per 800 CCU and move from Proxy that already working good.

Or maybe I am wrong in my calculations and much more games the server can handle
If I will use multisession.