I’m testing the performance of a Unity game server, which simulates avatar movement using CharacterController. Server-client communications are pure RPC.
Client sends user input to server. Server will distribute the input to clients near the sender.
After some testing, I observed that when the no. of nearby clients reaching some limit, messages within that client group start getting delayed. Clients outside that group are not affected.
When messages being delayed, memory leak starts. Server will crash eventually because out of memory.
My conclusion is, each NetworkPlayer has some bandwidth limit, for preventing a small group of clients occupying large bandwidth. But RPC is reliable and cannot be dropped. The send queue will keep growing and results in message delay and memory leak.
My question is, is there any straight-forward way to detect a NetworkPlayer has reached such limit? Thanks.
It sounds like you are bombing the whole networking and are actually running out of buffer space in general, getting messages dropped / overwritten in the ring buffer.
Keep in mind that RPC does neither respect the network send rate nor does it respect SetScope, so if you use an rpc on each input event / update, you will get into massive trouble, as already a handfull players with a high end pc (and thus high framerate) can flood the network
Yes, these are issues have to pay attention to. I have done something to address this. The basic idea is keeping a NetworkPlayer list (Built from spacial claster) and send RPC to each NetworkPlayer one by one.
And from my test, I found that the leaking condition does not relate to overall bandwidth usage at all:
- 20 dense clients, 300 RPC/frame → delay and leak
- 100 sparse clients, 3000 RPC/frame → no problem
It seems that leaking happens when some clients are receiving too many messages.
Sure that it leaks and that it is not your code?
Cause if you send too many messages to them, they basically will start having a latency on processing them (their buffer fills up to the point it at worst starts to drop incomming messages)
At worst you could send in a bug report with the project, so UT can attempt to reproduce it within the debug environment and get more detailed informations.
If you are on osx, consider running it with shark, that gives you the possibility to trace malloc for example. (and you can profile the app as whole)
You should potentially add in some throttling, so that if it becomes dense, that there is a priority structure so important stuff is sent first and unimportant is not sent through or far less frequent (interaction → position → animation → eye candy messages).
Good indicator for this situation can be the pings/roundtrip times to the client
I myself use RPCs pretty regularily, but I’ve added a delta time based RPC sending with the delta time beeing the one from the network settings. That way RPCs keep themself within reasonable borders
by the way: Interesting that others approached a “self managed sync list” approach with Unity. Wish you good luck with it 
I can’t guarantee that’s not my problem, but I’m quite sure:
-
Ping time. Ping time keeps roughly constant, no matter messages are delayed or not.
-
Fast server response. When I move an avatar in client, the avatar in server moves immediately, only the remote avatar in other clients is delayed. That means server received my movement command normally, processed and broadcasted the command correctly (because the server moves the avatar immediately). The only reason I can imagine is Unity/Raknet delayed the broadcasting to other clients.
That means you buffer the RPCs yourselves?
The game logic have completely sticked to Unity…Besides working around Unity networking on my own, I’ve no choice
Kind of, just that I ensure that only 1 is beeing sent instead of (in case of my core i7 + gtx 280) 10+, which the server never ever can process within reasonable amount of time.
Actually my server is setup to tick at a 30hz rate, there is no reason to do it faster as 30hz already is 33ms latency, thats faster than the vast majority of users even will be able to receive due to the common pings.
Should you be doing something in the direction of an MMO, 30hz is already quite a lot, as MMOs normally have average pings of 100-150ms, so even 15 - 20 still would do the job, but leave you much more server time.
What you need to do to make this happen is basically have an additional field to sum up the time that a given action was done since the last send so you can send through (if you send the input)
Yes, that’s a good idea to trim down client side outgoing messages.
I just implemented RPC ping for checking the delay (together with memory leak). The basic idea is sending a timestamp to clients and clients reply the timestamp. Server can detect the delay by comparing the replied timestamp with the current timestamp.
With some “eyesight” control when delay is detected, I can effectively solve the delay and leaking problem, provided that no client is cheating…Hackers can still crash the server by sending instrumented timestamp to server.
So I still want to know if I can access the send queue of NetworkPlayer through Unity.