Network problem, buffer overflow (in 5.1.3)

Unity Engine
1

Hello!

We’re running the online multiplayer shooter (webplayer clients, stack of authoritative Linux Unet servers), and now players started to report “jittery” behaviour during gameplay.

We use reliable sequenced QoS channel (with index 0) for sending game events and unreliable sequenced QoS channel (with index 4) for syncing vars.

We had similar issues before, they were cured with upping the values of NetworkDropThreshold and OverflowDropThreshold, but it seems that the new factor is introduced now.

We now tend to think that the problem is caused on (our) server side - the packet buffer gets flooded, probably due to poor connection with one or several clients, and then server start spamming the log with “no free events for message” entries so intensively that is causes lags on client side.

Less frequently, we also get loads of following log messages: “Warning: received system packet belongs to wrong session”

What would you suggest to handle this buffer overflow?

Please check below the log examples:

===
…
Log: connection {8} has been disconnected by timeout; address {178.134.72.3:60920}
aissp: time {1098566} lastRecTime {1078263} DisTme {20000}, rtt {93}

(Filename: Line: 1845)

Player [IP: 178.134.72.3 disconnect from 95.213.162.94 GUID: 8 ]

no free events for message

(Filename: Line: 358)

no free events for message

(Filename: Line: 358)

no free events for message

(Filename: Line: 358)

no free events for message

(Filename: Line: 358)

no free events for message

(Filename: Line: 358)

no free events for message

(Filename: Line: 358)

no free events for message

(Filename: Line: 358)

no free events for message

(Filename: Line: 358)

no free events for message

(Filename: Line: 358)

no free events for message

(Filename: Line: 358)

no free events for message

(Filename: Line: 358)

no free events for message

(Filename: Line: 358)

no free events for message

(Filename: Line: 358)

no free events for message

(Filename: Line: 358)

no free events for message

(Filename: Line: 358)

Failed to send internal buffer conn:3 channel:4 bytesToSend:1130

Send Error: 4 conn:3 channel:4 bytesToSend:1130

no free events for message

(Filename: Line: 358)

1 Like
2

no free events for message could indicate that your MessageQueue is running full before being able to send. The default limit on this is 128.

Try putting connectionConfig.MaxSentMessageQueueSize = 256; (or a higher number) in the Awake() of your Networkmanager to see if it makes a difference.

3

Firstly I’d try updating to latest Unity version (5.3.2 atm) to see if there have been improvements here.

As for buffer overflow issues in general (no free events), these can happen because this is on a reliable channel and the round trip time is too high, the packets need to be kept in buffers until the ack message from the remote party arrives (you can check rtt with NetworkTransport.GetCurrentRtt). Trying to send less would obviously help, how you’d do that depends on what your doing/using. The reliability layer also can combine messages into bigger packets which can help. You can tune those with MaxCombinedReliableMessageCount (default is 10, could bump to 20) and MaxCombinedReliableMessageSize (maximum size which is allowed to be combined, default is 100, could try 200 or more).

However, there is that disconnect message at the top, are there no free event errors above that? The connection could drop if there is too much packet loss or simply nothing was received for 2 seconds.

More information could help diagnose the issue, like how many clients are connected (and does it happen with fewer clients?), how much is being send per second, how big are the packets, what are all the connection parameters.

4

Thank you for you answer, Larus.

Please find below some additional information about our connection parameterss.

Our connection configuration:
AckDelay = 33
AllCostTimeout = 20
ChannelCount = 0
Connection timeout = 10000
Disconnect timeout = 20000
Fragment size = 500
Is Ack long = True
Max combined reliable message count = 10
Max combined reliable message size = 100
Max connection attempt = 32
Max sent message queue size = 2048
Min update timeout = 20
Network drop threshold = 40 // we had to set these high to avoid UNet disconnects during lag spikes
Overflow drop threshold = 40 //
Packet size = 1500
Ping timeout = 500
Reduced ping timeout = 100
Resend timeout = 500

The problem usually happens when there are 8 or more players connected. When this number is lower, everything works fine. During the battle server normally sends around 20-80 custom messages per second (4-30 bytes each) via reliable sequenced channel, but number of messages jumps to 120-150 per sec when there’s intense gunfight. Besides that there is somewhat like 500-700 (5-7 per client, 10 times per second) NetworkBehaviour objects serializations per second (over unreliable sequenced channel).

As for the disconnect - yes, there was no “No free events” errors before that. Moreover, this typically happens after server receives “Client is ready” message (in OnServerReady callback). As far as we could guess after this client starts receving spawn data and, supposedly, connection get stuck which causes disconnect.

But we also sometimes see disconnects during battles even if initialization was successful.

5

Thanks Scorr, will try that!

6

We’re investigating “no free events” as well, and we believe that when setting the ConnectionConfig.MaxSentMessageQueueSize to a non-default value, it may actually make things worse in some cases, since Networking.HostTopology.MessagePoolSizeGrowthFactor is ignored (Case 773411), which leads to the pools used to process the events in the LLAPI to be too small relative to the number received, and so the client will discard many of the events, and cause a lot of resends if the server sends a lot at once. We are testing setting our HostTopology pool sizes to match our ConnectionConfig.MaxSentMessageQueueSize

Any particular reason these can’t be larger? I’m considering using our ConnectionConfig.PacketSize for MaxCombinedReliableMessageSize and MaxCombinedReliableMessageCount to be orders of magnitude larger.