Buffer Overflow after disconnection

Unity Engine
,
1

I know that when using TCP (like WebSockets) updating the state of the game every frame is not the most clever way as UDP should be used for this. So I do not plan to do this in practice, but while testing my code I encountered a problem regarding this.

In UDP the problem doesn’t exist.

Here is what I do:

  • I use WebSocketNetworkInterface
  • I send packet every frame using the pipeline created with CreatePipeline() without any arguments, which in WebSockets will always give me ReliableSequenced as there is no way to change it (as far as I know).
  • Server is started and I connect a client to it.
  • Then I disconnect the client and most of the times (not always) I get this warning:

My guess is that it is related to the 32 packets in flight limit. Even though the server and the client are on the same machine, they send hundreds of packets per second. So somehow the buffer gets overflowed with the packets that are not confirmed yet. But what is interesting is that it only happens after I disconnect the client.

So I think this warning should not be displayed at all. I do all sort of checks to not use BeginSend and EndSend on connections that are disconnected (code below). And from what I have seen after the disconnection is detected, the BeginSend and EndSend are not being called. Also there is no error at any point returned by BeginSend and EndSend.

What may be important is the fact that I use ScheduleFlushSend().Complete() to send the packets at the end of the frame (to avoid waiting for the next frame and ScheduleUpdate().Complete() that is being called at the beginning of the frame). I also use all BeginSend and EndSend calls just before I flush send. I do all this in LateUpdate() from a script marked as [DefaultExecutionOrder(10000)]. To make sure all the packets will be sent at the very end of the frame.

I think it is potentially something that can be fixed, because I don’t see any point in showing a warning for a connection that was disconnected.

The code I mentioned above, how I check if the packet should be sent:

public static bool CanSendOrReceiveData(NetSocket netSocket, NetworkPipeline networkPipeline, int connectionIndex)
        {
            if (netSocket.owner == NetSocketOwner.Server && netSocket.status == NetSocketStatus.Connected)
            {
                if (netSocket.thisAsServer.multiNetworkDriver.IsCreated
                    && netSocket.thisAsServer.connections[connectionIndex].isUsed
                    && netSocket.thisAsServer.connections[connectionIndex].networkConnection != default
                    && netSocket.thisAsServer.multiNetworkDriver.GetConnectionState(netSocket.thisAsServer.connections[connectionIndex].networkConnection) == NetworkConnection.State.Connected
                    && (netSocket.thisAsServer.connections[connectionIndex].isConnected
                        || (netSocket.thisAsServer.connections[connectionIndex].isWaitingForAuthorization && networkPipeline == netSocket.rpcPipeline)))
                    return true;
            }
            else if (netSocket.owner == NetSocketOwner.Client)
            {
                if (netSocket.thisAsClient.networkDriver.IsCreated
                    && netSocket.thisAsClient.clientConnection != default
                    && netSocket.thisAsClient.networkDriver.GetConnectionState(netSocket.thisAsClient.clientConnection) == NetworkConnection.State.Connected
                    && (netSocket.status == NetSocketStatus.Connected
                        || (netSocket.status == NetSocketStatus.ClientWaitingForAuthorization && networkPipeline == netSocket.rpcPipeline)))
                    return true;
            }
            return false;
        }
2

So a small precision, no data is going through the reliable pipeline unless you explicitly ask for it, even when using WebSockets. So if you send on an empty pipeline, your packets will not touch the reliable pipeline stage at all. Of course, when using WebSockets the packets will be delivered reliably anyway, because that’s the nature of a TCP-based connection. But the reliability will be provided by the TCP stack, not the reliable pipeline. So the 32 in-flight packets limit would not apply in your scenario.

The “send buffer overflow” warning is emitted when the TCP send operation returns an error. I’m guessing here it’s because it’s trying to send on a socket after the remote peer has closed it. If that’s the case I agree with you that no warning should be logged here. It’s pretty weird that it only happens when there are multiple client connections though, I’d expect the issue to arise even with one client…

After the warning is emitted, the transport should generate a disconnect event on the connection for which the warning was logged. Could you check which of the two connections on the server receives the event? Is it the connection that was actually closed, or the other one?

On our end I’ll file a bug to make sure that the warning is only logged when it was intended to be (when the TCP OS buffers are full), and to properly handle other errors the TCP socket could report. Thanks for reporting this!

1 Like
3

Thank you for the answer.

Sorry for the confusion, I edited the post but you read it before my edit. It happens with one client too.

I also wonder if these warnings are issued only in the Editor or also in the build? I wonder if they impact performance and if there is a way to disable them in the build.

4

The warning is conditional on ENABLE_UNITY_COLLECTIONS_CHECKS, which is only defined in the editor.

And good to read that it also happens with a single client. It does point to mishandling the error from a disconnected socket. I’ve filed a bug on our end to address this. Thanks!

1 Like