Some newbie questions regarding Unity Transport

In my current project I'm using NGO and custom messages for communication between server and client. I need to do away with this high level solution so I'm looking to use Unity Transport instead. I'm just feeling my away along right now and have a few questions regarding Unity Transport and the Jobs system.

I couldn't see instructions on how to stop a running server, is with driver.Dispose()? If clients are connected presumably I need to wait until their disconnect jobs are complete before disposing of the driver?

For serialised messages in NGO I've been using FastBufferReader/Writer and serialising the individual fields with Read/WriteValueSafe. I'm struggling to see what the best approach would be using Unity Transport and have the sending of messages be part of a job. Is it still possible to follow this approach or is there a better way to do this?

Looking in the example code for ServerUpdateConnectionsJob it has this code:

        // CleanUpConnections
        for (int i = 0; i < connections.Length; i++)
        {
            if (!connections[i].IsCreated)
            {
                connections.RemoveAtSwapBack(i);
                --i;
            }
        }

This seems to swap the current position with the last, and if so does that mean the last connection is checked twice, or is the last element removed and if so is connections.Length reduced during the for loop?

That's all the questions for now. :)

Calling Dispose will close the underlying socket, which effectively stops the server. But this won’t notify the remote peers that the server is closed (they’ll just eventually timeout). To do this, you’d need to call Disconnect on all connections, run one last update job (with ScheduleUpdate), and then dispose of the driver.

You can use DataStreamWriter to serialize individual fields of structure message. It also offers methods to write packed versions of common types and deltas if you think that will be smaller on the wire. There’s no higher-level framework to serialize bigger structures because we want it to be flexible enough to integrate into any kind of system. For example, you could write an interface with serialize/deserialize methods that all your messages need to implement. That’s how Netcode for Entities deals with RPC data for example.

Regarding how to send messages as part of a job, did you have something specific in mind? Almost all operations of NetworkDriver can be performed in a job. For example the BeginSend and EndSend operations (along with all the serialization in-between) can occur in a job. You can simply pass a NetworkDriver to a job and use it from there. If you want to send on different connections in a parallel fashion, you can use NetworkDriver.ToConcurrent to obtain a version of the driver that can be passed to multiple concurrent jobs. Otherwise the actual sending (writing to the socket) always occurs in a job, either the ScheduleUpdate job or the ScheduleFlushSend job (the latter only performs send operations).

RemoveAtSwapBack actually decreases the length automatically (I agree that it’s not obvious at all). So basically it copies the given element with the last element of the list, then decreases the length by 1. It’s a way of removing an item from the list without copying too much memory around, at the cost of changing the order of the elements. The decrease of i is then required because we want to re-process the connection at that index (which is now the previous last connection in the list).

1 Like


That's pretty much what I was expecting, I'll use a flag or state to keep track or it.

The example is very similar to how it works in NGO, from what you've said and what I've seen it looks like I won't have to modify my code too much which is a bonus as the message system is fairly elaborate and I didn't want to have to rewrite it all.

What I have currently is a service that has a call SendMessage(ulong clientId, IMessage message) that kicks off the serialisation and sending of the message. So in theory I can create a job passing in something very similar and inside the job use the DataStreamWriter from BeginSend and pass that along in a similar fashion to FastBufferWriter for the serialisation. I'll have a play with it and see how I get on. This is quite a shift from what I'm currently used to, I've not had to use Update at all apart from running timers. :)


That makes sense, I wanted to check as that's the first time I've seen that behaviour.

Many thanks for the answers Simon, I think I've enough understanding to make a proper start.

Just be careful that passing non-blittable types (like interfaces) to jobs is not supported. You can use generics (a generic job or make your send method generic), but of course that requires knowing the concrete types of each call site at compilation time.

Hmm okay, something else to think about. I didn't get a chance to look at it yet but I did look at stopping the server and start on some way of triggering events based upon network events. I hope you don't mind if I run some code passed you in case I'm missing something I can make use of or my ideas are flawed.

For stopping the server I'm calling disconnect for each connection then setting a state to track where I am. This is the scheduling code called in Update:

    private void ScheduleByState()
    {
        switch (networkState)
        {
            case ServerNetworkState.Started:
                serverJobHandle = networkDriver.ScheduleUpdate();
                serverJobHandle = connectionsJob.Schedule(serverJobHandle);
                serverJobHandle = updateJob.Schedule(networkConnections, 1, serverJobHandle);
                break;

            case ServerNetworkState.ShuttingdownStart:
                Debug.Log("ShuttingdownStart");
                serverJobHandle = networkDriver.ScheduleUpdate();
                networkState = ServerNetworkState.ShuttingdownEnd;
                break;

            case ServerNetworkState.ShuttingdownEnd:
                Debug.Log("ShuttingdownEnd");
                networkDriver.Dispose();
                networkState = ServerNetworkState.Shutdown;
                break;
        }
    }

This appears to work fine, at least I'm not getting the usual Dispose related errors.

For network event handling I'm not sure on the best course of action. What I've done is have each NetworkConnection as part of a struct with an event type I can act on after a network event has occurred.

public struct ServerNetworkConnection
{
    NetworkConnection connection;
    ServerEventType eventType;

    public ServerNetworkConnection(NetworkConnection connection, ServerEventType eventType = ServerEventType.None)
    {
        this.connection = connection;
        this.eventType = eventType;
    }

    public NetworkConnection Connection { get => connection; }
    public ServerEventType EventType { get => eventType; set => eventType = value; }
}

And the update jobs Execute code (this is just prelimary and only Disconnect does anything):

    public void Execute(int index)
    {
        NetworkConnection connection = connections[index].Connection;

        if (connections[index].Connection.IsCreated)
        {
            NetworkEvent.Type eventType = driver.PopEventForConnection(connection, out DataStreamReader reader);

            switch (eventType)
            {
                case NetworkEvent.Type.Empty:
                    // do nothing
                    break;

                case NetworkEvent.Type.Data:
                    Debug.Log("ServerUpdateJob Execute Data: " + reader.Length);
                    break;

                case NetworkEvent.Type.Disconnect:
                    Debug.Log("ServerUpdateJob Execute Disconnect: " + reader.Length);
                    connections[index] = new ServerNetworkConnection(connection, ServerEventType.Disconnect);
                    break;

                default:
                    Debug.LogWarning("ServerUpdateJob unhandled eventType: " + eventType);
                    break;

            }
        }
    }

Thie idea is to use the eventType once outside the job to invoke an Action like OnClientDisconnected. The problem with this approach is when I want to set the eventType (and later clear it) I have to create a new struct each time which isn't ideal.

I had some other quick questions.

Is NetworkEvent.Type.Connect client side only?

In the example code they use a while loop to check for events:

while ((cmd = driver.PopEventForConnection(connections[index], out stream)) != NetworkEvent.Type.Empty)

is there a particular reason for this?

On a Data event is it possible to pass out the DataStreamReader to process the message outside the job, or should the message be read inside it?

Thanks for any guidance you can give.

In your shutdown code, I’d make sure that the ShuttingdownEnd case completes the last scheduled update job (if nothing else is completing it before then). Otherwise if the update job has not finished when you’re disposing of the driver, you’ll get errors (or weird and unpredictable errors in production builds).

Yes. Only clients can receive this event. Servers are “notified” of new connections by accepting them with Accept. Note however that Disconnect events can occur on both server and client.

Yes, because multiple events could be popped per update, even for a single connection. For example you could get multiple Data events if multiple packets were received since the last update. Or you could get a Data event followed by a Disconnect event (although not the opposite).

I think that’s something you’ll struggle with if using that connection structure that contains an event, since I’m not sure how that would handle multiple events per connection per update. You could perhaps change it to a queue of connection events, with each element of the queue containing the connection, the event type, and other pertinent information. NativeQueue offers a parallel variant that could be used to push all connection events to the same queue in that situation.

The stream reader is valid until the next update job is scheduled. So you can pass it around to a different job or back to the main thread for processing, as long as you ensure that all processing is done before you next call ScheduleUpdate on the driver.

1 Like

I see, plenty more food for thought there. :)

Perhaps I won't try parcel things up into a single struct, I'm struggling to work out how to get a struct out of the job with information I need. I need a better grounding in the Jobs system for sure so I'll do some more research.

Thanks again Simon!

I followed your advice and corrected my code and implemented NativeQueue.AsParallelWriter and with a bit of trial and error and ChatGPT trying to help I was able to the data out of the job as I needed. Here's the revised struct:

public struct ServerConnectionEvent
{
    int connectionId;
    ServerEventType eventType;
    DataStreamReader streamReader;

    public ServerConnectionEvent(int connectionId, ServerEventType eventType = ServerEventType.None,
        DataStreamReader streamReader = default)
    {
        this.connectionId = connectionId;
        this.eventType = eventType;
        this.streamReader = streamReader;
    }

    public int ConnectionId { get => connectionId; }
    public ServerEventType EventType { get => eventType; }
    public DataStreamReader StreamReader { get => streamReader; }
}

In all it's a better solution to what I was trying to make work and I should be able to make better progress with this part out of the way (not that it's properly tested mind you).

Thanks again for your help.

So I've been able to send a message from client to server. When the client gets a message to send it's added to a NativeList and in Update the messages are sent in parallel. The only sticky part is tracking when to know I can Clear that list after the messages are sent, I'm currently doing with a separate State field.

One difference I've noticed between NGO's FastBufferWriter and DataStreamWriter is that FastBufferWriter can be passed by value and retains its original writer buffer, with DataStreamWriter it has to be sent by reference otherwise it ends up with a new write buffer. Incidentally through testing with this if an empty message is sent to the server it triggers the Data event but there appears to be have no read buffer so I'm checking reader.IsCreated to guard against this.

One major issue I have is I can't use my current message structure. What I have in NGO is interface IMessage which is implemented by an abstract class ClientMessage/ServerMessage which is inherited by other message classes with have their own interface IMessageDetail field for other classes that make up the various message body types. This won't work at all and I'm not sure what's a good approach to go with, if you have any suggestions I'm all ears. :)

What do you mean by tracking when the message is sent? After a successful EndSend call, it can be assumed that the message will be sent and there should be no need to track it anymore on the client.

Sending empty messages is a “feature” as it could be used as some sort of notification mechanism if coupled with a specific pipeline. I think that’s something we should clarify in the documentation though.

Regarding passing writers by value, could you elaborate on the issues you encountered? My understanding is that it should be possible. What is problematic today is passing a writer across jobs due to the way its handle is allocated. That’s something we mean to address eventually.

Could you elaborate on the problems you’re encountering with using that architecture? Is it because it’s using managed types (interfaces, abstract classes) that are not compatible with jobs?

Where I have NativeList clientMessages I'm calling clientMessages.Clear() once those messages have been sent. Originally I wasn't using a list and was just scheduling the job immediately, but thought a list might be the better option?

That sounds good to me. I just wasn't aware there wouldn't be a read buffer so didn't expect the exceptions being thrown, with reader.Length for example as I thought it would return zero.

I created a simple message:

public struct ClientMessage
{
    int value;

    public ClientMessage(int value)
    {
        this.value = value;
    }

    public void DeSerialize(DataStreamReader reader)
    {
        value = reader.ReadInt();
    }

    public void Serialize(DataStreamWriter writer)
    {
        bool result = writer.WriteInt(value);
    }

    public override string ToString()
    {
        StringBuilder stringBuilder = new StringBuilder("ClientMessage");
        stringBuilder.Append(" value ").Append(value);

        return stringBuilder.ToString();
    }
}

In Serialize it looks like the writer is using its own write buffer as when Serialize returns and I call EndSend with the original writer returned by BeginSend the EndSend returns zero. If I have the Serialize call return its writer and call EndSend with that the data is sent. I've gone with sending in the original writer by reference to Serialize instead as the better option.

It is the problem the managed types, or rather it's a problem of coming up with the best alternative understanding the limitations. I could serialise the messages before adding them for sending within a job, but then I'd lose half the benefit of handling them within a job?

Using a list is fine. I guess I was just confused by your original message. I had understood that you had needed to track some kind of “send state” and that had me confused since there are not many states to sending a message (it’s basically just sent or not sent).

Oh that’s a good point. It really would be better to return 0 for the length if the reader is not created. I’ll make a fix for that. Thanks for pointing it out.

Had a quick look at the code, and you are correct. There’s important tracking information that’s stored directly in the DataStreamWriter structure and won’t make it back to the caller if it’s passed by value. So while technically the writer is still using the same buffer, the information of how many bytes were written to it will get lost if passed by value. That’s not ideal. I’ll create a task on our end to address that and in the meantime I’ll note it in the documentation.

Note that DataStreamReader has the same issue. The updated index we’re reading from will not make it back to the caller if passed by value. Probably less of a concern for you since you read the entire thing in one function and don’t use it afterwards, but still something to keep in mind.

You’d lose the benefit of serializing the message in a job, but you’d still benefit from having the network system calls performed off the main thread. All pipeline processing would also occur in a job if you call the BeginSend and EndSend methods in a job too. So there would still be some performance gains to be had.

Otherwise the solutions become complicated pretty fast. Most solutions usually involve tagging each message with a “type” that can then be used to retrieve the appropriate serialization/deserialization function in a job. Unfortunately that also often involves a lot of boilerplate code. Netcode for Entities uses source generators to address the boilerplate issue, for example.

Ultimately I’d recommend going with the simplest approach first and then profile to see if there are performance issues that need to be addressed. For example if it turns out that serializing on the main thread is too performance-heavy, you could look into serializing messages in jobs. The best approach depends on the traffic patterns, but let’s say you often had few message types to send, but many messages of each type. You could create a generic job that can serialize a list of a particular message type, and then pass the resulting serialized buffers to another job that will send them on the network.

Here’s a quick (untested, probably doesn’t even compile) example:

public interface IMessage
{
    void Serialize(ref DataStreamWriter writer);
}

public struct MessageSerializerJob<T> : IJob where T : unmanaged, IMessage
{
    public NetworkDriver Driver;
    public NativeList<T> Messages;

    public void Execute()
    {
        for (int i = 0; i < Messages.Length; i++)
        {
            Driver.BeginSend(..., out var writer);
            Messages[i].Serialize(ref writer);
            Driver.EndSend(writer);
        }
    }
}
1 Like

The simplest approach does have a certain appeal :smile:. I'm not expecting to handle a huge amount of network traffic and Jobs may be overkill for what I'm doing at the moment but it's interesting to learn and makes things more scalable.

Thanks for the example, I'll give it some thought and see what I can come up with. :)

I've been playing with this and decided for now to handle the serialisation and deserialisation outside of jobs. There's too many messages currently to handle individually so I'll look at amalgamating some of them together.

For the serialisation I'm using a new DataStreamWriter for each message then storing that writer in NativeQueue which in Update is scheduled in a job. This appears to work but inside the job the writers appear empty which I assume is due to the same problem you've outlined previously. Is there an alternative way of doing this for now? I think I'm missing a trick as everything I've tried so far results in an error.

Incidentally sending DataStreamReader out of the job looks to be working fine.

As a workaround for now I'm calling writer.AsNativeArray().ToArray() to get the serialised message (I can't use AsNativeArray directly due to its allocation type) and adding it to a job with the list of jobs scheduled in Update.

I've not been able to work out how to get an array of the serialised message arrays in a job, or an array of structs with each containing an array although as an array of DataStreamWriters does work there must be a trick to it.

Mmmh… maybe my previous advice was misguided. There does appear to be difficulties in pre-serializing everything in advance, especially if you’re going to use DataStreamWriter for the serialization.

Perhaps then I would suggest trying to keep the BeginSend and EndSend calls close to where the serialization is happening (e.g. in the main thread) and using the writer provided by BeginSend to serialize your messages. This would avoid copying the buffers around, and would take care of the queueing automatically. (Despite what their names are saying, BeginSend and EndSend basically only put buffers in a queue to be sent later.)

In terms of performance, BeginSend itself is pretty benign and only reserves a buffer from a pre-allocated pool, on top of performing some basic checks. I would not be overly concerned about calling it often on the main thread. EndSend does a bit more work, but its cost can be reduced quite a bit by calling it from Burst-compiled code. You could for example write a small job that simply calls EndSend on all your serialized messages and Run it directly on the main thread (scheduling the job would be risky because EndSend is required to be called in the same frame as BeginSend since they rely on a small temporary allocation). But first I’d profile the code and see if that’s necessary at all.

That should work. I’m guessing perhaps at some point you’re enqueuing a copy of the writer taken before it was written to. I tested this code on my end and it prints “4” as expected:

private struct TestJob : IJob
{
    public NativeQueue<DataStreamWriter> Q;

    public void Execute()
    {
        var writer = Q.Dequeue();
        Debug.Log(writer.Length);
    }
}

public void Test()
{
    var a = new NativeArray<byte>(4, Allocator.Persistent);
    var writer = new DataStreamWriter(a);
    var q = new NativeQueue<DataStreamWriter>(Allocator.Persistent);

    writer.WriteInt(42);
    q.Enqueue(writer);

    new TestJob { Q = q }.Schedule().Complete();

    a.Dispose();
    q.Dispose();
}

I did wonder if using DataStreamWriters this way was a good idea. I'll look into keeping it simple and as you suggest serialise and send in the main thread.


My code is not far removed from your test code and I see now I'm getting the same result as you. I was using a different constructor DataStreamWriter(256, Allocator.Temp) which wasn't allowing me to call writer.Length as it was WriteOnly.

Actually now I can check the length I can see the problem, I'm passing the dequeued writer into the out of BeginSend and the writer length is now zero. Here's the job code:

public struct ClientSendJob : IJob
{
    public NetworkDriver networkDriver;
    public NativeArray<NetworkConnection> connection;
    public NativeQueue<DataStreamWriter> writers;

    public ClientSendJob(NetworkDriver networkDriver, NativeArray<NetworkConnection> connection, NativeQueue<DataStreamWriter> writers)
    {
        this.networkDriver = networkDriver;
        this.connection = connection;
        this.writers = writers;
    }

    public void Execute()
    {
        while (writers.TryDequeue(out var writer))
        {
             Debug.Log("ClientSendJob Execute length: " + writer.Length);
            if (networkDriver.BeginSend(connection[0], out writer) == 0)
            {
                Debug.Log("ClientSendJob Execute BeginSend length: " + writer.Length);
                int result = networkDriver.EndSend(writer);

                Debug.Log("ClientSendJob Execute result: " + result);
            }
        }
    }
}

I guess passing in a writer rather than using the one provided by BeginSend is a bad idea?

Yes, that's a bad idea. The passed-in writer will be overwritten by the new one created in BeginSend.

Unfortunately there is no way to use a DataStreamWriter that you created yourself with the transport package. EndSend can only really work with writers that have been previously created with BeginSend (which itself will always create new ones from scratch). This is why I suggested serializing with writers directly obtained with BeginSend. Otherwise you'll have to basically copy the content of your custom writer to the one provided by BeginSend.

(And for reference, the reason we don't allow custom writers is that the backing buffers of the writers returned by BeginSend may be pre-registered with the OS network stack, which allows for zero-copy sends on some platforms.)

Got you. I'll look into going the conventional route and reduce the number of messages and serialise with the provided writer.

I'd looked into this previously but due to the setup of the project, where it's potentially an asset used by other developers I was passing out the message directly and didn't want to have messages with lots of fields where only some are relevant to a particular message type. I'll have to put in an extra layer to simplify the output.

Thanks for letting me know.