Cannot set Endpoint.Family to NetworkFamily.Custom

Hello,

The Unity Transport 2.1 change log states the following:

However, the property public NetworkFamily Family does not allow setting or getting a a NetworkFamily.Custom value.

/// <summary>Get or set the family of the endpoint.</summary>
/// <value>Address family of the endpoint.</value>
public NetworkFamily Family
{
    get => FromBaselibFamily((Binding.Baselib_NetworkAddress_Family)BaselibAddress.family);
    set => BaselibAddress.family = (byte)ToBaselibFamily(value);
}

This is because FromBaseLibFamily() and ToBaseLibFamily() do not have a case for Custom.

private static NetworkFamily FromBaselibFamily(Binding.Baselib_NetworkAddress_Family family)
{
    switch (family)
    {
        case Binding.Baselib_NetworkAddress_Family.IPv4:
            return NetworkFamily.Ipv4;
        case Binding.Baselib_NetworkAddress_Family.IPv6:
            return NetworkFamily.Ipv6;
        default:
            return NetworkFamily.Invalid;
    }
}

private static Binding.Baselib_NetworkAddress_Family ToBaselibFamily(NetworkFamily family)
{
    switch (family)
    {
        case NetworkFamily.Ipv4:
            return Binding.Baselib_NetworkAddress_Family.IPv4;
        case NetworkFamily.Ipv6:
            return Binding.Baselib_NetworkAddress_Family.IPv6;
        default:
            return Binding.Baselib_NetworkAddress_Family.Invalid;

Baselib_NetworkAddress_Family lives in
Unity.Baselib.LowLevel so I cannot modify it.

public enum Baselib_NetworkAddress_Family
{
  Invalid,
  IPv4,
  IPv6,
}

I am attempting to yeet custom in there like below, but I am still having trouble with it returning to ipv4 by the time I endpoint reaches my custom INetworkInterface’s bind() method.

private static NetworkFamily FromBaselibFamily(Binding.Baselib_NetworkAddress_Family family)
{
    if ((int)family == (int)NetworkFamily.Custom)
    {
        return NetworkFamily.Custom;
    }
   
    switch (family)
    {
        case Binding.Baselib_NetworkAddress_Family.IPv4:
            return NetworkFamily.Ipv4;
        case Binding.Baselib_NetworkAddress_Family.IPv6:
            return NetworkFamily.Ipv6;
        default:
            return NetworkFamily.Invalid;
    }
}

private static Binding.Baselib_NetworkAddress_Family ToBaselibFamily(NetworkFamily family)
{
    if (family == NetworkFamily.Custom)
    {
        return (Binding.Baselib_NetworkAddress_Family)NetworkFamily.Custom;
    }
   
    switch (family)
    {
        case NetworkFamily.Ipv4:
            return Binding.Baselib_NetworkAddress_Family.IPv4;
        case NetworkFamily.Ipv6:
            return Binding.Baselib_NetworkAddress_Family.IPv6;
        default:
            return Binding.Baselib_NetworkAddress_Family.Invalid;
    }
}

Goal is for these end points to be carrying steam IDs for my steamworks transport integration.
Any advice for a workaround?

Cheers.

1 Like

The workaround currently is just use one of the existing Ipv4 or 6 family in the slot and make sure the port is non-zero and address valid (so not 0.0.0.0 or 127.0.0.1 for Ipv4 / 0:0:0:0:0:0:0:0 or 0:0:0:0:0:0:0:1 for Ipv6).

In the very rare case whatever custom values you’re overriding the endpoint with might be either of those cases, I recommend setting the first byte to a non-127 value (like 255) and just use the next few unused bytes. That’ll guarantee your endpoint won’t get accidentally flagged by any sanitation function.

Something like this:

// * 00 - 00: Padding for skip sanity.
// * 01 - 09: ULong IntPtr.
// ? 10 - 15: [UNUSED].
// * 16 - 17: Port unused but required by sanity.
// * 18 - 18: Family byte.
// ? 19 - 23: [UNUSED].
NetworkEndpoint endpoint = default;
byte* ptr = (byte*)&endpoint;

// Ensuring that in a very rare case, the address isn't loopback.
*ptr = 255;

// Write IntPtr to next 8 bytes.
*(IntPtr*)(ptr + 1) = value;

// Signal valid aliased values instead of an actual endpoint.
endpoint.Port = 12345;
endpoint.Family = NetworkFamily.Ipv6;
return endpoint;

Right. Well I think I got around the sanitisation by yeeting in custom via the code posted above. Now I’m working through the whole "the endpoint you pass in to connect() is overwritten before it reaches bind()" debacle lol.

Not being able to set the family to custom is definitely a mistake on my part. Don’t know how I managed to miss that. This will be addressed in the next version of the package. Sorry about that bug.

Regarding the endpoint not making it to the Bind call, I think it’s because if a NetworkDriver is not bound when Connect is called, it automatically binds to a wildcard address. I’m guessing you’re getting NetworkEndpoint.AnyIpv4 in your Bind call? The fix would be to bind to your endpoint before connecting, but I don’t think Netcode for Entities allows doing that. At least not easily. If you can grab a reference to the NetworkDriver Netcode is using, a workaround would be calling Bind on this before connecting.

I’ll also modify the transport package to avoid binding to a wildcard address when using custom endpoints. That behavior makes sense when dealing with IP addresses, but not custom endpoints. So that issue should also be resolved with the next version of the transport package.

2 Likes

Yes you’re spot on, I deduced this myself, but then had trouble calling bind myself because netcode requires internal access to get the DriverStore before hand.

I ended up getting internal access and calling bind before connecting. So now I can call NetworkStreamDriver.BindAndConnect() instead of just Connect().

public static void BindAndConnect(this NetworkStreamDriver driver, EntityManager entityManager, NetworkEndpoint endpoint)
{
    driver.DriverStore.GetNetworkDriver(NetworkDriverStore.FirstDriverId).Bind(endpoint);
    driver.Connect(entityManager, endpoint);
}

With my temp fix for custom and this, I think I have got everything I need for this to work now.
While I have your attention may I ask a quick extra question…

When sending packets via steam, I have to choose some send flag options. I assume reliability should be handled by the layers above and so I should be sending unreliable packets only. But how about Nagle’s algorithm and NoDelay options? I’m trying to keep out of the way of the upper networking layers but not fully sure what they do…

    [Flags]
    public enum SendType : int
    {
        /// <summary>
        /// Send the message unreliably. Can be lost.  Messages *can* be larger than a
        /// single MTU (UDP packet), but there is no retransmission, so if any piece
        /// of the message is lost, the entire message will be dropped.
        ///
        /// The sending API does have some knowledge of the underlying connection, so
        /// if there is no NAT-traversal accomplished or there is a recognized adjustment
        /// happening on the connection, the packet will be batched until the connection
        /// is open again.
        /// </summary>
        Unreliable = 0,

        /// <summary>
        /// Disable Nagle's algorithm.
        /// By default, Nagle's algorithm is applied to all outbound messages.  This means
        /// that the message will NOT be sent immediately, in case further messages are
        /// sent soon after you send this, which can be grouped together.  Any time there
        /// is enough buffered data to fill a packet, the packets will be pushed out immediately,
        /// but partially-full packets not be sent until the Nagle timer expires.
        /// </summary>
        NoNagle = 1 << 0,

        /// <summary>
        /// If the message cannot be sent very soon (because the connection is still doing some initial
        /// handshaking, route negotiations, etc), then just drop it.  This is only applicable for unreliable
        /// messages.  Using this flag on reliable messages is invalid.
        /// </summary>
        NoDelay = 1 << 2,

        /// Reliable message send. Can send up to 0.5mb in a single message.
        /// Does fragmentation/re-assembly of messages under the hood, as well as a sliding window for
        /// efficient sends of large chunks of data.
        Reliable = 1 << 3
    }
1 Like

You are correct that reliability will be handled by upper layers in the transport, so the network interface is expected to send everything unreliably. Regarding the other options:

  • NoDelay should be safe to set. The transport is set up to handle unreliable networks. In fact it’s probably better to set it and let the transport handle the packet losses than have the Steam SDK buffer the packets and send them all in one batch later on.
  • NoNagle is up to you, really. It’s a tradeoff between latency and bandwidth efficiency. Disabling Nagle’s algorithm improves latency, at the cost of possibly sending many smaller packets that are less bandwidth-efficient. I’d disable it, personally (that is, I’d set the NoNagle flag). For video games, latency is usually more important than bandwidth-efficiency. Plus, if using Netcode for Entities on top, the framework will already be doing a good job of filling packets with as much data as possible so I’m not sure there’d be much bandwidth savings to be done by keeping Nagle’s algorithm enabled.
2 Likes

FYI, version 2.2.1 of the transport package is now available, and contains fixes for both issues mentioned here (not being able to set a custom family, and not auto-binding to the connect endpoint for custom endpoints). It also increases the size of network endpoints (60 bytes are available for custom data).

2 Likes

Ok I’m still having trouble with Custom.
Am I crazy or is Custom family still not really supported…
Here’s my thinking.

It seems like one core issue is that to access the NetworkEndpoint.Port property, we have to go through
the internal unsafe Binding.Baselib_NetworkAddress* BaselibAddressPtr property, whose getter calls
CheckFamilyIsIPv4OrIPv6 which will throw an exception if you have a custom family.

Therefore, when family is set to custom, accessing NetworkEndpoint.Port throws an exception.

Point 1:
With a custom family set, you cannot call NetworkStreamDriver.Connect.

  • NetworkStreamDriver.Connect: 175 → SanitizeConnectAddress
  • SanitizeConnectAddress: 85 → if (endpoint.IsLoopback)
  • IsLoopback → Port → BaselibAddressPtr → CheckFamilyIsIPv4OrIPv6 → Exception :frowning:

Point (question?) 2:
This second point is more of a curiosity. It’s not blocking anything in my project.
On the server, you have to call NetworkStreamDriver.Listen(), passing in one endpoint.
That one endpoint might have some custom behaviour if your family is custom.
In my case, because it’s the server, it’s going to try to create a default IPC driver and, and a Steamworks driver as a replacement for the UDP driver.

If your custom driver needs a custom family, how is the IPC driver meant to work with the same endpoint?
Fortunately for me, the Steamworks driver doesn’t need any info from the endpoint to listen, so I can just set my server’s endpoint to be exactly what the IPC driver needs.

If someone needs custom endpoint data for their listener, when we end up calling NetworkStreamDriver.SanitizeListenAddress, accessing the Port field will thrown an exception. Is this the intended design?

My solution for this is to localise Transport and comment out CheckFamilyIsIPv4OrIPv6(); at NetworkEndpoint.BaselibAddressPtr:110. It seems like Netcode could also stop trying to access address fields when the family is Custom, but I’ll leave the solution up to you guys haha.

I think is is now an implementation issue for the entities netcode folks as they haven’t made their sanitation functions aware of custom bindings in their logic.

And the single endpoint for all drivers when binding is also their implementation.

@CMarastoni

It is the expected behavior to throw an exception when trying to access the port on a custom endpoint. The reason is that normally the port is stored at some specific offset inside the endpoint data. If the endpoint is IPv4 or IPv6, then we know what the layout of the data is and know that you’ll be accessing the port. But on custom endpoints, the data at that offset might well be garbage.

With that said IsLoopback should not be accessing the port for custom endpoints. That’s a bug. I’ll get that fixed for the next release of the transport package. Thanks for pointing it out!

For the listen address, I agree that this should be handled in Netcode for Entities. There seems to be an underlying assumption here that all drivers can listen on the same kinds of endpoints (IP address and a port) and I’m not sure what’s the best way to address this more generally. I’ll open a conversation with that team on this topic.

2 Likes

Until support of custom endpoints gets sorted out with the Netcode for Entities team, the size increase of the NetworkEndpoint structure does open up a (very ugly and hacky) workaround. Basically, you could keep using IPv4 endpoints that satisfy Netcode for Entities, but stuff your interface-specific information in the empty space now available after the IP data.

The structure we use to store IP addresses is 24 bytes long, which leaves 36 bytes unused when the family is IPv4 or IPv6. Whatever is in those 36 bytes should survive all the way to the network interface, leaving all layers above (netcode and transport) thinking that they’re dealing with a normal IP endpoint.

You’ll just have to be a bit crafty to get access to these bytes:

var endpoint = NetworkEndpoint.LoopbackIpv4.WithPort(4242);

var ptr = (byte*)UnsafeUtility.AddressOf(ref endpoint);
UnsafeUtility.MemCpy(ptr + 24, yourCustomData, yourCustomDataSize);

(Note that I haven’t tested this workaround.)

2 Likes

Easiest workaround for me is to comment out the check. I need to localise the package anyway as the MTU constant is still being referenced in netcode so I need to change that, and I can leave all my code alone for when it’s finally working as expected.

Edit: Actually netcode MTU thing might be fine.

Had a quick discussion with the Netcode team about this, and their plan is to give public access to the NetworkDriverStore backing NetworkStreamDriver (along with other changes to make creating custom network drivers easier). This would allow calling Bind and Listen individually on each driver, with different endpoints for different drivers if necessary. It would also allow calling Bind before Connect for clients, although that’s less of a necessity with the changes in Transport 2.2.1.

Unfortunately I don’t know yet when these changes will be released in a new version of Netcode for Entities. It is actively being worked on however. Thank you again for all your feedback on this matter!

2 Likes