Distributed Authority and shared network object pools

[Wataghwan]

I am attempting to build a small-scale semi-competitive MOBA style multiplayer game utilizing the new Distributed Authority topology. The reason I’m using distributed authority instead of a dedicated server is to keep costs down, speed up development time, increase scalability, ease of host migration and so I can tailor the experience so that no one player has a particular advantage over the other.

I am introducing object pooling to increase the performance of the game, and i want the items in these pools to be syncronized and distributed over the network so that any player can own any object in the pool and utilize it accordingly.

For example, each team has a pool of “minions” that gets spawned and navigates towards the enemy base. I want each one of these minions’ ownership to be distributable so that any player can manage them.

The problem I am running into is while the objects are disabled in these pools, it seems as if the ownership of them and any objects related to them (ie the minions projectiles) cannot be updated as netcode for game objects does not support disabled game objects.

What is the best way to manage this situation? I attempted to utilize the “NetworkHide/NetworkShow” extensions on the network object, however I believe these have been designed for a server authoritative topology as even with “spawn with observers” is set to false the client that spawned it is still assigned as an observer. If i then proceed to call “NetworkHide” right after spawning, it removes it from the scene.

The only solution i can think of is to always have the network objects enabled in the scene, but set their position to some arbitrary amount so they’re not visible in the game area and disable all the relevant components other than the network object component.

Is there an alternative that i am unaware of?

Thanks

[CodeSmile]

Do you follow the official Object Pooling strategy?

You can’t use a run-of-the-mill object pooling system with NGO since there are special requirements when acquiring and releasing objects from / to the pool.

@Wataghwan

For example, each team has a pool of “minions” that gets spawned and navigates towards the enemy base. I want each one of these minions’ ownership to be distributable so that any player can manage them.

I think you’re taking a wrong turn, like so many others, thinking that the object is or must contain your data.

In its simplest form, each player has a count of “acquirable minions”, say one player has 10 the other has 15. Every time a player spawns one of these minions, it deducts that number and then gets am object from the minions pool and that’s when you set ownership but also any other data that the object may require. For instance, you could assign a different mesh to the minion if you have different visual styles or types.

What you do NOT need is to have a precise 10 minion objects in the pool for the current player and another 15 pooled minions for the other player nor need they be owned by any player while they are in the pool. Unless by “pool” you mean something other than object pooling, say a visualized cage that every player carries around.

[Wataghwan]

thanks for the speedy response.

The problem that I am encountering is that these minions are only being spawned by a singular player, like a traditional MOBA would spawn the generic AI army every x seconds, however i want every player to have a reference to the same list of network objects so if the player that manages spawning them leaves then the responsibility of spawning them transitions smoothly to another player and they use the same list of network objects. There will only ever be a maximum of 10 minions active for each team.

Another issue is because i want all the items in this pool to have distributable authority, so if i have 10 items in this list and 5 players in the game, each player will own 2 of these objects each. If I have to assign ownership when grabbing items from this pool, keeping track of the distribution of the objects becomes more difficult and means the functionality of the distributable option becomes meaningless as i’d be managing it all myself.

I can think of many ways of resolving this, but none of them seem very eloquent to me, so I was wondering if there was a solution specifically designed to solve the issue of synchronizing shared object pools in a distributed authority setup.

[CodeSmile]

@Wataghwan

if the player that manages spawning them leaves then the responsibility of spawning them transitions smoothly to another player and they use the same list of network objects.

Can’t you just, when a player leaves, grab these references and change ownership? There’s the SpawnManager which keeps a list of all networked objects, so you’d only need to go over these once and check if their owner ID matches the client ID who just left.

There’s a possibility that the necessary callbacks run “too late” though, meaning the objects may already be destroyed. OTOH you could set the NetworkObject to “don’t destroy with owner” and you could run this part of the code when the user exits gracefully through a “end game” button.

@Wataghwan

synchronizing shared object pools

What exactly are you synchronizing, and how?

I assume something like NetworkList<NetworkObjectReference> or with ulong (the former is recommended, it just wraps the ulong and adds an object getter)?

[Wataghwan]

Thinking about this a little further, the real problem I am finding is leveraging the automatic change of ownership that distributed authority provides when players leave/join.

I have a list of inactive minions assigned to each team, and I want each minion’s ownership in this list to be set to distributable. Each of these minions also have a pool of objects themselves to represent their projectiles. The ownership of these projectiles is set to transferable, as when the minions ownership changes, i can then proceed to change the ownership in “OnGainedOwnership()” of these projectiles to correspond accordingly.

The problem is that when a player joins, it successfully distributes the ownership of the the inactive minions, but I get this warning when i try to change ownership of the inactive projectiles in “OnOwnershipGained”

Minion is disabled! Netcode for GameObjects does not support disabled NetworkBehaviours! The NetworkTransform component was skipped during ownership assignment!

I’m confused as to how it is able to distribute the ownership of the minions automatically when a player joins/leaves, but I can’t change ownership of inactive objects myself.

[NoelStephens_Unity]

I think you might have a misconception on how pools work vs how object distribution works.

NetworkObject pools are simply a pool of objects that, typically for NGO, are associated with an INetworkPrefabInstanceHandler implementation that is registered with the NetworkManager.PrefabHandler. The instances you create for the pool you should view as “blank slates of that specific network prefab type”.
When you first create the instances you would:

• Instantiate an instance
• Set any default values
• Set the instance to be inactive (i.e. SetActive(false)).
• Add the instance to the pool (list, hashset, etc).
• Repeat for (n) instances you want to have “pre-warmed” for each client.

When you want to spawn a network prefab that belongs to a pool, you would:

• Grab an instance from the pool.
• Set any additional values (i.e. position, rotation, etc) for that instance.
• Make the instance active, (i.e. SetActive(true)).
• For distributed authority, the client would then just Spawn the instance.

When any non-owner receives the spawn notification (i.e. CreateObjectMessage), because that network prefab type is registered with the NetworkPrefabHandler, it will invoke the INetworkPrefabInstanceHandler.Instantiate method which should:

• Pull an object from the pool
• Set it active
• Apply the passed in position and rotation
• return the NetworkObject.

That is all the non-owners need to do, as the pooled objects are only there to prevent a client from having to run through the cost of allocation when instantiating. You don’t need to assign inactive/non-spawned objects to a client or team. However, if you want to show that a team has (n) remaining minions they “could” spawn then that is more of a counter that for each minion created (i.e. spawned) the counter goes down.

When the owner despawns the spawned object it should only need to despawn it (with destroy == true) and the prefab handler should invoke the INetworkPrefabInstanceHandler.Destroy method that should:

• Reset any setting specific to a team or the like (i.e. convert it back to a “blank minion”)
• Set it to inactive (i.e. SetActive(false)).
• Return it back to the pool.

As the non-owners receive the despawn message, they will invoke the same INetworkPrefabInstanceHandler.Destroy method which would run through the same script and return each non-owner client’s instance back to the pool.

So, think of it like this:

• When a minion is not spawned then it is represented by “a potential” minion to be spawned (i.e. a count or the like).
  • When a minion is spawned then the count (or the like) is decremented.

You would follow the exact same logic with your projectiles. They don’t need to be assigned until they are spawned, upon being spawned they are then assigned ownership by the minion spawning the projectile (which, by default the owning client will be the client that owns the projectile).

Think of object pools as being used any time the associated network prefab is spawned by any client. Who owns the network prefab depends upon which client spawns the network prefab.

Distributable objects are objects that can be distributed to any client (unless locked)(no matter the team)
Transferrable objects are objects that can be owned by any client (unless locked) (no matter the team).
Request Required objects are objects that require a client to request ownership and the owning client decides whether it will grant ownership (which would be more along the lines of only clients on the same team can be granted ownership).

Does this help?

[Wataghwan]

Thank you for the detailed response.

I think i am fairly familiar with how the pooling works, however this isn’t really what I’m after in the way you’ve mentioned it as the player activating items from the pool won’t necessarily be the owner of the object they are activating from the pool.

I’ve reworked my logic which i think is more aligned with what i want, which is a NetworkList of NetworkObjectRefrences which retains a list of all minions which is shared among all clients. This list is initially populated via a “prewarm” count, and all set to “inactive”.

The specific thing I’m after is being able to distribute the ownership of every item and it’s dependencies (ie projectiles) in the network list among all clients whenever anyone leaves/joins, even if those network objects are “inactive”.

One client is responsible for “activating” objects from this pool, by calling an Rpc(SendTo.Owner) method to the owner of the network object which is about to be activated.

To me, the ability to distribute the ownership of inactive items in a “pool” seems as if it would be quite a common use case in a distributed authority setup, however assigning ownership on deactivated game objects is not supported so the solution I am implementing is to handle the “activity” status of an item in the pool by disabling relevant components rather than the game object itself.

Let me know if that makes sense.

[NoelStephens_Unity]

Ahh…I see.
So, the one thing that you might want to take into consideration is why we don’t allow changing ownership of despawned NetworkObjects:

The reason why you can’t change ownership of despawned NetworkObjects is that the spawn process assigns a NetworkObjectId to the NetworkObject when spawned but when not spawned there is no “associated” NetworkObjectId assigned/tracked for the NetworkObject. The NetworkObjectId tells NGO how to route messages…so if you want to change ownership of a NetworkObject then the message includes the NetworkObjectId of the spawned NetworkObject that is changing its ownership. When a NetworkObject is not spawned, there is no way to know “which instance” you are referring to.

If you use NetworkObjectReference, then you are basically creating a wrapper that serializes the NetworkObjectId. For any client instance trying to deserialize/resolve the NetworkObjectReference it will try to look for the NetworkObjectId within the NetworkSpawnManager.SpawnedObjects table. Which…if the NetworkObject is not spawned then it will not be in that table and the NetworkObjectReference will fail to resolve.

Even if you were to disable Recycle Network Ids in order to assure you aren’t pointing to a re-assigned NetworkObjectId, the NetworkObjectId stored within the NetworkObjectReference will not be “valid” since NetworkObjectReference will not resolve to a spawned object.

There are a couple of paths I could see you taking. The first one would be to have an “inactive” state for the minion and projectiles that doesn’t mean it isn’t spawned but does mean it is spawned but not active and when “inactive” the visuals, colliders, and such are disabled (when “active” they are enabled). However, if you plan on having a bunch of minions and projectiles this could become a large initial synchronization for late joining clients having to spawn a bunch of things that might not be “active” for a period of time.

Another way to handle this would be to create minion and projectile “state” structs (or a single “ObjectState” struct) that implements INetworkSerializable and contains the base information for the object to spawn, team, client it is assigned to (a form of ownership), and things of that nature. Then when you need to spawn a minion or projectile for any given ObjectState you just pull from the pool and spawn it. Having this separation from the actual spawned object and the “Object State” would allow you to handle the assignment of “ownership” for minion(x) and projectile(y) without having to try and assign a despawned object (not recommended).

The “ObjectState” could include a unique identifier you assign to each ObjectState instance which you could then create something like a NetworkVariable<Dictionary<ulong, Dictionary<short, ObjectState>> property that signifies: [ClientId][ObjectId][ObjectState] that would automatically synchronize amongst all clients (or something along those lines) so a client is provided all of the “minions” and “projectiles” assigned to them without having to actually assign an instantiated but despawned instance per minion and projectile.

However, trying to assign a specific instance from a pool to a specific client will be tricky since the INetworkPrefabInstanceHandler.Instantiate method only passes in the client identifier that is the owner of the NetworkObject to pull from the pool…in other words… while you could get the client-owner side to spawn a specific despawned instance, the rest of the non-owner clients would not use that specific instance when they spawn its clone instance locally.

I would recommend thinking about separating the actual pooled object from a data set that represents the “yet to be spawned” network prefab as you shouldn’t really need to worry about “which network prefab instance” you are using to represent a specific minion or minion’s projectile…but…based on what I can tell… you are more concerned about assigning “pre-assigned minion and projectile slots” to a specific client that is on a specific team.

Let me know if any of this helps?

[NoelStephens_Unity]

@Wataghwan

The specific thing I’m after is being able to distribute the ownership of every item and it’s dependencies (ie projectiles) in the network list among all clients whenever anyone leaves/joins, even if those network objects are “inactive”.

Just to make sure I am following.
If you have:

• 4 clients in a session
• Each client has 1 pool of 30 minions
• Each client has 6 minions spawned that they own. (the minions might be firing projectiles)
  • Each client has 6 remaining minions “despawned” left in their respective pools or all clients have consumed 24 of the 30 minion instances from their respective pool.

If one of the clients disconnects and the minions are marked as “distributable” then you would have:

• 3 clients remaining in the session
• Each client has 8 minions spawned that they own. (2 from each of the disconnected client gets distributed to the remaining clients)
  • Each client has 6 remaining minions “despawned” left in their respective pools or all clients have consumed 24 of the 30 minion instances from their respective pool.
• There are (n) projectiles fired by the 6 minions belonging to the disconnected client.
  • How do these projectiles get distributed to the correct minion/client (ownership)?

Are you trying to determine how to associate a “spawned and in-flight” projectile with a specific minion instance in order to assure that the projectile fired by a minion, that was previously owned by the disconnected client, gets associated with the correct minion when it is distributed to one of the 3 remaining clients?

[Wataghwan]

Thanks again for all the detailed replies, it’s definitely been helpful.

I’m pretty sure you have the right idea, the initial hurdle was first distributing all the objects in the pool amoung all the players and depending on whether a player leaves/joins.

So here in this image it might give you a better example of what’s going on.

We have a drone squadron, which is distributble network object and consists of 3 drones, which inherit the ownership from the squadron. Each drone then has x amount of bullets associated with it which also inherits the ownership of the corresponding drone and squadron.

Each one of the drones is spawned every x seconds, and so to optimize the game i preinstatiated all the squads, drones and bullets into a set of network lists of network object references which i am using as my object pool.

The activity of these objects is handled by activating/deactivating components on the object, rather than deactivating the entire object itself. This is so i can keep a script active on the object to handle when the ownership of it changes due to the squadron being distributed (when a client enters/leaves).

Here is a snippet of the code I’ve written of the script which handles this functionality:

    private void Start()
    {
        _droneSquadManager = GetComponent<DroneSquadManager>();
        _droneSquadManager.SetActivityStatus(_droneSquadManager.net_isActive.Value);

        if (!_droneSquadManager.DronesInSquad.Any())
        {
            foreach (var droneSquadReference in _droneSquadManager.net_dronesInSquad)
            {
                if (!droneSquadReference.TryGet(out var droneSquadNetworkObject))
                {
                    Debug.Log($"Drone squad network object not found: {droneSquadReference}");
                    continue;
                }

                _droneSquadManager.DronesInSquad.Add(droneSquadNetworkObject.GetComponent<DroneManager>());
            }
        }
    }

    public override void OnGainedOwnership()
    {
        _droneSquadManager.DronesInSquad.ForEach(x => { x.NetworkObject.ChangeOwnership(OwnerClientId); });
    }
}

Now, whenever a client joins the network, it reads from the list of network objects and creates it’s own local “copy” of the list which I’ve defined as a List. I’ve done this for ease of debugging and the ability to perform Linq expressions on it.

The problem i am facing now as you correctly identified is if a player leaves and bullets that they own are in flight, once they transfer ownership they will stop dead still in the scene. This is due to the new owner having no reference of their velocity. I think this is a relatively simple fix, as i can just store it’s velocity when someone leaves to a network variable and reapply that force on ownership change.

[NoelStephens_Unity]

@Wataghwan

The problem i am facing now as you correctly identified is if a player leaves and bullets that they own are in flight, once they transfer ownership they will stop dead still in the scene. This is due to the new owner having no reference of their velocity. I think this is a relatively simple fix, as i can just store it’s velocity when someone leaves to a network variable and reapply that force on ownership change.

Yeah… as a side note (before I provide some existing ways to do this) we are internally tracking the concept of being able to “associate” NetworkObjects as being dependent upon other NetworkObjects so the idea would be that when NetworkObjects are redistributed if they have other NetworkObject dependencies then they would automatically “group” them during distribution and assign the same new owner to all within that group.

However, since that isn’t available today… you have some options:

Keeping the same owner (optional)

• Within your projectiles, upon being “fired” by a drone/minion you would also assign a NetworkVariable property on the projectile itself that is the drone/minion’s NetworkObject.
  • When redistributed, within the OnOwnershipChanged (or gained) method you could just check to see if the client is the owner of the drone/minion and if not then just change the ownership (owners can assign ownership to other clients).
  • The other alternative (other than changing ownership which will delay the entire process and cause a bunch of network messages), is to just know when the projectile hits the damage is coming from a specific minion…which you can still use the NetworkVariable property set on the projectile that points to the drone/minion so when the projectile hits something you know “who is hitting who”.
    • Distributed authority is meant to keep the ownership/authority “distributed” amongst clients so if you start thinking more along the lines of your scripts assuming (for distributable objects) that any client could potentially have to handle being the authority and keep in mind that just because the client who is the authority might not always be the “correct owner”. Especially for things like projectiles where you might want to know who fired the projectile when applying damage knowing that the projectile might be getting updated by someone other than the owner of the object/entity/drone/minion at that point in time.
    • Of course, this kind of thing is associated with a few edge case scenarios (like a client leaving while its drones/minions have in-flight projectiles) that get handled for a brief period of time and then the new owner of the originating projectile will continue to have ownership over things like projectiles from that point forward…it is just when there is a new client joining or a client leaving that yields a few of these edge case…which we are aware of an are looking into ways to make it less complicated to deal with in future updates.

Keeping the velocity
You can do what you are describing, but you also might take a look at the PhysicsObjectMotion class within the Distributed Authority SocialHub example that shows how to keep physics bodies in motion when distributed to a new owner while still moving around. It derives from the BaseObjectMotionHandler class that, in turn, derives directly from NetworkTransform.

It also provides an example of using the RigidbodyContactEventManager to avoid having to use MonoBehaviour collision events (i.e. OnCollisionEnter). Instead, the RigidbodyContactEventManager handles collisions within a job and invokes a callback for each registered Rigidbody to handle the collision event. It provides a less processor intensive way to handle collisions to keep your over-all frame processing time down.

So…the general idea you are having is correct in regards to keeping a body in motion when it changes ownership (especially if you are using Rigidbodies)…
The PhysicsObjectMotion component handles the assignment of the “last known” velocities within the OnOwnershipChanged method and keeps the NetworkVariables updated within the overridden OnAuthorityPushTransformState method.

Anyway, let me know if any of this helps you with your project’s goals/needs?

[Wataghwan]

Thanks again for the detailed response, this is all fantastic information! I have now resolved the issues i was facing