Recommendations for dealing with a high number of dynamic ghosts

[ts_headfirst]

With a high number of dynamic moving ghosts, a lot of data needs to be transferred each frame.

Simply excluding the transform and simulating movement on the client, will not work, since you need to somehow place the spawned entity in the first place, and also want to synchronize positions with the server regularly.

What are the recommendations for dealing with a high number (thousands) of dynamic moving ghosts?

[NikiWalker]

Hey! Can you give more details about the gameplay associated with these ghosts?

Broadly

• Use Interpolated Ghosts

• Very High Interpolation Time (via ClientTickRate.InterpolationTimeMs). E.g. 300ms.

• Enable extrapolation on the transform positions of these ghosts.

• Heavily quantize the Transform component. Do you need rotation and scale? Every field counts.

• Set these ghosts to be very low importance compared with player ghosts.

• Set the GhostSendSystemData.MinSendImportance to something like 20 (at 60Hz simulation), to reduce send frequency.

• Increase FirstSendImportanceMultiplier so that new ghosts have higher prio.

• I would probably pool and re-use these ghost entities, to reduce Ghost creation / destruction bandwidth overhead…

• Especially if they only live for a short time.

• And especially if they tend to spawn near where they get destroyed.

Specific optimizations

• If these ghosts move with any kind of predictability (e.g. linear or ballistic motion, path following motion (like A* or tower defense pathing), random noise motion) etc, I would attempt to employ dead reckoning.

• You can use a seeded RNG and soft determinism to get pretty accurate and predictable results.

• Can you replicate the logic those ghosts use in their movement calculation? E.g. if these ghosts are following a specific player, only replicate their new target entity ghost and start position + tick, and make them static.

• If their movement is essentially unpredictable, you unfortunately have to just send the data.

• But can you group these clusters of ghosts? E.g. If the gameplay allows a player to shoot a homing missile swarm, you only need 1 ghost. The individual missiles could just be VFX swarming around the actual invisible single homing missile ghost.

• This also works for formations of armies. Have one ghost acting as the formation leader, then replicate other ghosts as relative offsets against the formation. Very good delta-compression due to small deltas.

In 1.2, play around with settings on the NetCodeConfig asset, with the NetDbg browser window open. You can live view the bandwidth consumption changes.

[PolarTron]

@ts_headfirst

With a high number of dynamic moving ghosts, a lot of data needs to be transferred each frame.

Simply excluding the transform and simulating movement on the client, will not work, since you need to somehow place the spawned entity in the first place, and also want to synchronize positions with the server regularly.

What are the recommendations for dealing with a high number (thousands) of dynamic moving ghosts?

Importance scaling and the optimizations available for you is the first step ( https://docs.unity3d.com/Packages/com.unity.netcode@1.0/manual/optimizations.html ). If that doesn’t really work for you then you need to figure out what kind of smoke and mirrors fakery you can get away with. You might not need to actually update all these positions, but instead fake it so that the client can’t really tell the difference.

I posted a reply to a similar thread here: https://discussions.unity.com/t/902954/4

@PolarTron

Use waypoints instead of realtime position updates. A path is a collection of waypoints. Only when the path changes it is synchronized to clients. Instead of 20hz updates you now get a really low update frequency because the path stays the same for many entities over many ticks.

public struct PathingWaypoint : IBufferElementData
{
    [GhostField]
    public float3 Position;
    [GhostField]
    public int ArrivalTick;
}

Position can now be inferred by using spline interpolation logic.

[ts_headfirst]

Thanks for answering, much appreciated.

For the majority of these ghost, their movement is predictable and can be simulated on the client.
They will each seek a player. They will spawn off screen and likely die a good distance from where they spawn.

So far I have tried simply sending the seek target (as a singular byte) and have the ghosts interpolated and static.
However they quickly get massively out of sync with the server, so before digging further into this, I wrote this post to get some pointers :).
I also struggle to find a good solution to control spawn position in the above approach. In the spike I simply had an actual SpawnPosition on the entity, then would then be disabled/removed on the client post spawn.

The ghost have a low (1) importance, but this effectively causes a low perceived frame rate as the ghost positions are only updated every x frame, so they get very jerky movement.
Our solution so far has been to increase the NetworkStreamSnapshotTargetSize to allow all ghost to be sent in a single frame, but of course then we have the issue of sending loads of data.

If we do a mix, i.e. low importance ghost with infrequent sync and client side simulation, isn’t there generally an issue with that? I mean it is bound to get out of sync, how will that be corrected on the next snapshot?

I’ll go through each suggestion, with a few more questions:

@NikiWalker

• Very High Interpolation Time (via ClientTickRate.InterpolationTimeMs). E.g. 300ms

What does this do, i.e. what is the expected outcome of doing so?

@NikiWalker

• Enable extrapolation on the transform positions of these ghosts.

How is this done, is that Smoothing = SmoothingAction.InterpolateAndExtrapolate on the GhostField, so we would create a variant for this? Edit: hmm this is already how the existing variants are configured.
Is this what will smooth out the movement if the ghosts end up only getting synchronized every x frames and are out of sync when the next snapshot arrives?

@NikiWalker

• Heavily quantize the Transform component. Do you need rotation and scale?

Yes, we can likely reduce this to just the position, and a [GhostField(Quantization = 1000)] on that.

@NikiWalker

• Set these ghosts to be very low importance compared with player ghosts.

We have that set to 1 with players set to 10. However on its own, it causes jerky movement (as described above)

@NikiWalker

• Set the GhostSendSystemData.MinSendImportance to something like 20 (at 60Hz simulation), to reduce send frequency.
• Increase FirstSendImportanceMultiplier so that new ghosts have higher prio.

What is the expected outcome here? Won’t this cause even jerkier movement?
As a small sidenote, the FirstSendImportanceMultiplier is missing from the NetCodeConfig asset.

@NikiWalker

• I would probably pool and re-use these ghost entities, to reduce Ghost creation / destruction bandwidth overhead…

I had not considered this, since pooling in ECS isn’t generally recommended.
How much of a benefit is this likely to yield, i.e. how large is the overhead of ghost creation/destruction over the wire?

[NikiWalker]

@ts_headfirst

However they quickly get massively out of sync with the server, so before digging further into this, I wrote this post to get some pointers :).

Yeah this makes sense. Given that they can be static ghosts with this setup, I would have a server system which periodically updates a replicated float3 Position, NetworkTick AtTick,Entity Target component, which your client interpolation system can then manually correct them to (with smoothing of course). It’s critical to send the tick here, so that you know how much to offset from said position when the client finally receives this ghost data (which could be tens of ticks later). You could even have these ghosts appear to “run over to” (i.e. repath to) the new corrected position.

@ts_headfirst

In the spike I simply had an actual SpawnPosition on the entity, then would then be disabled/removed on the client post spawn.

The problem is that the ‘eventual consistency’ network model means that you may not even receive that SpawnPosition enabled value. Just have it be a “Position at tick”, as above, no need to toggle it.

@ts_headfirst

The ghost have a low (1) importance, but this effectively causes a low perceived frame rate as the ghost positions are only updated every x frame, so they get very jerky movement.

They’re updated every render frame, they’ve just run out of data. You need to enable extrapolation (or better: manual simulation). See below.

@ts_headfirst

Our solution so far has been to increase the
NetworkStreamSnapshotTargetSize
to allow all ghost to be sent in a single frame, but of course then we have the issue of sending loads of data.

I’d recommend against this yeah, as this is a last resort kind of thing. It also increases the cost of your ghost serialization.

@ts_headfirst

What does this do, i.e. what is the expected outcome of doing so?

Essentially it adds a buffer time before the playback of interpolated ghosts, which means that said interpolated ghosts will typically have more snapshot data to work with. Your ghosts are stopping because a) they ran out of snapshot data and b) extrapolation was either disabled for them, or clamped. This buffer window improves the former, but at the cost of 300ms of additional delay on interpolated ghosts playback (i.e. you are viewing a version of them that is further in the past).

There should also be a MaxExtrapolationTimeSimTicks variable, which you can increase for your case. This allows extrapolation to extrapolate further into the future.

But note: Given that you can predict the movement of these ghosts, you should not be sending raw Transform position at all. It’s too heavy at your scale. You need smoke and mirrors, as PolarTron suggests. Use the approach above instead, with static ghosts, and client simulation (almost like you’re predicting these ghosts). I’d recommend only updating n chunks of ghosts positions per server tick, which will lead to the best possible replication cost (as the GhostSendSystem only has to add these few chunks into the snapshot, per connected player). Pre-serialization may also help here, but that’s an optimization topic for after this starts working.

@ts_headfirst

How is this done, is that Smoothing = SmoothingAction.InterpolateAndExtrapolate on the GhostField, so we would create a variant for this?

Yes, exactly. See here: https://docs.unity3d.com/Packages/com.unity.netcode@0.50/manual/ghost-types-templates.html#ghost-component-variants

@ts_headfirst

Yes, we can likely reduce this to just the position, and a [GhostField(Quantization = 1000)] on that.

This will be an enormous improvement. If you go with the Transform replication approach, do this (via the variant link above) ASAP. Simply delete the fields you do not need from the variant struct. If you can get away with it, I’d even use a Quantization value of around ~100 (1cm) (or even less).

@ts_headfirst

As a small sidenote, the FirstSendImportanceMultiplier is missing from the NetCodeConfig asset.

Ah yes, apologies, I forgot that we didn’t hook this one up. Fixing in progress. Less important if you pool.

@ts_headfirst

How much of a benefit is this likely to yield, i.e. how large is the overhead of ghost creation/destruction over the wire?

Implementation dependent. I’d recommend viewing the stats in the NetDbg window for this.

[CMarastoni]

for thousands of objects, that does not require continuous update from the server (or at low frequency) the best approach is as usual a mix.

You should probably increment the ClientTickRate.InterpolationTimeNetTicks to something like 5 or 6 ticks (or more) to buffer data long enough such that:

• The interpolated ghost are interpolating data 6 ticks behind the last received server data, effectively giving you more time receive new ghost snapshot.
• You can augment a little bit the MaxExtrapolationTime if necessary to have some interpolated data linearly extrapolated.

Then most depend also on your camera and field of view and what you can achieve with a custom written priority scaling for you ghost. Without that in place there is still to much stuff to send.

If you can use static optimisation (so only the spawn position is needed) and you can change the data only X time per seconds (to provide some sync) this can give decent result. If the path movement is also predictable and expressible with a math curve, only the curvilinear coord (that is a float) can be sent potentially.

If you can exploit formations or substantially some leader/follower approach such that only a few moving point need to be sent, and everything else is in delta in respect that (for example fixed offset, and very occasionally this positions changes) you can also compress tons of unit information in a few bits.

If that can’t be the case, and you need have these ghost seek another one, unless the target has predictable movement you will be never able to get the exact same movement on the client and the server.
However, you should be still be able to get close enough if the send rate of the target player is high (so sent every tick) and your client seeking logic mimic what the server does. Also, please consider the fact the fact you are dealing with interpolated ghosts. That means, you may now want to target the player position at tick X, but the player position at the tick close to the interpolated tick (so you should check your snapshot buffer or keep an history).

[ts_headfirst]

I have now tried various easily implemented suggestions.

• Setting ClientTickRate.InterpolationTimeMs (or InterpolationTimeNetTicks) to 100 (~6 ticks)

• Setting ClientTickRate.MaxExtrapolationTimeSimTicks to 60

• Setting GhostSendSystemData.FirstSendImportanceMultiplier to 5 (randomly chosen)

• Set NetworkStreamSnapshotTargetSize back to normal

While this probably needs tweaking, it runs smoothly with much less data transfer per frame.
An example with ~1100 enemies a single frame looks similar to this:

@CMarastoni

Then most depend also on your camera and field of view and what you can achieve with a custom written priority scaling for you ghost. Without that in place there is still to much stuff to send.

We have looked into relevancy, and will introduce this at some point, but mostly entities will all be on screen.

Regarding the alternate, and as I read it, recommended approach, I am still not a 100% sure I see the full picture correctly, so here a high level break down of what I understand you suggest:

ServerSeekSystem

• We set and send a single byte size id (SeekId) to represent the target to seek. Possibly we also need to include a tick to seek the target where it was and not where it is, on the client.

ServerPositionSyncSystem

• Updates PositionSync { float3 Position, NetworkTick AtTick } for some number of chunks

ClientSeekSystem

• Identify target and seek it.
• Process PositionSync { float3 Position, NetworkTick AtTick } with a change filter and a LastPositionSynced { Tick } to ensure we only process each sync once.
  This calculates a desired position by PositionSync.Position + (InterpolationTick? - PositionSync.AtTick) * SeekVector * Speed * deltaTime.

Given that we are ‘predicting’ based on interpolated data, the tick delta should be small, 0 at constant latency, right?

From this a correction vector is found, i.e. DesiredPos - ActualPos, and we consume this over x frames adjusting the seek vector. If anything remains of the adjustment vector next sync update, we add them and decrease x.

Does this sound something like what you had in mind?

[NikiWalker]

@ts_headfirst

While this probably needs tweaking, it runs smoothly with much less data transfer per frame.
An example with ~1100 enemies a single frame looks similar to this:

Amazing! I’d love to see a before and after with an optimized TransformVariant.

@ts_headfirst

(InterpolationTick? - PositionSync.AtTick) * SeekVector * Speed * deltaTime

This looks correct (with your smoothing comment later).

@ts_headfirst

Given that we are ‘predicting’ based on interpolated data, the tick delta should be small, 0 at constant latency, right?

Just to clarify: It’s not latency which changes this, it’s the quantity of ghosts affecting how often a ghost replicates (i.e. the frequency of replication).

• If you have one ghost, it’ll be added to every snapshot.
• If you have 2k ghosts, only ~100 (example taken out of thin air) would be added to any given snapshot

Thankfully, the ghost is marked as StaticOptimize in this case (or should be!), which means the GhostSendSystem will probably only find this ghost’s chunk in the list of chunks that need to be replicated, so actually changes to a ghost should be replicated very quickly yes.

@ts_headfirst

From this a correction vector is found, i.e. DesiredPos - ActualPos, and we consume this over x frames adjusting the seek vector. If anything remains of the adjustment vector next sync update, we add them and decrease x.

Correct (while also having the seeker constantly moving towards the current position of the target, identically to how the server would).

@ts_headfirst

Possibly we also need to include a tick to seek the target where it was and not where it is, on the client.

EDIT: Yes you’d need this (especially so if your target is predicted). You’d need to use the interpolated value of your predicted ghost at the correct tick… We have public APIs for fetching historic snapshot data for a given ghost + tick, note that “slotIndex” is used incorrectly in this sample code. Finding better answer…). For a first pass, using the latest position probably shouldn’t be too bad.

[NikiWalker]

BTW: Another option (which we haven’t mentioned yet) is to make these enemies be actually client authoritative. I.e. The server has authority over a) spawning, b) destruction, c) who each enemy is targeting, then said target’s client is responsible for actually simulating them. Thus, no corrections for the player most likely to care about said Enemy. But then you need client authority over all interactions with said enemies (like shooting them), so it’s a highly game dependent choice.

You’d also need to replicate this information to the server, and again we don’t have a fleshed out API for doing so yet. This is also on our (ever growing) list. Maybe the server can use dead reckoning here?

[ts_headfirst]

Thanks for the elaborate answer. We will have a go at implementing this soon.
We just need to sort out an issue with our current path finding solution not behaving when running on the client.

With regards to client authoritative, we could totally go that way since it’s pure coop, but I think we will stick to what has engine support, for now at least.

[ts_headfirst]

Is it correct that selection of entities to send over the wire in a given frame is based off of change filters on the ghost components?
So that if one entity has a change, the entire chunk is sent?

If yes, that would mean that, if at all possible, it makes sense to change a component on all entities in a chunk at once, rather than allowing them to be updated individually over several frames.

E.g. if I have a NextSyncTick to control when to update a SyncPosition component, it makes sense to have NextSyncTick be a chunk component as opposed to having each entity have their own (that could then differ within a chunk)?

[NikiWalker]

@ts_headfirst

Is it correct that selection of entities to send over the wire in a given frame is based off of change filters on the ghost components?
So that if one entity has a change, the entire chunk is sent?

Yes exactly. Apologies, it wasn’t clear, but it was my meaning behind this:

@NikiWalker

I’d recommend only updating
n chunks
of ghosts positions per server tick, which will lead to the best possible replication cost (as the GhostSendSystem only has to add these few chunks into the snapshot, per connected player).

[ts_headfirst]

@CMarastoni

so you should check your snapshot buffer or keep an history

I have now tried with keeping a history buffer, but it isn’t exactly precise since it is recorded on the client, i.e. it consists of predicted positions.

So I wanted to check if using the snapshot history would be better, but I am stuck.
There seems to be no documentation on how to extract the data of a component (LocalTransform in this case) for an entity for a specific tick.

SnapshotDataLookupHelper is very much married to ghost spawns, and has no method for extracting data for a specific tick, unless I have missed something?

Could you provide some insight?

[NikiWalker]

It’s the slotIndex optional parameter in the if (snapshotDataLookup.TryGetComponentDataFromSnapshotHistory(ghostType, data, out LocalTransform localTransform, SLOT_INDEX)), but unfortunately it does not map perfectly to the tick (and package changes are needed to get the tick). I would recommend manually keeping a history (or your can attempt to parse positions at different slot indexes into something usable).

[ts_headfirst]

We have now migrated to static ghost with no transform data being sent.
For a simulation of 2000 units (twice the amount of the previous update, the traffic for a given frame looks like this:

the graph looks like so:

For those interested, this is a high level overview of what is involved:

All Enemy movement is simulated on both server and client.

Each Enemy unit is born with a

    public struct PositionSync : IComponentData
    {
        [GhostField(Quantization = 100)]
        public float2 Position;

        [GhostField]
        public NetworkTick Tick;
    }

This gets updated from the server every X frames (currently 1000), controlled by a ChunkComponent.

On the client we record position history in a ring buffer (repurposed DynamicBuffer). This is done for both the Enemy units themselves and their potential targets that they seek.
For enemies, interpolation of missing history entries is done on consumption, since this happens only every 1000 frames.
For targets, it happens when recording. For the local predicted player, this history is recorded with the ServerTick, other players and enemies record with InterpolationTick.

Once a change to PositionSync occurs, the delta between actual position (at that tick) and server position is calculated. This adjustment is then applied over a number of frames (smoothing).

Enemies look up the position of their intended target in the target’s history, offset by a number of frames that is the delta between last target switch frame and the interpolation tick at that time. This is to account for frequency of replication for enemies, i.e. they are not necessarily sent across the wire on the frame where a target change occurs, due to others being in line to be sent before.

Of course the devil is in the detail, and it has taken a fair chunk of work to get this working