GhostField Entity Field Not Synchronizing Consistently (with lots of entities)

Unity Engine
, , , ,
1

I’ve been working with Netcode for Entities and recently encountered a potential issue. I’m unsure whether this is a bug or if I’m overlooking something fundamental.

Setup

I am spawning ghost entities that reference other ghost entities. Specifically, my setup consists of:

  • Character Entity (Ghosted prefab, Interpolated, Static, Importance: 5)
  • CharacterManager Entity (Ghosted prefab, Interpolated, Static, Importance: 1)

Both are separate prefabs (CharacterManager prefab is not a child of the character, the reason being that some characters will be dynamic, others static, so it is separated into a new ghost prefab to ensure the static-ness of this entity as the components inside it don’t update that often)

Each Character Entity contains a EntityManagerLink component that references its associated CharacterManager Entity:

public struct EntityManagerLink: IComponentData
{
    [GhostField]
    public Entity Entity; //During runtime in a baking system, I assign this to be the CharacterManager Prefab entity, I also tried a different setup, more info below
}

Entities are spawned at the start of the session in a Host configuration (server + client). The client connects only after all required prefabs are loaded.

Spawning Process

Entities are spawned in a loop as follows:

var characterPrefab = SystemAPI.GetSingleton<CharacterPrefab>();
int entitiesPerAxis = 10;

for (int x = 0; x < entitiesPerAxis; x++)
{
    for (int y = 0; y < entitiesPerAxis; y++)
    {
        state.EntityManager.InstantiateEntity(characterPrefab, new float3(x, 0, y) / 1.5f, quaternion.identity, 1, "Character");
    }
}

(Note: My InstantiateEntity is just an extension that sets position, rotation, scale, nothing special.)

Each Character Entity initially has an InitTag component, which is removed at the end of the frame. While InitTag is enabled, the corresponding CharacterManager Entity is instantiated, and the EntityManagerLink component is assigned accordingly to the Character Entity:

foreach (var (_, entityManagerLink, e) in SystemAPI.Query<EnabledRefRO<InitTag>, RefRW<EntityManagerLink>>().WithEntityAccess())
{
    var characterManagerEntity = state.EntityManager.InstantiateEntity(entityManagerLink.ValueRO.Value); // Instantiate CharacterManager Entity which by default is a prefab entity
    entityManagerLink.ValueRW.Value = characterManagerEntity ; // assign our instantiated EntityManagerLink Entity (GhostField)
}

Observed Issue

When spawning a small number of entities, everything works as expected. However, when spawning hundreds of entities, I notice that some EntityManagerLink components on the client remain on their default unassigned value (Entity.Null) indefinitely, while others have their EntityManagerLink field properly synced.

Upon inspecting the server side, all entities have their EntityManagerLink field properly set. I understand that GhostField data may take a few ticks to propagate to the client, but in my case, some fields never synced to the client.

Debugging Attempts

Network Debugger
No apparent bandwidth issues.

Importance Factor Adjustment:
Changing the importance of the Character Entity changes the success rate of EntityManagerLink GhostField syncing. For testing I would wait for all ghost entities to appear on the client and then look at the Network Debugging to make sure there isn’t any more data being transmitted, after which I would dump results (3 tests per importance value change, spawning 900 character entities (which would chain spawn an additional 900 CharacterManager Entities and link them via the EntityManagerLink component):

(Character Entity Importance: 1, CharacterManager Entity Importance: 1) results:

  • [Client] ‘28’ unassigned GhostField Entity values out of ‘900’
  • [Client] ‘31’ unassigned GhostField Entity values out of ‘900’
  • [Client] ‘20’ unassigned GhostField Entity values out of ‘900’

(Character Entity Importance: 1, CharacterManager Entity Importance: 5) results:

  • [Client] ‘676’ unassigned GhostField Entity out of ‘900’
  • [Client] ‘765’ unassigned GhostField Entity out of ‘900’
  • [Client] ‘873’ unassigned GhostField Entity out of ‘900’

(Character Entity Importance: 25, CharacterManager Entity Importance: 25) results:

  • [Client] ‘887’ unassigned GhostField Entity out of ‘900’
  • [Client] ‘900’ unassigned GhostField Entity out of ‘900’
  • [Client] ‘900’ unassigned GhostField Entity out of ‘900’

We can clearly see a pattern in the success rate of the GhostField being assigned and the importance factor. On a side note my client indeed sees all 900 ‘CharacterManager’ Entities.

Using GhostGroup
This resulted in a 100% success rate of the GhostField getting synced, however when I look in the Network Debugger, these static entities are sending continuous data every tick, also upon looking at them in the inspector, I can see the snapshot index change as if it were a dynamic entity. (in my scenario I don’t need these to sync in the same snapshot, so checking GhostGroup was just for testing sake)

Edit 1:

Built Application
I tried running two application instances, the first in host mode, wait a couple of seconds for everything to spawn, connect a second client, the same issue would still occur (on both host client and the additional client).

Edit 2:

Changing the Character Entity to be Dynamic (instead of Static)
This also resulted in a 100% success rate in the GhostField syncing

Changed EntityManagerLink component

public struct EntityManagerLink: IComponentData
{
    [GhostField]
    public Entity Entity;    

    [GhostField (SendData = false)]
    public Entity Prefab; 
}

This was more of a sanity check to see if anything different would happen, I would spawn the entity instance from the Prefab field instead and then assign it to the GhostField, which resulted in no change.

Questions

  1. What could cause some GhostField entity values to never synchronize, even though they are correctly assigned on the server?
  2. Is there a limitation with ghosted static entities that I am unaware of?
  3. Could this be a mapping issue, where the client receives an entity reference but doesn’t know which entity to map it to?
  4. Why would using ghost groups result in 100% success rate of the GhostField syncing and why do they send continuous data even if both prefabs are set to be static

Thanks in advance, and I apologize for the long post! My project is quite a bit more complex; otherwise, I would have provided a sample. I wanted to ask first in case I missed something obvious before creating one.

Any insights or suggestions would be greatly appreciated!

Curently using:
Netcode for Entities 1.4.0
Entities 1.3.9
Unity Editor version 6000.0.37f1

2

As far as I know, Ghost Fields, unlike RPC requests, are transmitted over UDP, so apparently Netcode is simply not able to synchronize a sufficiently large number of entities. It might be worth switching to an RPC request.

3

Both reliable and unreliable communication in com.unity.transport is implemented on top of UDP [in standalone builds]. TCP isn’t great for real-time games due to latency issues in case of packet loss.

4

The problem with RPC in this case is for instance if a player disconnects and reconnects, I would have to figure out all these unset fields which introduces unnecessary code. RPC seems more suited for one shot events that don’t need resyncing if a player disconnects and reconnects. I have done more testing and this sync failure only seems to happen with static entities (Edit 3 in the post), perhaps this is why “GhostGroup” worked too as it seemed like it made my static entity behave more “dynamic” in regards of snapshot index updating continuously. In general GhostField’s seem way more powerful than RPC’s in Netcode for Entites due to things like Relevancy / Importance and the GhostComponent attribute which allows very niche filtering without any extra code. In general though the docs for ghosts states “eventual consistency” which appears to not be true for static entities (when the number is high enough).

5

Exactly this! This is unfortunately a known issue with Netcode for Entities snapshot replication of Entity structs, and it’s caused by the following:

  1. Netcode for Entities uses snapshot synchronization (via its own snapshot acking form of reliability) built on top of UDP.
  2. Netcode for Entities can only replicate entities that both worlds (i.e. Client & Server) know about. In other words; only GhostInstances can be replicated (as we literally send the GhostInstance.ghostId value in the snapshot, then use a HashMap<GhostId,Entity> when resolving an Entity field). Also note: We don’t currently support Entity references to ghost’s children, as again, there is no GhostId to map them to.
  3. Thus, this creates a problem: What happens if I try to replicate an Entity struct pointing to a ghost that this client has not yet received (or won’t ever receive, due to relevancy)? We cannot resolve that ghostId the moment deserialization occurs, and so it is left as Entity.Null. This is the problem you’re experiencing.

Sending the Entity via RPC therefore will not work either, as the RPC receive code will have the same problem (the client cannot resolve the ghostId to an Entity we have not yet received).

The very purpose of GhostGroup is to ensure that two (or more) ghost instances are always replicated together. I.e. Within the same snapshot. Thus, if you support GhostGroup on the ghost entity containing the EntityManagerLink, and add said linked entity to this GhostGroup - you can never encounter the above problem by definition.

We’re thinking about how to improve the UX of this (e.g. attempt to re-resolve the ghostId once the linked entity has been independently replicated), but unfortunately nothing concrete yet.

Unfortunately this is due to yet another nuance: FYI that Ghost prefabs supporting GhostGroup cannot currently be static-optimized (as we would have to pay the cost of traversing each group elements chunk when working out if we can early out). This (by-design but hidden) restriction is also true of ghosts containing replicated child components. I have a PR open that’ll improve the awareness & debuggability of this.

6

I saw this randomly when came back to fill the survey and it seems very good info to add to the documentation.

7

Updating documentation would definitely help, as I initially expected the client to automatically resolve the Entity reference due to reading about “eventual consistency.” Also when hovering over the Ghost Group in the user inspector on the prefab, I would expect it to indicate that it turns the entity dynamic (perhaps it can automatically do this in the inspector).

From my understanding after carefully reading your comment:

  • Replicating an Entity as a GhostField in components on statically optimized ghosts is a bad idea because the entity reference can be received after the component data, resulting in a null reference on the client.
  • The client does not automatically resolve this issue.
  • This is not directly related to entity count but happens more frequently as entity count increases due to the order of receiving data on the client .
  • The only way to properly resolve this is using Ghost Groups, but defeats the purpose of splitting entities (to separate static data from dynamic data)

It should be common practice (correct me if I’m wrong) to split a complex entity, such as a character, into ghosted dynamic and static entities. Each would have its own systems and occasionally need to access each other’s components via indirect entity lookups.

For example, consider a character’s health stat:

  • The character itself is a dynamic entity.
  • The health stat is a static entity.

Imagine we want to display a visual effect on the character when their health changes (e.g., changing the material). The health stat entity would detect a change, then need to signal the character entity via an indirect lookup (through a ghosted entity field).

Making the health stat dynamic might work in this case, but it doesn’t scale well. Consider world entities that should be 100% static unless a player interacts with them. Relevancy can hide distant ones, but what if they are close together? There are many cases like this where separating static and dynamic entities is necessary (while maintaining a link via a ghosted entity field).

Resolving the Entity Reference

You’ve mentioned a HashMap<GhostId, Entity> on the client that tracks ghosted entities. However, I’m struggling to find more information in the docs.

Is there an accessible singleton for this so we can manually look it up?

My initial approach is to ghost the ghost instance (GhostId) instead of the Entity:

public struct EntityManagerLink : IComponentData
{
    [GhostField]
    public int GhostId;    

    [GhostField(SendData = false)]
    public Entity GhostEntity;
}

A client-side system could then look up GhostId in the HashMap<GhostId, Entity> map and update GhostEntity when it is received.

To avoid unnecessary queries after resolving the entity, I could use an IEnableableComponent to disable it afterwards.

Another approach would be to use SystemAPI.Query to find all ghost instances and apply the same logic.

Would this be a viable (temporary) solution?

8

Yeah, we see customers choosing this option fairly often. Ideally we would like to be able to enable static-optimization per-component, but that’s a relatively huge change to make to the serializer.

It’s the SpawnedGhostEntityMap singleton’s .Value field.

Good call - yeah that should work, but note: My previous comment was actually a slight simplification. You need to replicate a [GhostField] public Unity.NetCode.SpawnedGhost Ghost; rather than just the int, as the GhostInstance is actually 2 ints (ghostId & spawnTick).

9
  • Replicating an Entity as a GhostField in components on statically optimized ghosts is a bad idea because the entity reference can be received after the component data, resulting in a null reference on the client.

Yes, this is a sounding but also bad idea. We do resolve the entity later, if the static ghost is at least update once after the other entity has been received.

  • The only way to properly resolve this is using Ghost Groups, but defeats the purpose of splitting entities (to separate static data from dynamic data)

GhostGroup are not a solution for this, even for static.

Yes, this is what you should and normally is done by other project. Instead of replicating the Entity itself, you replicated the GhostId + Tick and check against the SpawnedGhostMap on the client.

There are caveats there has well. You need to put some guard or timer to avoid waiting forever for a ghost that may not arrive ever.

Making the health stat dynamic might work in this case, but it doesn’t scale well. Consider world entities that should be 100% static unless a player interacts with them. Relevancy can hide distant ones, but what if they are close together? There are many cases like this where separating static and dynamic entities is necessary (while maintaining a link via a ghosted entity field).

It may looks like a win, but in reality is kinda of a loss. If the heal does not change it cost 0 bits (well let’s say 1 bit because of the change mask).
If you have many of these, you can still spliot them in many groups and use 1 bit per group of changed variables. And still get some decent compression out of it (max 4 bit per unchanged field).
So, for simple stuff like Healt, without the additional complexity of the lookup yo can still achieve good results.
Different story if you have way more complex data structure to sync. In that case, unfortunately at the moment, that would cost a little too much bandwidth to resend if an entity become again relevant (i.e you can presume that that entity data can stay constant).
In that case I can understand the use of static ghosts for that purpose.

However, in general. For this kind of setup I would use relevancy at my own advantage: First I would force syncing all the static stuff. And then, after the client has received all that entries, then I would start syncing the dynamic ones.