[1.0.0-exp.13] Failed to reconnect

[optimise]

Previously at 0.51 I can reconnect game and it’s work nicely but it’s no longer working properly anymore at 1.0 and I get the following error when reconnect game.

InvalidOperationException: Could not find ghost type in the collection
Unity.NetCode.PredictedGhostSpawnSystem+InitGhostJob.Execute (Unity.Entities.ArchetypeChunk& chunk, System.Int32 unfilteredChunkIndex, System.Boolean useEnabledMask, Unity.Burst.Intrinsics.v128& chunkEnabledMask) (at C:/xxx/Library/PackageCache/com.unity.netcode@1.0.0-exp.13/Runtime/Snapshot/PredictedGhostSpawnSystem.cs:197)
Unity.NetCode.PredictedGhostSpawnSystem+InitGhostJob.Unity.Entities.IJobChunk.Execute (Unity.Entities.ArchetypeChunk& chunk, System.Int32 unfilteredChunkIndex, System.Boolean useEnabledMask, Unity.Burst.Intrinsics.v128& chunkEnabledMask) (at :0)
Unity.Entities.JobChunkExtensions+JobChunkProducer1[T].ExecuteInternal (Unity.Entities.JobChunkExtensions+JobChunkWrapper1[T]& jobWrapper, System.IntPtr bufferRangePatchData, Unity.Jobs.LowLevel.Unsafe.JobRanges& ranges, System.Int32 jobIndex) (at C:/xxx/Library/PackageCache/com.unity.entities@1.0.0-exp.12/Unity.Entities/IJobChunk.cs:409)
Unity.Entities.JobChunkExtensions+JobChunkProducer1[T].Execute (Unity.Entities.JobChunkExtensions+JobChunkWrapper1[T]& jobWrapper, System.IntPtr additionalPtr, System.IntPtr bufferRangePatchData, Unity.Jobs.LowLevel.Unsafe.JobRanges& ranges, System.Int32 jobIndex) (at C:/xxx/Library/PackageCache/com.unity.entities@1.0.0-exp.12/Unity.Entities/IJobChunk.cs:357)

[CMarastoni]

The fact the PredictedGhostSpawnSystem is triggering the error is interesting. Are you predictively spawning entities on the client? (in general)
I will try on some of our sample, but it would be great to understand your logic a bit more.

[optimise]

@CMarastoni

The fact the PredictedGhostSpawnSystem is triggering the error is interesting. Are you predictively spawning entities on the client? (in general)
I will try on some of our sample, but it would be great to understand your logic a bit more.

I think I predictively spawning entities on the client only for skill. Then all the rest of it should be no.

[CMarastoni]

In 1.0 we changed quite some stuff in regard predicted spawning. Probably the biggest change in that sense is that, by default, the entity referenced by a spawner can be used to spawn the entity on both client and server, without need to add the PredictedGhostSpawnRequestComponent.

Now, the problem that I can see, is that if you spawn the player or the skill, before the GhostCollection has processed these prefabs (their GhostComponentType.GhostType has been assigned), we will not find then in the GhostCollection the entry relative to this prefab (and this is why it is asserting).

Now the question is: what it is the difference in the game flow when you re-connect? Or better: how do you connect/disconnect?

[optimise]

@CMarastoni

Now the question is: what it is the difference in the game flow when you re-connect? Or better: how do you connect/disconnect?

The flow is just the same using same code path that uses NetworkStreamDriver to connect and re-connect by passing ip address and approach is identical to 0.51.

@CMarastoni

Now, the problem that I can see, is that if you spawn the player or the skill, before the GhostCollection has processed these prefabs (their GhostComponentType.GhostType has been assigned), we will not find then in the GhostCollection the entry relative to this prefab (and this is why it is asserting).

From what I see when re-connect game, 1.0 should make it works just like 0.51 that 1.0 will just dynamically figure out itself and restore entire game state and spawn all the required ghosts at client. Game developer dun need to do a lot of tedious work just to make re-connect game works. Currently is there any way I can do at my side to wait for GhostCollection finish its processing before I spawn any ghost? Btw does this GhostCollection processing only process predicted ghost?

[optimise]

@CMarastoni What is the current status now? Quite urgent to get a working reconnect hotfix.

[larus]

Hey, @CMarastoni is now on vacation unfortunately and I don’t know the status of this issue on his side. I’ve tried to reproduce this error with our samples by just hosting on a player and then connecting/disconnecting with another and can’t see it. Seems like the predict spawn ghost type is not found this the ghost collection and it’s hard to say exactly why, could be if the subscene data with the reference to the ghost prefab wasn’t loaded yet or something similar but some new issue must have appeared if the exact flow for disconnect/reconnect worked fine for you before (and it’s possible like said above the predict spawn routine did get updated in 1.0). Could be related to world state as well, like do you destroy the client world when disconnecting and then go through the same flow when reconnecting as the initial connection did? Is it possible to reproduce with our samples or could there be something else I should try with our samples to replicate your steps better?

[optimise]

@larus

Could be related to world state as well, like do you destroy the client world when disconnecting and then go through the same flow when reconnecting as the initial connection did? Is it possible to reproduce with our samples or could there be something else I should try with our samples to replicate your steps better?

I didn’t use destroy client world. Basically both back to game main menu to reconnect and kill game then reopen game to reconnect will fail to reconnect. At editor I also can reproduce the issue. I’m not sure how to reproduce it with official samples since this error comes from my production project.

[CMarastoni]

Hey,
So… I’m technically in vacation. But since this problem is quite important to be understood, I spent some time to dig into it.

The result, as I expected, is mostly game specific. However, the inner reasons are generic enough.

Following a big and long post. This is going to describe in details how things work and what you ( @optimise ) should look at.
At the same time, this is important to everyone to understand some important details.

Here we go:

in the project I looked into, after a client connect, the game scene is loaded and then the client communicate to the server that is ready to and the connection is set to be in game.
Pretty standard setup.

Then… What happen: both server and the client start processing ghost prefabs loaded from the game scene, agreed upon the loaded data and then the server stream ghost to client. This at a very high level.

The GhostCollectionSystem is responsible, for both client and server, to process and setup the ghost prefabs and create at runtime the information necessary to serialise the ghost. In particular two buffers are populated:

• The GhostCollectionPrefabSerializer buffer. This buffer contains the serialization information for each prefab. See the component documentation for further info.
• The GhostCollectionComponentIndex, that contains for each serialised component some metadata.

An hash is calculated and assigned to the processed ghost, and it is used to verify that client and server agree on how the ghost is serialised.

The server will process the prefabs as soon as there is a connection that is game. It will add the currently used ghost types to the GhostCollectionPrefab and setup a serialiser for the ghost.
The server will send to the “in-game”, clients, as part of the ghost snapshot, the GhostCollectionPrefab list. Until the list (or a part of it) is not “acked” by the client, no ghost are sent.

If the list is long, the list is sent in multiple frames (up to 32 ghosts per tick IIRC). The list is sent sorted by ghost hash (THIS IS VERY IMPORTANT).

The client should load the same ghosts the server and ack the ones he has loaded (at any point in time). This information is sent as part of the command stream.
As soon as the client receive a GhostCollectionPrefab list update, it will complete the prefab processing, verify that the hash match, and reply to the server how many ghosts from the begin of the list he has loaded. This information is reflected on the server by the SnapshotAckComponent.NumLoadedPrefabs field.

The server uses the number of loaded prefabs to determine the subset of ghosts that he can stream to the client (the one he is able to deerialise).

Because the list is sorted, depending when and how the data is loaded, the client may have only some of the required ghosts present in the list. For example:

The server has loaded 1,2,3,4,5,6,7. At the moment the client receives the list, he has loaded 1,2,3,6,7.

The client will report to the server that is has currently have loaded 3 prefabs (1,2 and 3). This even though it has loaded also 6 and 7. When the client load more ghosts, i.e 4,5, it will report the server tha the num of loaded prefabs is 7 (or any other partials, depending on the case).

In such a case, an hole or a partial number of ghosts are availble, the client should inform the GhostCollectionSystem that the required resource (the ghost prefab) is going to the be loaded soon.

To do that, the client should set the missing entries the GhostCollectinPrefab buffer as loading, by setting the LoadingState to LoadingActive.
The GhostCollectionSystem will not further check or validates other ghosts until the missing resource (in the list order) are loaded (the entry has a prefab entity associated to it).

The client must continue to set the LoadingState to LoadingActive as long as the prefab is not loaded or its state is set to LoadingNotActive.

Failing to do so, will result in a validation error. The GhostCollectionSystem will trigger an error “The ghost collection contains a ghost which does not have a valid prefab on the client” and disconnect the client.

NOTE: This logic is completely missing in the game I checked. And in at least one case I saw an holes in the list (I can give more info privately). If you are loading multiple scenes, at the same time, that may happen depending on the loading order.

The game also have something like 201 ghost types. The first ones in the list are usually player characters, the remaining, are ability and other things. The ability, for what I saw, are ghosts.And in particular they are predictively spawned".

The problem that @optimise isseeing the in is due to the predictively spawned abilities, in conjunction with the the long list of prefabs.

The server spawn the player character and start streaming the ghost data. As soon as the hero is available on the client, when the game re-connect an ability is triggered. And because they use predicted spawning, the client will create a new entity for it. So far so good.

However, because of that, the ghost will processed by the PredictedGhostSpawnSystem, that will try to setup and initialize the new ghost (client only). And this REQUIRES that the ghost type (the prefab) has been already processed by the GhostCollectionSystem. The GhostCollectionPrefabSerializer must contain an entry for that ghost.

If the list of prefab is long enough, and the ability trigger before the ghost has been processed, then the entry may be not present.

This is exactly the error you are seeing: “Could not find ghost type in the collection”. That does not tell you all that (sorry about it, we will add more verbose info).

How to solve this?

Solving the prefab loading issue

If you are experiencing the problem that some prefabs are not available when the client start receiving the ghost type list (and so the GhostCollectionSystem trigger error and disconnect the client) I would suggest to add a system that:

• MUST run before the GhostCollectionSystem
• set the LoadingState to LoadingActive for all the prefab in the GhostCollectionPrefab buffer.

That system should continue to set the loading state as active as long as the the GhostCollectionPrefab.prefab field not assigned (different than Entity.Null).

Solving the predicted spawn problem.

This issue is a little more tricky, depending on how we would like to deal with it.

Ideally, on the client side a ghost should be predictively spawned only if the prefab has been alredy processed and a serialiser has been setup for it.
Unfortunately, we don’t add tags or mark the prefab in any way, so discnerning that on the application side become harder than it should be.

So… what you can do?
A robust solution, in general, involve adding some modification to the package. I have two hot-fix/patch idea here,that aren’t ideal but they are fast to implement.

1. Add to the prefab a tag or use a set/dictionary to track which prefab has been already processed by GhostCollectionSystem. That involve modifying the GhostCollectionSystem.ProcessGhostPrefab method a little bit (very very simple)
2. Modify the InitGhostJob inside the PredictedGhostSpawnSystem and insted of triggering an exception, just return (wait for the serialisation data to be ready). This is definitiively the easiest. However, there is some downside: The ghost is spawned, so it get simulated, but it may requires some time before the real instance is sent from the server (some round trip). That may be ok, depending on the situation.

I definitively tried the second one (single line fix) and it work. Then there was an issue trying disconnecting again (a missing singleton in your game) but that seemed to me completely unrelated.

We should definively provide a more robust solution for this problem in general, but at the moment we are speaking these are some workaround.

(my apologies for the insane long post)

[kirkgamesllc]

NOTE: This logic is completely missing in the game I checked. And in at least one case I saw an holes in the list (I can give more info privately). If you are loading multiple scenes, at the same time, that may happen depending on the loading order

Is this still true? Does the developer have to maintain these lists by setting the GhostCollectinPrefab LoadingState to LoadingActive, or is this now being handled by Netcode for Entities?

[diakou]

@CMarastoni

NOTE: This logic is completely missing in the game I checked. And in at least one case I saw an holes in the list (I can give more info privately). If you are loading multiple scenes, at the same time, that may happen depending on the loading order.

@kirkgamesllc

NOTE: This logic is completely missing in the game I checked. And in at least one case I saw an holes in the list (I can give more info privately). If you are loading multiple scenes, at the same time, that may happen depending on the loading order

Is this still true? Does the developer have to maintain these lists by setting the GhostCollectinPrefab LoadingState to LoadingActive, or is this now being handled by Netcode for Entities?

I would like to know this as well! Couldn’t really find info on whether or not this is the case or if it’s been addressed since then.

[CMarastoni]

Is this still true? Does the developer have to maintain these lists by setting the GhostCollectinPrefab LoadingState to LoadingActive, or is this now being handled by Netcode for Entities?

We have no clues where and how these resources are mapped (at least by using as simple sub-scene flow) and as such is up to the dev do that.
But the logic is pretty simple to implement, depending on how you do things in your game. I have ready some sample that show how to do it.

[kirkgamesllc]

@CMarastoni

We have no clues where and how these resources are mapped (at least by using as simple sub-scene flow) and as such is up to the dev do that.
But the logic is pretty simple to implement, depending on how you do things in your game. I have ready some sample that show how to do it.

Thank you for the response.

I think this answered two questions for me:
Is dynamic loading of SubScenes possible? Yes
How would I go about pausing snapshots until all of the ghost prefab hashes have been sent to the clients? set the LoadingState to LoadingActive

I was under the impression that the entire ghost prefab ‘database’ would be sent to the client(s) upon connection to the server.
From what you’re saying, the server only sends ghost prefab hashes ‘as needed’ dynamically, I’m assuming for Ghost Prefabs in the scene, and prespawned ghosts.

Q) If I had a SubScene that contains all of my game’s ghost prefabs and it loads first on client and server before connection, then if I later dynamically load a SubScene that has ghosts in it, does the server resend all of those ghost prefab hashes in the newly loaded scene ‘again’ to the client, or does it already know that the client has them?

Q) If I don’t have a SubScene that has all of the ghost prefabs represented, the server will only send the prefabs as-needed for a scene that loads that has new ghost prefabs it’s using?

[kirkgamesllc]

@CMarastoni

We have no clues where and how these resources are mapped (at least by using as simple sub-scene flow) and as such is up to the dev do that.
But the logic is pretty simple to implement, depending on how you do things in your game. I have ready some sample that show how to do it.

Please may I see the sample code?

I added this to my existing Ghost Classification System

if (ghostPrefabTypes[i].GhostPrefab == Entity.Null && ghostPrefabTypes[i].Loading == GhostCollectionPrefab.LoadingState.LoadingNotActive)
{
       var prefab = ghostPrefabTypes[i];

       prefab.Loading = GhostCollectionPrefab.LoadingState.LoadingActive;
       ghostPrefabTypes[i] = prefab;
}

[diakou]

@CMarastoni

We have no clues where and how these resources are mapped (at least by using as simple sub-scene flow) and as such is up to the dev do that.
But the logic is pretty simple to implement, depending on how you do things in your game. I have ready some sample that show how to do it.

Ah gotcha! That clarifies things, would love to see the sample if it’s out there!