Prediction in GameObject Netcode - DevLog Entry 4

We’re back with the 4th entry in our dev logs on the new GameObject layer for Netcode. You can check what we’ve been up to with our first entry here.

Prediction

GhostBehaviour Prediction

Netcode for Entities (N4E) has had prediction for a while, but you had to write ECS systems in order to benefit from it. With the new GameObject layer, we’re exposing a PredictedUpdate method in GhostBehaviours to allow writing prediction code in an object-oriented way.

public partial class MyCharacter : GhostBehaviour
{
    GhostField<int> m_Health;
    GhostComponentRef<MyInput> m_Input; // Design TBD on inputs, we have this working for now, but can do better.

    // This gets called once server side at a fixed tick rate and multiple times per frame on a client, from the latest received
    // snapshot packet's tick to the tip of your predicted tick.
    public override void PredictedUpdate()
    {
        this.m_Health.Value = 123; // This is rolled back automatically
        Move(this.m_Input.Value.MoveDirection); // Inputs are set automatically in the background to the appropriate tick
        // transforms are automatically rolled back
        Instantiate(MyMissilePrefab); // Predicted spawn works the same as instantiating a new GhostObject from the prediction loop. We have checks in place to make sure you don't instantiate from a client outside of prediction (since N4E is server authoritative).
    }
}

Tick Accuracy

Any GhostField you access from this method should be safe to use within the prediction context. It’ll be rolled back and progress to the appropriate tick. This is where you would consume your inputs too (which are also tick accurate). The same applies to transforms. As we’re adding compatibility, more and more unity objects will become tick accurate.
Prediction will behave similarly to the regular Update() and FixedUpdate(), but updating at a different rate (from 0 to n times per frame). For example, if your GhostBehaviour is disabled, PredictedUpdate won’t run. If you specify script execution order, PredictedUpdate will respect it.
Note that this applies to all ghosts, not just player ghosts. Netcode does full world prediction, making interactions between your predicted players and other predicted environment objects tick accurate.

For background, you can see our doc on prediction here for more details on prediction as a concept and how it behaves with entities.

Predicted Spawn

Calling GameObject.Instantiate inside PredictedUpdate will allow you to predictively spawn objects you own. This will allow for reactive and synchronized gameplay like ball/grenade/missile throwing. If you mispredicted this spawn for any reason (e.g. your player was stunned server side and your client did not know about it yet), Netcode will automatically destroy mispredicted spawns for you.

Owner Prediction and Interpolation

Setting a ghost prefab as OwnerPredicted will dynamically switch instances’ mode to Predicted if on the owner client and Interpolated if on other clients. Interpolated ghosts are server driven and their state will be buffered before being made available to gameplay code. Note that all ghost fields are buffered when in interpolation mode, not just transforms. Which means any logic depending on order of arrival of certain field updates will be tick accurate for a given ghost in both predicted or interpolated mode. In other words, if a ghost is on tick x, all its GhostFields, transforms, rigidbody, etc will be on tick x, whether interpolated or predicted. Any other non-GhostFields will remain untouched.

Manual Prediction Switching

Prediction has a runtime cost. You execute gameplay logic for all predicted ghosts, multiple times per frame. In a world with 10,000 ghosts, this would take too many resources.
Netcode allows each client to dynamically change the ghost’s mode between interpolated (i.e. server driven) and predicted for each ghost. You can have a prediction bubble around your player, for example, where anything the player interacts with is predicted (to stay on the same timeline) and all other ghosts remain interpolated.

On the GameObject side, this looks like this:

// MyGhostBehaviour
this.Ghost.ConvertToPredicted(transitionTime: 0.1f); // converting between timelines means a jump in time. To hide this, transitionTime allows lerping this.
// This means that for 0.1f second, the ghost's position will be smoothed. This is similar to Rigidbody's interpolation. The GameObject's position will be the lerped value outside of PredictedUpdate(), then reset back to the predicted value while your code is in the PredictedUpdate method.

// To get the non-smoothed tick accurate value at all times, you can use
this.Ghost.Position
this.Ghost.Rotation
this.Ghost.GhostTransform

A good rule of thumb is to keep a maximum of a few dozen predicted ghosts and every other ghosts stay interpolated.
See the entities doc for more details.

Next Steps

Predicted everything:
We’re currently working with the physics team on predicted physics primitives. We’re first looking at Rigidbody. CharacterController and Colliders would be next. Just like we have a PredictedUpdate method, we’d also have a FixedPredictedUpdate method for gameplay code that needs to interact with predicted physics. Just like with N4E, the package takes over the ticking of the physics simulation.
I don’t have much more to share here yet and will post more on this later.

NetworkTime:
N4E has the NetworkTime singleton to get tick information. We have created a Netcode.Time API that can be accessed from PredictedUpdate, but we’re ideating on other options. More on this later.

Tooling:
We’re currently working on tracing and analyzer tooling to help detect and debug mispredictions. More on this later.

OnCorrected:
@Yoraiz0r suggested adding some way to detect if a value was corrected in our last dev log.
Niki is working on some changes that would make this kind of detection easier, she’ll post about this later.

7 Likes

Sorry if I’m late to the party but does this system need ECS for all to be released or is already usable? Can you explain a little bit to me if ECS for all will help or change something on this?

Upcoming Foundational Prediction Improvements

You all have identified four pain points with client prediction in N4E that we’ll be addressing.


#1: Partial rollbacks

The situation today is; we will only rollback and resimulate predicted ghosts that are included in a received snapshot (with one exception made for client predicted spawns - see RollbackPredictedSpawnedGhostState). This is fine if all predicted ghosts are always added to the snapshot, but a few things (like MaxSendRate, NetworkTickRate, and/or large ghost counts) can lead to snapshots only containing a partial subset of predicted ghosts - which we call ‘partial snapshots’.

Therefore: Partial snapshots lead to partial rollbacks, leading to gameplay quality degradation i.e. severe mispredictions.

Instead: Via a new opt-in toggle, we will expand the prediction history buffer into a ringbuffer to support an array of entries for each ghost, and whenever we need to rollback one ghost, all predicted ghosts will be rolled back and resimulated.

In testing, we observe that the largest overhead to the prediction group is in ISystem.OnUpdate calls and Job Scheduling, rather than per-entity ‘work’, so we don’t expect any noticeable performance degradation here, except when predicted ghost count is exceptionally high. Obviously, client memory consumption will increase when using this mode, as the ring buffer capacity will be sized to your client’s ping.

This mode also allows us to hide two toggle options on the GhostAuthoringComponent: RollbackPredictedSpawnedGhostState (now always true) and RollbackPredictionOnStructuralChanges (now always true), and it modifies the behaviour of the CalculateNumPredictedGhostToRollback job to instead verify which tick needs to be rolled back to (which will almost always be the latest snapshot tick, unless you have received multiple since you last ran the prediction loop).

Special thanks to @Jawsarn for raising the viability of this.


#2: Frequency of rollbacks

We currently always rollback and resimulate predicted ghosts when they exist inside a received snapshot, even if the data received broadly matches your predicted value. This makes client prediction consistently CPU intensive. Our plan is to perform a quick fuzzy equality check on your previously predicted state instead, skipping the entire rollback and resimulation step if the snapshot state and the historical states are within a configurable margin.

When applied to previous projects, this optimization reduces rollback frequency significantly.


#3: Static-optimized ghosts are not always rolled back

As ghosts only rollback when added to a snapshot, static-optimized, predicted ghosts won’t rollback if they don’t change on the server. Therefore, if you modify these ghosts within your client prediction logic, you will accumulate mispredictions until the ghost actually changes on the server.

This behaviour will be significantly improved when we always rollback all predicted ghosts, as long as there is always at least one other dynamic predicted ghost in your scene.

Again, shout out to @Jawsarn for this repro! :grin:


#4: Prediction smoothing is poor, particularly when quantizing LocalTransform state

The prediction smoothing API isn’t intuitive, nor very configurable. Worse: It doesn’t smooth jitter caused by quantization, which is one of the primary optimizations users expect to enable on predicted ghosts. The above fuzzy equality checks will significantly improve this, but we also intend to make the API simpler, and more configurable – with the goal here being reliability with smoothing.


Single World Host Smoothing

N4E’s server authoritative architecture executes at a fixed tick rate. This means that; if your simulation rate differs from your presentation (i.e. render) rate, gameplay will be choppy. Traditional N4E binary worlds mode solves this by running “partial ticks”; where a tick is run with the accumulated sub-tick delta time, then undone, repeatedly - see here for more information.

However, for single world hosts, we’re further leaning in on ‘performance is correctness’, and therefore decided on interpolation smoothing as a replacement for partial ticks. With interpolation smoothing, all presentation GameObjects and Entities will be smoothed between the two latest simulated values, using simple linear interpolation (where the time value is the sub-tick fraction). This approach introduces one tick of render latency (in absolute terms).

In the short term, it will only apply to Transforms (via writing over the LocalToWorld, for Entities, and by modifying the GameObject.transform for GameObject’s), though we expect to expand this feature as it sees use.

Similarly: It will only be available for host worlds, to begin with, with the long-term goal to make it an alternative to partial ticks for binary worlds, also.

5 Likes

ECS for All will change the underlying association between GO and entities. In the version in 6.6 (that’s currently hidden behind a define) we “fake” that relationship Netcode side. You can see more details on this in the GameObject to Entity Mapping section of the second devlog here.

2 Likes

is already usable?

In theory the code we’re working on is already available in 6.6, but hidden behind a define. It’s all very much work in progress and we’re still making changes to those APIs though, so we wouldn’t feel comfortable for folks to use it in productions just yet.

2 Likes

So basically you are making it work for 6.6 and then it will “just” work for 6.7?

Outside of the release versions (which are not announced yet), yes. We’re making it work now, developing our APIs in parallel to the work Entities for All is working on. Once Entities for All is ready, we’ll adapt to that. I’m sure there will be adaptations here and there, but we’ve already architected the GhostObject layer in preparation for Entities for All, so the transition should be smooth.

6 Likes

This is really nice to hear!

I think it’s interesting to point out that this should be similar to how predicted dynamic/kinematics already are interpolated from SmoothRigidBodiesGraphicalMotion.

And as I’ve said in previous post, it would be nice to see this on Client only worlds as well. With some nice API, maybe an opt in attribute + option to specify custom interpolation method/system.

If I understand correctly, depending on the context, Instantiate can be predicted or regular. I’m a bit surprised that you decided to implicitly switch the implementation based on whether it was called in PredictedUpdate or not.

I mean, I would do instantiation in some factory that is injected via DI and it feels a bit fragile.

Do you think it would work?

You understand correctly yes. A predicted instantiate would behave the same as a server authoritative instantiate. It’s when instantiating outside prediction that we log errors and destroy your GameObjects (as N4E doesn’t allow client authoritative spawns right now).
The only diff in behaviour between a predicted spawn and a server authoritative spawn is the predicted spawn has a sort of timer associated with it that’ll destroy the ghost if it wasn’t supposed to spawn.

And just to be clear, we’re not changing the engine side Instantiate. All of this either happens in Awake or in ECS systems later in the frame.

I’m a bit surprised that you decided to implicitly switch the implementation based on whether it was called in PredictedUpdate or not.

The goal is to make this seamless. The less logic difference between client and server logic in your prediction, the better. You want the same logic running on both to avoid mispredictions.
Your PredictedUpdate gameplay method will be called both client and server side, so if you can Instantiate server side, you should client side as well.

Dunno if that answered your question? Do you mind expanding on what would make this fragile?

How will pooled objects be handled when you want to “instantiate” them? For frequent objects you may not want to instantiate a new object every time and instead reuse them. To me, it only looks like the predicted spawning only applies to new objects and not pooled objects.

1 Like

We have working prototypes for pooling, but nothing to share API wise at the moment unfortunately.
The idea is you’d need to call some extra method to signal Netcode “I’ve taken this ghost out of the pool and is now spawned again”. That method would be callable from anywhere Instantiate would be called. So predictively client side and authoritatively server side.
So you will be able to do predicted spawns with pooled objects, just slightly differently.

2 Likes

Thanks for the explanation. Your intention to make it seamless sounds very plausible, but my guts tell me that in a large system anything implicit is painful to debug and test.

I don’t know how you achieved this context dependency for method calls, but I think that DI and reflection is a good example of how it can be fragile.

Let’s say my character needs to spawn something. It has a service injected via reflection, this service has a factory injected too. So the flow is character (PredictedUpdate) => service => factory.

So if the factory calls Instantiate, will it know that it should do exactly the predicted instantiation instead of the regular one?

It just feels like I want to have control over this?

1 Like

will it know that it should do exactly the predicted instantiation instead of the regular one

yes. Netcode.Time.IsInPredictionLoop will be set and can be accessed statically.
In the background, an entity will be created. It’ll be picked up asynchronously by systems running after PredictedUpdate and initialize it appropriately, giving it an ID, proper spawn tick, etc. Just like for normal server side instantiate.

It just feels like I want to have control over this?

What would you like to have control over? You can’t do an authoritative instantiate on a client. The only instantiates on a client are predicted ones while in the prediction loop, or invalid ones while outside. On a server, all instantiates are authoritative.

There’s something I forgot to talk about in my original message: guarding against multiple creations for the same ghost. You usually only want to instantiate once, even when rolling back and replaying the same tick.
A way to prevent this is to check Netcode.Time.IsFirstTimeFullyPredictingTick and only Instantiate once.
We could potentially provide some form of “Netcode.SafeInstantiate” that could do some of those checks for you.

Here’s a sequence of events in the frame to further clarify:

Client and Server

  • MonoBehaviour - Update()
  • ECS OnUpdate()
    • PredictedSimulationSystemGroup (called both client and server side. Once server side, n times client side)
      • IsInPredictionLoop is set
      • GhostBehaviour - PredictedUpdate() → Goes through all the PredictedUpdate for ALL your ghosts.
        • Gameplay logic calls Instantiate → creates ghost entity in the background
          • GhostObject Awake checks for IsInPredictionLoop and logs error
      • Other Netcode systems - OnUpdate()
        • detect newly created ghost entity and initialize it
      • IsInPredictionLoop is unset
  • MonoBehaviour - LateUpdate()

The big difference is behaviour wise. A client side instantiate will get destroyed automatically (after some time) if it mispredicted. A server side instantiate won’t, since it has the authority.

And if you try to do this:

Client

  • MonoBehaviour - Update()
    • Instantiate(myGhostPrefab)
  • ECS OnUpdate()
    • Netcode systems → Detect new ghost entity, this is invalid on client and destroys it

(Do note this only applies to ghost prefabs; regular offline GameObject prefabs are not affected)

2 Likes

Alright, I see, it makes sense. Thank you.

I have another question regarding physics prediction. Do you think it would be possible to predict a movement of a body by two players simultaneously? Imagine it like two guys moving a sofa - movement of one should affect the sofa and his friend.

If I understand correctly, prediction requires some client to be the owner of the object and other clients just replicate it. When two clients interact with the same object, who’s gonna be the owner?

And the last one, pinky promise. I tried many other solutions offering prediction, and most of them require splitting the object in two - visual, that’s being updated by prediction, and the rest. (I might be wrong, but the idea was that prediction requires another object). I assume this new prediction doesn’t have such limitation?

If I understand correctly, prediction requires some client to be the owner of the object and other clients just replicate it.

That’s actually not the case. Netcode allows predicting non-owned objects as well.

Basically, Netcode has 3 modes for a ghost. Interpolated (server driven), Predicted (by anyone) and Owner Predicted (where the owner client predicts it and other clients interpolate it).

So in your case, the sofa wouldn’t be owned by anyone and both clients would try to predict its movements.
Rocket League does a similar thing where they fully predict the whole world, including the ball and all the players. (you should watch their GDC talk, it’s great!)

Notice I said “try”. Depending on your gameplay, how fast your players move, you might run into mispredictions. It’s hard to predict other humans. Rocket League gets away with it since cars have inertia. Netcode is flexible enough for you to try, but in cases where it becomes too jittery, you could also make everyone interpolated (player 1, 2 and the sofa). You can use the ConvertToPredicted/Interpolated APIs to switch your various ghosts’ mode dynamically at runtime. So on pickup, all ghosts could be switched to interpolated, they’d all be on the same timeline with no jitter. when you drop it, you come back to predicted.

It’s what Survival Kids did with some of their coop box pickup mechanic. It’s also what some hero shooters are doing with abilities that interact with other players/the environment (imagine a shield interacting with incoming server driven bullets). You want interactions to be on the same timeline. Either predicted (but it can be hard to predict other players) or interpolated (it’s less reactive).
It really depends on your gameplay.

And the last one, pinky promise.

Keep them coming, love these questions!

I tried many other solutions offering prediction, and most of them require splitting the object in two - visual, that’s being updated by prediction, and the rest. (I might be wrong, but the idea was that prediction requires another object). I assume this new prediction doesn’t have such limitation?

“visual, that’s being updated by prediction, and the rest.” → I try to think as the reverse actually. Prediction touches anything gameplay that evolves over time. Visuals just depend on this.
But no there’s no requirement to split things.
We have a few instances where we lerp the transforms (when switching between prediction/interpolation timelines for example). In those cases, we apply a similar pattern to Rigidbody’s interpolation, where while you’re in the “real” context of prediction, you have the “real” value and when outside the transform is flip/flopped to the lerped transform.

2 Likes

That sounds very awesome! I hope you’ll release some examples or tutorials on YouTube channel covering these new features and API.

Looking forward to trying it! Thanks!

3 Likes

Hi Niki,

Are you able to say which version would have this behaviour change? We’re making a title that heavily uses predicted physics as well as interactions between static-optimized ghosts. Would love to test out what kind of improvements we see with our setup.

Also, I had a question about how Single World Host prediction / rollback works (sorry if it’s been addressed elsewhere). With a binary world setup the server remains a certain tick count behind so that it is simulating with the most correct input commands from players. Does the SWH send snapshots from the past to be more correct? And I’m assuming that single world hosts still do rollback, are the snapshots sent from the host affected by rollback too?

It’s currently targeting Unity 6000.7, but I cannot confirm that yet.

A SWH sits on the Server’s authoritative timeline, because it is both the server and a client. Therefore, it is the only client that is not ahead of the server. Any client connecting to this SWH server will be ahead of it.

Single world host’s do not rollback ever, as there is no need for them to. They are the authority, so they have all the information they need when they simulate. Binary world clients only rollback because they have to run ahead of the server (and ahead of the received snapshot stream).

2 Likes