Netcode rollbacks happening too often and compression causing jitter

[Pezzano]

Hi,
Info:
DOTS project.
Entities version used 0.51.1
Netcode for Entities version 0.51.1

I am seeing 2 different issues that are contributing to jitter and performance, the first issue is regarding how many rollbacks are fitting in a given time frame and the second issue is about prediction loop and quantization.

Rollbacks:
I’ve noticed that we’re seeing more frame rollbacks than expected for a given Round Trip Time (RTT). I’m keeping an eye on the RTT from the Networking.NetDbg window.
For an RTT of 30 ms, we’d normally expect just a couple of rollbacks when the server confirms a frame for the client, usually around 2 frames. But instead, I’m seeing our system getting rolled back 7 or 8 times in these situations.
These frequent rollbacks seem to increase as the RTT goes up. In fact, with higher RTT values, we’re looking at up to 15-16 frame rollbacks.
I’m wondering if there’s a way to implement some kind of state comparison, so that if the predicted frame’s values closely match or are the same as the server-confirmed frame, we can skip the rollback.
I’m also curious about the reasons behind all these rollbacks. Our simulation is running consistently on both the server and client, and we’re using a fixed tick system group.
Any insights on this issue would be greatly appreciated.

Quantization:
I am having some weird mispredictions between server and client even when hosting locally (0 ping). When debugging it further, I noticed that quantization only happens when snapshots are serialized and not between each tick. For example:
Client(frame1) → predicts float 1.234567
Client(frame2) → predicts +1 and results to 2.234567
Server(frame1) → confirms 1.234567 and quantizes to 1.234 (quantization 1000)
Client(frame1 rollback) → sets float to 1.234
client(frame 2 replay) → +1 and results to 2.234000
At this point there will always be a small difference between server and client frames because the engine doesn’t seem to be quantizing the values locally when doing the simulation, only when serializing and sending them as packets via the network. That small value can make the difference between hitting a step and running some ledge mount logic or not. This is prone to cause frequent mispredictions.
Is there a way to force ghost fields to quantize every time they get a new value assigned to them, to be sure they stay in sync every time they change?

Does anyone know if this is a known issue that has been addressed in version 1.0? Currently, we’re using version 0.51 and working on an update. It would be helpful to understand whether there’s still work required in these areas.

[CMarastoni]

For an RTT of 30 ms, we’d normally expect just a couple of rollbacks when the server confirms a frame for the client, usually around 2 frames. But instead, I’m seeing our system getting rolled back 7 or 8 times in these situations.

For 30ms at 60hz you have at least 2 plus another additional 2 because of the command slack (tha by default is ticks). So at minimum you are ahead of about 4 ticks.

In 0.5 there were some issues with predicted time, causing it to be farther ahead than necessary in certain cases (that should have been addressed in 1.0).

There is an also a bug I recently notice that can cause them: partial snapshots. There issue start occurring when the server can’t send all ghtost in the same snapshots. In some situation, ghosts that didn’t receive an update for a while, instead of continue prediction (as they should) they rollback to the last received tick from the server (that can be further back). That is causing a constant prediction of way more ticks than necessary,

Rollback also occurs every single frame on the client (pretty much) either because it receive data from the server (and this cause multiple tick re-simulation) or because of partial ticks (so re-simulated the same tick multiple time from the beginning of the tick with different delta time).

so that if the predicted frame’s values closely match or are the same as the server-confirmed frame, we can skip the rollback.

Yes, it is possible to add but required some changes in the package. It is not complex to implement but the “benefits” van be largely over estimated. First and foremost because it is necessary to implement some “fuzzy matching”, otherwise because of unavoidable misprediction (just because quantisation exist for example) states can be considered non the same. Second, because in case of a real misprediction, we are adding to the cost of the re-simulation also the ghost of checking the ghost state.
But we already thought of adding this (for predicted ghosts) and we can help out implementing the logic if necessary.

[quote[
I am having some weird mispredictions between server and client even when hosting locally (0 ping). When debugging it further, I noticed that quantization only happens when snapshots are serialized and not between each tick
[/quote]
Yes, this is the way it work and it is expected a small misprediction because of quantisation can occur. This is not problem by per-se (depending on the quantisation level). If we are speaking about quantisation of 1000 or 10000, the difference can be probably completely un-noticable in most of the cases.

This is why I mentioned that you need fuzzy checks in the point above. It is possible to implement the re-quantize approach in your project by your self if necessary. It is not hard to do.
The reason is not implemented by default is that Netcode for Entities does not require a fully deterministic simulation in first place. We need up to a certain degreed of predictability and simulation correctness, and the misprediction correction in many cases solve or hide that good enough.

Is there a way to force ghost fields to quantize every time they get a new value assigned to them, to be sure they stay in sync every time they change?

This is not necessary. And this is not the way the simulation work neither on the server nor the client.
The server quantize the state of the float when it send the data (or better when it copy the ghost component data to the snapshot buffer).
Whe the client receive that data, it dequantize and do all the necessary sim ticks.

I think the only place where it is really necessary to re-apply quantisation is at the beginning of the tick, so
the “initial state” of the simulated tick is consistent on both client and server.

[TKGFelipe]

@CMarastoni

Yes, this is the way it work and it is expected a small misprediction because of quantisation can occur. This is not problem by per-se (depending on the quantisation level). If we are speaking about quantisation of 1000 or 10000, the difference can be probably completely un-noticable in most of the cases.

The quantization difference can be quite noticeable, here’s a couple of videos demonstrating it:

The below video has quantization turned on, values are being quantized to 1000 and you can see the player jittering up and down as it collides with slopes and other objects.

k5oq6w

This one has Quantization completely removed from all GhostFields and you can see that the movement is quite smooth when moving across those same surfaces.

kjiae7

[CMarastoni]

You are using also quite a old version of the package… Before 1.0 the default for the quantisation level of translation was 100 (too low for physics, you need at least 1000 or 10000).
Are you using custom variants for all the components ? What happen if you add more quantisation to both translation and rotation (say 10000 or 100000) and the physics velocity too ?

Would be curious to debug this a little to understand what going on because this is definitively larger than expected. Looks like a big misprediction. And we have physics samples too that uses 1000 for both translation and physics velocity and has inclined planes etc etc and there is 0 sliding or misprediction.

Again could be some other bug we fixed.