Inputs occasionally missed or processed twice on the server

Unity Engine
,
1

Today’s issue :slight_smile:

Occasionally, the server will either miss a client’s input, or process it twice on sequential frames. Missing input occurs perhaps 10% of the time a button is pressed. Double sequential processing much less (1-2%?). The code is very simple, so I can’t see what I’m doing wrong.

ReadHeroInputSystem, running in GhostInputSystemGroup:

  • Set input data to default.
  • If a button was pressed, set an input event and an int bitfield value.

UpdateHeroAbilityButtonsSystem, running in PredictedSimulationSystemGroup:

  • If bitfield contains specific flag, process input.

This is a log screenshot when everything works as expected.
9847824--1417485--upload_2024-5-21_15-52-28.png

This screenshot is when the input is missed.
9847824--1417488--upload_2024-5-21_15-52-39.png

Also, I’ve just also created a log where I check the input event if the bitfield is 0 - the input event IsSet is true, but none of the other fields have values (reset to default). This input event is 100% not being set from anywhere except when also setting the button data.
9847824--1417491--upload_2024-5-21_16-19-59.png

Is there special handling of InputEvent fields that could/should also be applied to regular fields?

There’s no fancy shenanigans or whatever occurring in the code around these systems, it’s pretty much as straightforward as possible.

Note that the client is connected via IPC, and I’ve disabled local prediction (due to the other issues I’ve reported).

Unity 2022.3.23
DOTS: 1.2.1

(this is reproducible in the uploaded project for support ticket #1851449)

2

Thanks a lot for the report. The missing processing of input is a known issue, assuming that it is the same problem. The issue here is that the server is batching ticks together by skipping one tick, but making the next ticks DeltaTime twice as big so that the simulation will still be roughly the same. This has the side-effect that those skipped ticks will skip the input, which can in some cases be fatal. We are still discussing what the best solution from our side is.

As a workaround, can you try setting the MaxSimulationStepBatchSize in NetCodeConfig to 1 so the server doesn’t batch? When you’re doing this, can you also check whether you are still getting the duplicate input processing?

3

Thanks for the reply. Unfortunately, I see no difference in behaviour when setting MaxSimulationStepBatchSize to 1 - both missing and duplicated events are still being seen at the same rate.

The biggest red flag is that when an input is missed, the InputEvent.Count is set to 1, but all other fields have been reset to default.
9849231--1417866--upload_2024-5-22_10-16-39.png

This is the entire code for setting inputs, and the only place UseInputEvent.Set is called (run from within a system in GhostInputSystemGroup). It’s therefore impossible from my code for UseInputEvent.Count to be non-zero but for all other fields to be default. There must be something funky going on under the hood with InputEvents. I’ve poked around the generated source but nothing stands out.

9849231--1417869--upload_2024-5-22_10-18-26.png
9849231--1417869--upload_2024-5-22_10-18-26.png468×610 209 KB

As a side note, when poking around the generated source file, I noticed that the first field in CalculateChangeMaskGenerated returns 1u or 0, when all other fields return just their bit (e.g. 1u<<1). This means that if the first field is changed, ALL fields are considered changed?

4

Also, following on from my side note, the generated system XCompareCommandSystem.CompareJob uses (1u<<0) for determining the changemask of every field, except those of type Entity.

9849288--1417884--upload_2024-5-22_11-3-1.png
9849288--1417884--upload_2024-5-22_11-3-1.png1096×484 368 KB

5

If I remember correctly, Entities are not supported in commands. Are supported for RPCs but not for command stream.
The change mask for commands is used only to understand if the command is different than a previous one (when encoded) and so as soon as a field is changed, it is marked as such.

6

Ah? I’ve never had a problem with including Entity fields in commands.

Yup, the changemask comment is that every field uses 1u<<0, but they should all be unique and match the million other places those fields are serialized/deserialized.

7

Actually the only use of the XCompareCommandSystem.CompareJob static function is to see if it returns non-zero, so I’m sure the function could be simplified :slight_smile:

8

Not really. We have two distinct changemask calculation for the commands and ghost (when the command is replicated)

9

Of course it can be simplified slightly a bit, that not the point here. We can potentially do a memcmp but we don’t have guarantee that there aren’t private properties or whatever else. So field by field comparison is unfortunately necessary unless we enforce some stricter rules.

10

Fair enough. At risk of losing sight of the original issue of this thread, all this changemask stuff was the only red flag I saw while trying to debug.

11

Events are different than other fields. The count is restored using the difference in between the stored counter in the command buffer for the previous and current input used for the tick.
The count itself actually represent the delta (number of set) operation in between the two inputs.

Input are sampled at variable frame rate, so for a given tick you may press the button multiple times. The input itself is reset (so if you don’t move or do anything it stay all 0) but the even count continue to increase each time.

12

Ah ok, makes sense. I’m definitely seeing only one Set per frame on the client (when the button is clicked) which is correct, but as all the other fields are occasionally default, it seems that something else is going on - the InputEvent field is always correct, but not always all other fields.

13

The difference is that in case the server either lost an even or packet (or discard) the resumed data for the field is the actual value of the command for tick X (if exist) or the last input tick (if it doesn’t).

You can always check what tick the event retrieved is for by checking the Input.Tick value. If this is not the same as the tick you are processing, you can infer it is using the latest tick received.

14

I will try checking the tick, although when the server misses the event it always misses it, it’s not like it occurs a few frames later. However, I’m struggling to see why I need to go down this rabbit hole. If I can’t trust the server to always see and process the client’s input accurately, especially for an IPC client, it feels like an issue deeper than gameplay logic. This ghost input to server processing flow is about as simple as I can make it, and it still isn’t 100% reliable. Are there any improvements to these systems coming in 1.3, or planned for later?

15

I’m going to check the project you pushed to understand what this is, but usually press event because of the counting mechanism are not lost on the server in term of transition, especially for IPC connection.

1 Like
16

Hey @CMarastoni , have you had a chance to look into this?

17

I just tried setting the simulation tickrate to 30 from 60 in the netcode config hoping that having the server not need to catchup as much would make it miss less, but it actually increased the miss rate to about 80-90%.

18

@Richay not investigate yet. I will do asap.

3 Likes
19

Hey @CMarastoni , any updates on this?

20

Hey, bump, status? Any updates ?Etc…