(IN-33037) [1.0.0-pre.44] Too slow at PredictedSimulationSystemGroup

Currently there’s one WaitForJobGroupID takes insane amount of time at FilterUserCommandSystem. FilterUserCommandSystem is the system created by me. If official need profiler data I can pm the file.

Reply from an official staff:
Quickly looking at the profile, The FilterUserCommandSystem is waiting on a job scheduled in the system that ran just before it. However the dependencies for the job are dictated by the netcode library and the ECS component dependency manager. I haven’t dug into it but I can see there is lots of idle time intermixed with time where the job threads wake up looking for work to run but aren’t doing anything. This indicates that there likely aren’t any ready jobs due to outstanding dependencies. I’ll need to look more closely at the ComponentDependencyManager and the dependencies being set for the TogglePredictedJob since that is the very small job being waited on. I suspect if you aren’t taking the TogglePredictedJob explicitly as a dependency you are implicitly getting a dependency on the TogglePredictedJob due to the dependency manager adding in the job handles scheduled by the previous system via state.Dependency. The mitigations for this are:

  • We will need to look at the ComponentDependencyManager and state.Dependency setups to make sure they are minimal (I suspect they are not)
  • You can modify the netcode package to not schedule the TogglePredictedJob and instead call .Run() on the job
  • You can move other systems to run in between the GhostPredictionDisableSimulateSystem and the FilterUserCommandSystem using the [UpdateBefore/After(typeof(systemtype))] attributes

Claiming this as a “bug” it may be not appropriate. I would have started that as a question first. :slight_smile:

As the official staff support said, the TogglePredictedJob must run and enable/disable the prediction flags on al the predicted ghosts, based on the ShouldPredictAtTick.

It is a very tiny and fast job. However, this is only on part of the equation. Is also depends what the FilterUserCommandSystem is doing.

If it is running on the main thread (so it is not scheduling jobs) and you are trying accessing something that a job is going to write/read to, you are going to wait until that dependency has been completed.

Definitively there are jobs there that are updating/modifying the ghost data (from the GhostUpdateSystem), ones that modify the Simulate tags, and other that copies/apply inputs as well.
So, at our best we can check what it is going on there by inspecting your code and profiling data. If they are attached to the case at hand we will use them.

Thanks for reporting!

1 Like

One other tip: If you want to get a better picture of “total CPU time spent on this system”, go to Preferences and disable job threads (it resets on editor relaunch). For convenience, you can toggle this at runtime with the profiler open.

This obviously (usually) makes parallel systems perform worse, but the good thing is that you can see the total amount of work they do across threads, and crucially, you won’t receive false positives due to job scheduling waiting.

You can use this view to determine where most of the “work” is performed on a given frame, which may help with optimization efforts.

Saying that, when/what/how you schedule jobs is also a big part of framerate optimization, so profiling with jobs enabled is important too.

1 Like