Input stutters (Android)

[RflectN_TapLab]

Lately I was investigating a “laggy scroll” problem in the main menu of our game (Android platform) made with Unity. Initially, I thought the UI system and layouts were to blame. I opened the profiler and observed a few noticeable spikes, caused by layouts and some of our scripts. But these spikes where less frequent than reported “lags”. However, I decided to optimize them since they were already noticeable. Having successfully optimized these points, I thought that this task was completed, but the feedback I got proved me wrong. With no evidence in the profiler, I requested a more detailed description of the problem. It turned out that small but frequent stutters were the main problem. And those stutters were more noticeable when scrolling relatively slow.

After that feedback, I double-checked the Unity Profiler. I used the Android GPU Profiler to get and analyze a system trace. I disabled almost all elements in the main menu. No luck. Those stutters wheren’t visible in any profiler, but I and other people could see them in the game. What was also interesting to me, that in other games from the Store, which I was sure were made with Unity, I could also spot the same problem if I looked closely. So, there is at least some problem which affects not only our game. And then, while talking to a collegue about this issue, we noticed a strange thing. Those stutters were only present during scroll dragging. As soon as you let it go, even if it has enertia and keeps moving, the problem will disappear. This gave me an idea: performance isn’t the problem, an input system is.

So I decided to create a new minimalistic project to reproduce and test this problem. In this project, I wrote a script to log frame delta time, touch delta time and OnDrag delta time. And tested following configurations:

• Unity 2021.3.23, Old Input System
• Unity 2022.3.7, Old Input System
• Unity 2022.3.7, New Input System
• Unity 2022.3.7, New Input System and pollingFrequency = 120
• Unity 2022.3.7, New Input System and pollingFrequency = 240

Using these devices:

• Samsung Galaxy A70 (Android 11)
• Samsung Galaxy S10 with Exynos SoC (Android 11)

Scroll stutters were observed in all of them. And I got some confirmations of my hypothesis about an input system. While frame delta time was pretty consistent at around 16.7 ms, touch and OnDrag delta weren’t. Since I didn’t know the root cause of this issue other than “it’s something on an input system side”, I decided to make this post. But before that, I retested this problem to structure the results a bit and to visualize them. Unfortunately, I didn’t have a lot of time to properly retest all of those configurations on multiple devices. So I retested only two of them:

• Unity 2022.3.7, Old Input System
• Unity 2022.3.7, New Input System

Using only one device:

• Samsung Galaxy S22 with Snapdragon SoC (Android 13, Screen refresh rate set to 60 Hz)

So here are a few graphs of results, each of which represents a separate swipe:

The New Input System seems to handles this problem better, but still not good enough. Here is a link to a Git repository with the project, builds, logs, screen recordings and graphs of those last two tests.

So I am curious to know:

• Am I missing something?
• What is the root cause of this problem? Is it the platform’s input system, or Unity’s input systems, or some combination of both?
• Could we, app developers using Unity, somehow fix or minimize this problem?
• Could you, Unity/Input System developers, fix or minimize this problem?

P.S. Those stutters more visible on something contrast, like a cell title in the project.
P.P.S. Several times during those tests, I have observed some strange input behaviour in Android debug overlay:

[RflectN_TapLab]

A few weeks ago I again spent some time to investigate this problem a bit more. If last time I checked the difference between new and old input systems, this time I decided to check the difference between the Unity and Input System package versions themselves.

TL;DR Seems like some kind of regression in the 2021 version of the engine. The only solution I know of is to downgrade a project to Unity 2020.3.

Setup

• Unity versions: 2019.4.40, 2020.3.48, 2021.3.30, 2022.3.18, 2023.2.7
• Input System package versions: 1.1.1, 1.3.0, 1.5.1, 1.7.0 (only versions 1.1.1 and 1.3.0 of the package are tested in Unity 2019, since there was some error with tests assembly reference and I didn’t bother to spend time to fix it, especially when I didn’t even consider to downgrade the project to such an old version)
• Input System update mode: Dynamic Update
• Other Input System settings: Default
• Target frame rate: 60
• Optimized Frame Pacing: Disabled
• Test device: Samsung Galaxy S22 (Snapdragon 8 Gen 1, Android 14, screen refresh rate set to 60 Hz in system settings, touchscreen polling frequency is around 120 Hz in getevent tool)
  
  

Methodology
In the new version of the test, I decided to use the low-level functionality of the new Input System, specifically InputSystem.onEvent. To filter pointer position events, InputControl.path is checked for the string “/position”.

The value of InputEventPtr.time is taken as the time of creation of the input event. The time (Time.realtimeSinceStartupAsDouble and Time.realtimeSinceStartup in Unity 2019) when the event is received in a InputSystem.onEvent handler is considered to be the time of the input event dispatch. So the input event dispatch delay is the difference between those two values.

The definition of a stutter is a little bit more complicated. First, let’s define an input event with a missed update. The input event with a missed update is an input event that was created in frame F_i, but dispatched only in one of the following updates. In other words, the event was created before MonoBehaviour.Update in F_i, but dispatched later.

But to say that every frame with such an event is a frame with a stutter is incorrect. For example, a frame that has four dispatched events and one that is not. For a user, this frame will not contain a stutter, because at least some of his input was dispatched and processed. Only the input delay for some of his actions is increased. Another example, all input events are dispatched without any missed updates for the first n frames. Then none of the created and active input events are dispatched in the next frame. But after that all events are dispatched without missed update again. This is a case of a stutter, because for the user their continuous input is completely ignored in one frame.

So, we will consider frames that contain only input events with missed updates to be a stutter. In other words, at the moment of MonoBehaviour.Update execution in such frames there should be only input events created before this moment and that have not yet been dispatched. (This definition also has some flaws, what will be said in the conclusion)

Also a scripts update delta, an input system update delta and an input events creation delta will be demonstrated in later graphs. The scripts update delta is the time difference between the current MonoBehaviour.Update call and the previous one. The input system update delta is the time difference between InputSystem.onBeforeUpdate calls. Graphs of these two values are used only to demonstrate the stability of these updates and to exclude the correlations between these deltas and stutters (although it is worth noting that these graphs are not enough to definitely exclude such correlations). The input events creation delta is the time difference between the time of creation of the two sequential input events. In general, the last value isn’t very useful, because it can be large if, for example, pointer was pressed but didn’t change its position, which is not a problem. Or, another example, we can get an event for each of the axis (x and y) in one frame, which could result in an near-zero delta. This value is also affected by redundant input events merging inside of the Input System, which can be disabled with this parameter (why make a parameter representing a disabled state rather than an enabled one? it makes it harder to read and understand). So it is plotted more out of interest than anything else.

The test process itself:

• Install target app version on a test device
• Launch the installed app on the test device
• Launch screen recording
• Clear logcat in Android Studio while recording countdown isn’t complete
• After recording is started, “draw” a big “S” figure with fast start and slow end
• Pause logcat
• Stop screen recording
• Copy logs for V1 and V2 versions of the test from logcat to a log file (V1 is the test from the previous message in the thread and V2 is the one, described in this message)
• Generate graphs for each version of the test

[RflectN_TapLab]

Results
You can find all the results here.

Different Input System package versions
Let’s look at the difference between different versions of the Input System package within Unity 2021.3.30.

The “Scripts updates” graph shows the ratio of MonoBehaviour.Update calls with stutters to MonoBehaviour.Update calls without stutters and with some input (such as active and non-dispatched input events at the moment of the update, or any input event dispatch between previous and current updates). The number in brackets corresponds to respective updates count.

As you can see, there isn’t any really noticeable difference between the different versions of the Input System package in Unity 2021.3.30. And in all cases the relative count of stutters is around 10%, which, in my opinion, is pretty high. Additionally there is a fairly high input events dispatch delay in all of those cases. The higher delay in the case with Input System 1.3.0 is probably an anomaly of that session.

[RflectN_TapLab]

Different Unity versions
Unity versions will be compared using the Input System 1.7.0. Except Unity 2019, where the Input System 1.3.0 will be used instead, since it is the newest version of the package that I was able to run on that version of Unity.

Unity 2019 and 2020 stand out from the rest in this test because they have less stutters and lower input events dispatch delay. Starting with Unity 2021, you can see the worsening of these metrics.

[RflectN_TapLab]

It is also worth noting that Unity 2023.2, for some reason, has even worse event dispatch delay than Unity 2021 and 2022. And in this case it doesn’t look like an anomaly, because such a behaviour was observed in all tests with Unity 2023:

[RflectN_TapLab]

Also let’s look closer at some of the stutters from Unity 2021.3.30 with the Input System 1.7.0:

As you can see from these graphs, there is no significant correlation between stutters and updates deltas instability. Also these graphs demonstrate quite well what I wrote in the definition of stutter. For example, let’s look at “Input event 139” in the “Input stutter context 9” graph. For some reason this event wasn’t dispatched in the first update after its creation. But this update isn’t a stutter because there are other input events (137 and 138) in this frame that were dispatched (the same can be said about events 135 and 136 in the same graph). The only stutter in this graph is the update marked with the red dashed line, as it has only events 133 and 134, both of which were created before the update, but dispatched only after it.

[RflectN_TapLab]

Optimized Frame Pacing
In my experience there were a few instances where the Optimized Frame Pacing was blamed for some of the stutters and frame time inconsistency. And turning if off really minimized those problems. So I turned it off for all tests, just in case. But then I decided to check it too. This test, as well as the subsequent ones, was done only with Unity 2020.3.48 and Unity 2021.3.30 and with the Input System 1.7.0.

However, as you can see, there is no any visible connection between stutters and Optimized Frame Pacing.

[RflectN_TapLab]

Redundant input events merging
It was also interesting to check the disableRedundantEventMerging option in the Input System settings, since some problem in this merging logic could easily lead to such a problem.

There are more stutters in Unity 2021.3.30 test. The reason for this behaviour isn’t clear, but partially it can be explained by a session anomaly, although the difference is somewhat high for that, in my opinion. Also the input events dispatch delay has increased and input events creation delta has decreased. And this last change, in my opinion, is the most interesting in this test, even though earlier I called this parameter useless. As you can see, nearly all input events, excluding a few cases, were created with a time difference of less than 10 ms, which corresponds to the OS touchscreen input polling rate of around 120 Hz (~8.33 ms). So we can say with a high degree of certainty that at least input events are dispatched from OS to the app correctly.

Let’s also take a lot at a stutter context graph for Unity 2021.3.30:

What is interesting, now there are cases when input events skip two updates before being dispatched (events 102, 117, 120, 124, 128, 134). Otherwise, nothing new.

[RflectN_TapLab]

Input System polling frequency
And finally, let’s check the InputSystem.pollingFrequency parameter. Let’s set it to 2 times greater that Application.targetFrameRate, that is 120 Hz in this case.

Nothing new. But it is worth noting that it is possible that this parameter doesn’t affect touchscreen sampling at all. I don’t know how exactly touchscreen input is read on Android, but considering the getevent tool output, I think that the app itself does not poll this input and instead it’s just getting input events from the OS.

[RflectN_TapLab]

Conclusion
Is it a comprehensive analysis of the problem?

No, it is definitely not. At least these tests should be run multiple times. While fixing some bugs and improving the tests, I had to run it a few times, and during those runs I saw about a 5% difference in stutters and about a 20% difference in input events dispatch delays. It would also be nice to automate those tests to exclude a human factor and to run these tests on multiple devices.

This version of tests (V2) also contains some shortcomings. For example, with the current definition of an event with skipped update (and, as a result, stutter) there is at least one case where we could get a false-positive - if the input event is created after InputSystem.Update, but before MonoBehaviour.Update in the same frame.

Is it sufficient?

In my opinion, yes. Especially considering that it has been confirmed to some extend by most people who have interacted with the builds from this test. They also confirmed improvements during internal testing when I’ve downgraded our main project from Unity 2021.3.30 to Unity 2020.3.48. The difference in how the input system performs in Unity 2020 and 2021 is really noticeable to users. And it seems to be more noticeable on powerful devices with fairly stable frame rate. Which in theory is quite understandable, since when there are no any input all animations and movements look very smooth. And then, when the user starts to move a scroll or, for example, a map, it starts to “lag”.

P.S. I don’t really want to downgrade the project to 2020.3.48)
P.P.S. It would be really great to get some tool for input events analysis and debugging in the editor.

[Un1ty01]

Have you found any solution to this issue?

[RflectN_TapLab]

@Un1ty01

Have you found any solution to this issue?

The only way to minimize this problem that I found in my tests is to downgrade a project to Unity 2020.3. Unfortunately, I think there is nothing else we, as users of Unity engine, can do. Excluding some extreme options like writing an input system from scratch.

I reported it as a bug (IN-70161) a few weeks ago and the report has since been moved to some internal team board (UUM-67528). There are no any other news I know about this problem, unfortunately.

[Un1ty01]

Do you have a link to track the issue?

[RflectN_TapLab]

@Un1ty01

Do you have a link to track the issue?

Yeah, but it seems like that I cannot share it. After my bug report I got an email with a link to a request with that report on the Unity’s Jira board. But the same email says not to share this link…

But, to be honest, at that moment there is no any interesting information. The report got the ID IN-70161, then it was reviewed and passed to some internal board without local reproduction where it got the ID UUM-67528. That is basically all I know from this link. And, as a person who has never reported a bug to Unity before, I don’t know what will happen next. Is it just a “black box” from the moment it got to that internal board? Or will all the updates from that board be passed to this Jira report? I don’t know. But when there are any interesting updates about this issue, I will post them here.

[lyndon_unity]

I’ve just reviewed the status of this issue and made it public:

https://issuetracker.unity3d.com/product/unity/issues/guid/IN-70161
(same link expanded :
Unity Issue Tracker - Input stutters in Android player built with Unity 2021 and newer )

I’ve also added some notes as I think it could be related to this time.deltaTime change in unity 2020.2

Fixing Time.deltaTime in Unity 2020.2 for smoother gameplay: What did it take?

The Android fix for deltaTime landed to 2021.1.0a2 for both OpenGL ES and Vulkan.
Time.deltaTime Not Constant: VSync CameraFollow and Jitter page-10#post-6630178

This newer task in the input backlog hasn’t been schedule yet.

[optimise]

@lyndon_unity

I’ve just reviewed the status of this issue and made it public:

https://issuetracker.unity3d.com/product/unity/issues/guid/IN-70161
(same link expanded :
Unity Issue Tracker - Input stutters in Android player built with Unity 2021 and newer )

I’ve also added some notes as I think it could be related to this time.deltaTime change in unity 2020.2
Fixing Time.deltaTime in Unity 2020.2 for smoother gameplay: What did it take?

The Android fix for deltaTime landed to 2021.1.0a2 for both OpenGL ES and Vulkan.
Time.deltaTime Not Constant: VSync CameraFollow and Jitter page-10#post-6630178

This newer task in the input backlog hasn’t been schedule yet.

Hi. Have u verify input performance with the latest Unity 6 preview (6000.0.x) release

[villevli]

In my testing this issue appears in Unity 2021.3.21 (or 20). Not present in 2021.3.19
This in the release notes of 2021.3.20 could be related:

• Android: Fixed the Time.deltaTime values so they are now stable on Android devices. (UUM-10244)

[valiantjohnc_unity]

Is there any workaround for this?

[Un1ty01]

Will there be any fix for this any time soon?

[lyndon_unity]

The issue has been passed over to the Android platform team for investigation.