Windows mouse Raw Input mouse handling broken in Rewired

A change made in this beta branch has broken Rewired's native mouse input handling in Windows. Raw Input mouse messages sent by Windows to Rewired's messaging window are now intermittently missed.

When the user moves the mouse and clicks or releases a mouse button simultaneously, the button down/up events may be missed causing both button press misses and stuck forever-on buttons. I have logged this at the lowest level possible when the messages are delivered to Rewired's message window.

I cannot explain this as Rewired consumes input directly from the Windows Raw Input API, registering with Windows for HID events for devices and does not rely on Unity for any of this information. This beta version of Unity definitely causes Rewired to miss mouse events.

Tested in Unity 2021.2.0b12.
This cannot be reproduced in 2021.1.21f1.

6 Likes

OOF, that's not good :(.

The only thing I can think of that could have caused this is this fix: https://issuetracker.unity3d.com/issues/editor-slows-down-when-rapidly-moving-mouse-with-high-polling-rate-in-play-mode

We found that when using very high input polling rate mice (tested with 8000 Hz), the framerate would drop dramatically, to the point where processing mouse events was taking longer than receiving them. The bottleneck was between PeekMessageW() and GetRawInputData() Windows APIs - it would acquire a mutex inside the kernel every time you tried to retrieve a raw input message, and with 8000 events per second the contention was just too high. I could reproduce this issue even in an empty win32 app that did practically nothing (no Unity involved).

The fix was to use the GetRawInputBuffer() API instead, which allows processing all queued raw input events at once without pumping them through the Windows message loop. Using that API removes these events from Windows event queue so they no longer arrive through the WndProc.

That fix landed to Unity 2021.2.0b9. Could you try using 2021.2.0b8 and 2021.2.0b9 to see if it started happening in b9?

How do you implement raw input? Do you spawn your own invisible window that you register for raw input notifications (like DirectInput)? Or are you piggybacking off Unity's WndProc?

3 Likes

Always appreciate your details insights @Tautvydas-Zilys

[quote=“Tautvydas-Zilys”, post:2, topic: 856247]
OOF, that’s not good :(.

The only thing I can think of that could have caused this is this fix: https://issuetracker.unity3d.com/issues/editor-slows-down-when-rapidly-moving-mouse-with-high-polling-rate-in-play-mode

We found that when using very high input polling rate mice (tested with 8000 Hz), the framerate would drop dramatically, to the point where processing mouse events was taking longer than receiving them. The bottleneck was between PeekMessageW() and GetRawInputData() Windows APIs - it would acquire a mutex inside the kernel every time you tried to retrieve a raw input message, and with 8000 events per second the contention was just too high. I could reproduce this issue even in an empty win32 app that did practically nothing (no Unity involved).

The fix was to use the GetRawInputBuffer() API instead, which allows processing all queued raw input events at once without pumping them through the Windows message loop. Using that API removes these events from Windows event queue so they no longer arrive through the WndProc.

That fix landed to Unity 2021.2.0b9. Could you try using 2021.2.0b8 and 2021.2.0b9 to see if it started happening in b9?

How do you implement raw input? Do you spawn your own invisible window that you register for raw input notifications (like DirectInput)? Or are you piggybacking off Unity’s WndProc?
[/quote]

Thanks for the reply and the helpful information. I also received many reports of the high refresh-rate mouse issue, bought an 8000 Hz mouse, discovered the issue happened without Rewired in the mix, and filed a bug report with Unity. It’s very interesting to know the underlying cause is the Raw Input API. That means I would also need to change Rewired to use the buffered method to avoid this, which means forwarding Raw Input messages to Unity is no longer an option (see below).

I have verified the issues does not occur in 2021.2.0b8 and does occur in 2021.2.0b9

That is definitely the cause.

  1. Rewired creates a hidden message window.
  2. Rewired registers with Raw Input to receive mouse, keyboard, and HID controller events. This prevents Unity from receiving those events because only one Window may receive events per application.
  3. Rewired processes the mouse event, then forwards it to Unity’s WncProc so Unity can use the event also so Rewired doesn’t break Unity’s mouse input. Keyboard and HID device events are not forwarded because Unity does not use them (in the legacy input system). (And the forwarding trick fails in the new input system because of threading.)
  4. Rewired processes the first message, then forwards it to Unity, at which point GetRawInputBuffer is called, consuming all the remaining Raw Input events for the mouse for that frame.

Because Unity is consuming all the Raw Input messages from the queue, Rewired doesn’t receive anything after the first event, and therefore misses many events.

The only solution I can see would be to change Rewired to no longer forward the messages to Unity and just break Unity mouse support when using Rewired.

Would it help if we exposed a way to forward the raw input events to Unity without going through SendMessage/PostMessage? That way you could consume all the events with GetRawInputBuffer yourself, and then forward them all to Unity without going through Windows message queue. I'm thinking of something like this:

namespace UnityEngine.Windows
{
    class Input
    {
        static void ForwardRawInput(NativeSlice<uint> rawInputHeaderIndices, NativeSlice<uint> rawInputDataIndices, NativeSlice<byte> rawInputData);
    }
}

Then you could build the data on C++ side and send it to managed land in one call:

typedef void (*DispatchRawInputToUnityFunc)(const void* rawInputHeaderIndicesPtr, const void* rawInputDataIndicesPtr, uint32_t rawInputEventCount, const void* rawInputData, uint32_t rawInputDataSize);
static DispatchRawInputToUnityFunc DispatchRawInputToUnity;
HWND g_RewiredHwnd;

template <bool DataIsFromGetRawInputBuffer>
static void AddRawInputEvent(RAWINPUT* rawInput, std::vector<uint32_t>& rawInputHeaderIndices, std::vector<uint32_t>& rawInputDataIndices, std::vector<uint8_t>& rawInputData)
{
   auto oldSize = static_cast<uint32_t>(rawInputData.size());
   rawInputHeaderIndices.push_back(oldSize);

   auto dataOffset = offsetof(RAWINPUT, data);
   if (DataIsFromGetRawInputBuffer)
   {
#if defined(_M_IX86) || defined(_M_ARM)
       static bool isWow64 = IsWow64();
       if (isWow64)
       {
           // RAWINPUTHEADER is 16 bytes on 32-bit, but 24 bytes on 64-bit. Since this data is memcpy-ed from the kernel,
           // we need to adjust the data pointer by 8 bytes if we're in a 32-bit process but 64-bit kernel.
           // We only need to do this with data coming from GetRawInputBuffer as GetRawInputData fixes up the single
           // event it returns.
           dataOffset += 8;
       }
#endif
   }

   rawInputDataIndices.push_back(oldSize + dataOffset);

   rawInputData.resize(oldSize + rawInput->header.dwSize);
   memcpy(&rawInputData[oldSize], rawInput, rawInput->header.dwSize);
}

static bool IsWow64()
{
   BOOL isWow64 = FALSE;
   if (IsWow64Process(GetCurrentProcess(), &isWow64))
       return isWow64 == TRUE;

   return false;
}

void ProcessRawInputMessages(HRAWINPUT rawInputHandle)
{
   std::vector<uint32_t> rawInputHeaderIndices;
   std::vector<uint32_t> rawInputDataIndices;
   std::vector<uint8_t> rawInputData;

   uint8_t rawInputBuffer[8 * 1024];
   uint32_t result = GetRawInputData(rawInputHandle, RID_INPUT, rawInputBuffer, sizeof(rawInputBuffer), sizeof(RAWINPUTHEADER));
   if (result == (static_cast<uint32_t>(-1))
   {
       LogError(GetLastError());
       return;
   }

   AddRawInputEvent<false>(reinterpret_cast<RAWINPUT*>(rawInputBuffer), rawInputHeaderIndices, rawInputDataIndices, rawInputData);

   while (true)
   {
       auto rawInputCount = GetRawInputBuffer(rawInputBuffer, sizeof(rawInputBuffer), sizeof(RAWINPUTHEADER));
       if (rawInputCount == 0)
           break;

       if (rawInputCount == static_cast<uint32_t>(-1))
       {
           LogError(GetLastError());
           break;
       }

       rawInput = reinterpret_cast<RAWINPUT*>(rawInputBuffer);
       for (uint32_t i = 0; i < rawInputCount; i++)
       {
           AddRawInputEvent<true>(rawInput, rawInputHeaderIndices, rawInputDataIndices, rawInputData);
           rawInput = NEXTRAWINPUTBLOCK(rawInput);
       }
   }

   DispatchRawInputToUnity(rawInputHeaderIndices.data(), rawInputDataIndices.data(), static_cast<uint32_t>(rawInputDataIndices.size()), rawInputData.data(), static_cast<uint32_t>(rawInputData.size()));
}

LRESULT CALLBACK RewiredMainWndProc(HWND hWnd, UINT message, WPARAM wParam, LPARAM lParam)
{
   if (message == WM_INPUT)
       ProcessRawInputMessages(reinterpret_cast<HRAWINPUT>(lParam));
}

extern "C" __declspec(dllexport) void SetDispatchRawInputToUnityCallback(DispatchRawInputToUnityFunc callback)
{
   DispatchRawInputToUnity = callback;
}

extern "C" __declspec(dllexport) void PumpInputMessages()
{
   MSG msg;
   while (PeekMessage(&msg, g_RewiredHwnd, 0U, 0U, PM_REMOVE))
   {
       TranslateMessage(&msg);
       DispatchMessage(&msg);
   }
}

Then on C# side:

struct CustomTimeUpdate
{
   [DllImport("Rewired_Core.dll")]
   static extern void PumpInputMessages();

   public static PlayerLoopSystem.UpdateFunction s_EngineTimeUpdate;

   public static void OnTimeUpdate()
   {
       s_EngineTimeUpdate?.Invoke();
       PumpInputMessages();
   }
}

delegate void DispatchRawInputToUnityDelegate(IntPtr rawInputHeaderIndicesPtr, IntPtr rawInputDataIndicesPtr, int rawInputEventCount, IntPtr rawInputData, uint rawInputDataSize);
static DispatchRawInputToUnityDelegate dispatchRawInputToUnityDelegate = DispatchRawInputToUnity;

[DllImport("Rewired_Core.dll")]
static extern void SetDispatchRawInputToUnityCallback(DispatchRawInputToUnityDelegate d);

void Awake()
{
    SetDispatchRawInputToUnityCallback(dispatchRawInputToUnityDelegate);

   // Insert our PumpInputMessage() right after PlayerTimeUpdate to minimize input latency
   var playerLoop = PlayerLoop.GetDefaultPlayerLoop();
   ReplaceTimeUpdate(ref playerLoop, 0);
   PlayerLoop.SetPlayerLoop(playerLoop);
}

private void ReplaceTimeUpdate(ref PlayerLoopSystem currentSystem, int depth)
{
   if (currentSystem.type == typeof(Initialization.PlayerUpdateTime))
   {
       var functionPtr = Marshal.ReadIntPtr(currentSystem.updateFunction);
       if (functionPtr != IntPtr.Zero)
           CustomTimeUpdate.s_EngineTimeUpdate = Marshal.GetDelegateForFunctionPointer<PlayerLoopSystem.UpdateFunction>(functionPtr);

       currentSystem.type = typeof(CustomTimeUpdate);
       currentSystem.updateFunction = IntPtr.Zero;
       currentSystem.updateDelegate = CustomTimeUpdate.OnTimeUpdate;
   }

   if (currentSystem.subSystemList != null)
   {
       int subSystemCount = currentSystem.subSystemList.Length;
       for (int i = 0; i < subSystemCount; i++)
       {
           ReplaceTimeUpdate(ref currentSystem.subSystemList[i], depth + 1);
       }
   }
}

[MonoPInvokeCallback]
static void DispatchRawInputToUnity(IntPtr rawInputHeaderIndicesPtr, IntPtr rawInputDataIndicesPtr, int rawInputEventCount, IntPtr rawInputDataPtr, uint rawInputDataSize)
{
    var rawInputHeaderIndices = NativeSliceUnsafeUtility.ConvertExistingDataToNativeSlice<uint>(rawInputHeaderIndicesPtr, rawInputEventCount);
    var rawInputDataIndices = NativeSliceUnsafeUtility.ConvertExistingDataToNativeSlice<uint>(rawInputDataIndicesPtr, rawInputEventCount);
    var rawInputData = NativeSliceUnsafeUtility.ConvertExistingDataToNativeSlice<byte>(rawInputDataPtr, rawInputDataSize);
    UnityEngine.Windows.Input.ForwardRawInput(rawInputHeaderIndices, rawInputDataIndices, rawInputData),
}

Apologies if there are bugs in that code - I didn't test it (or even compile it), it's meant to be just an example of what it'd look like.

[quote=“Tautvydas-Zilys”, post:5, topic: 856247]
Would it help if we exposed a way to forward the raw input events to Unity without going through SendMessage/PostMessage? That way you could consume all the events with GetRawInputBuffer yourself, and then forward them all to Unity without going through Windows message queue. I’m thinking of something like this:
[/quote]

Thanks for the code example! There are a number of features in there I wasn’t aware of like PlayerLoopSystem.

As a test, I implemented GetRawInputBuffer in Rewired and it works as I expected it would. Rewired receives the Raw Input event before Unity, empties the buffer, then sends the event to Unity which results in the reverse problem of Unity’s mouse input only receiving the first event and missing all the rest. Even though Unity’s mouse input is now broken, Rewired’s now works again. I did notice in my testing that Unity’s legacy input system’s mouse button events still work however, leading me to conclude Unity is using some other means for that information like WM_LBUTTONDOWN.

Rewired does not actually have its own message pump at present. It creates a message window and registers to receive the Raw Input messages for devices to that HWND. Unity’s message pump dispatches the messages, Rewired’s WndProc receives them, then Rewired forwards the message to Unity by calling its WndProc manually using CallWindowProc.

Not that it’s important, but Rewired’s Raw Input implementation is fully C# managed code using P/Invoke.

If you think exposing a way to forward the Raw Input messages back to Unity is feasible, I would be open to using it. Otherwise, the user has two options:

  • Use Rewired’s native mouse handling knowing Unity’s mouse handling will no longer work.
  • Disable Rewired’s native mouse handling and use UnityEngine.Input as the source of mouse input in Rewired. This is the only method that will allow both Rewired’s and Unity’s mouse input to function.

Having the ability to forward Raw Input events to Unity could also have an additional side benefit. In my early testing of the new input system, I was unable to forward messages back to Unity successfully I believe because input is now handled a different thread (?). This now applies to keyboard as well in the new system. (Unsure about other HID devices.) Similarly, I was also unable to implement Raw Input on a separate thread in Rewired for this same reason as Raw Input messages can only be processed by the thread to which they were delivered, so no cross-thread forwarding to Unity is possible. If Unity had a thread-safe buffer which could receive these events, handling input on different threads between Rewired and Unity while allowing both to function would be possible. Alternately, having the ability to get a callback from the new input system’s input thread would allow me to create a message window on that thread so I could process input and forward it back to Unity (would also need WndProc for the input thread’s message window), although with the GetRawInputBuffer issue, this is kind of moot. Just thinking out loud.

[quote=“guavaman”, post:6, topic: 856247]
I did notice in my testing that Unity’s legacy input system’s mouse button events still work however, leading me to conclude Unity is using some other means for that information like WM_LBUTTONDOWN.
[/quote]

Old input system uses WM_MOUSE* and WM_BUTTON events for button presses and to determine absolute mouse position, and only uses raw input for measuring mouse delta (to make it not be accelerated, etc).

[quote=“guavaman”, post:6, topic: 856247]
Rewired does not actually have its own message pump at present. It creates a message window and registers to receive the Raw Input messages for devices to that HWND. Unity’s message pump dispatches the messages, Rewired’s WndProc receives them, then Rewired forwards the message to Unity by calling its WndProc manually using CallWindowProc.
[/quote]

Oh, that makes sense. That means you’re already set up to be called at the right time in the player loop.

[quote=“guavaman”, post:6, topic: 856247]
Not that it’s important, but Rewired’s Raw Input implementation is fully C# managed code using P/Invoke.
[/quote]

I thought I remembered seeing a native DLL in there somewhere, so I assumed you did raw input in it :).

[quote=“guavaman”, post:6, topic: 856247]
If you think exposing a way to forward the Raw Input messages back to Unity is feasible, I would be open to using it.
[/quote]

Yup, I created this bug to track it: https://issuetracker.unity3d.com/product/unity/issues/guid/1368835/. It’s on my top priority list to address next.

[quote=“guavaman”, post:6, topic: 856247]
Alternately, having the ability to get a callback from the new input system’s input thread would allow me to create a message window on that thread so I could process input and forward it back to Unity (would also need WndProc for the input thread’s message window), although with the GetRawInputBuffer issue, this is kind of moot. Just thinking out loud.
[/quote]

Unity receives raw input from Windows on the main thread. I believe we only use dedicated thread for HID devices (not 100% sure on that, would have to check).

1 Like

[quote=“Tautvydas-Zilys”, post:7, topic: 856247]
Old input system uses WM_MOUSE* and WM_BUTTON events for button presses and to determine absolute mouse position, and only uses raw input for measuring mouse delta (to make it not be accelerated, etc).
[/quote]

That makes sense.

[quote=“Tautvydas-Zilys”, post:7, topic: 856247]
I thought I remembered seeing a native DLL in there somewhere, so I assumed you did raw input in it :).
[/quote]

Yes, that was added a few years ago because IL2CPP didn’t support a specific IL code used by SharpDX’s Direct Input implementation so it stopped working. I added the native DLL at that time because I could no longer use SharpDX for that. Direct Input support is optional.

[quote=“Tautvydas-Zilys”, post:7, topic: 856247]
Yup, I created this bug to track it: https://issuetracker.unity3d.com/product/unity/issues/guid/1368835/. It’s on my top priority list to address next.
[/quote]

Thanks!

[quote=“Tautvydas-Zilys”, post:7, topic: 856247]
Unity receives raw input from Windows on the main thread. I believe we only use dedicated thread for HID devices (not 100% sure on that, would have to check).
[/quote]

As of my last testing of the new input system (quite a long time ago), HID devices were not using Raw Input because registering them to Rewired’s window didn’t stop them from working.

Correct, we use HID APIs for that: https://docs.microsoft.com/en-us/windows-hardware/drivers/ddi/hidsdi/

[quote=“Tautvydas-Zilys”, post:9, topic: 856247]
Correct, we use HID APIs for that: https://docs.microsoft.com/en-us/windows-hardware/drivers/ddi/hidsdi/
[/quote]

Yeah, that’s what I figured. I had to implement this API specifically for Xbox One controllers a few years ago due to a bug in an early update of Windows 10 that caused Xbox One controllers to crash windows when unplugged if they had been registered for Raw Input events. Good to know this is still being used in the new input system and I don’t have to worry about forwarding events for those. Thanks!

Alright, the raw input forwarding API landed to 2021.2.0b16. Attached is the API documentation preview. Hopefully it allows you to use buffered raw input reads yourself and then forward them to Unity too :). Lastly, this should allow you to forward both mouse and keyboard events to Unity as it will also work with the new input system too.

7549720--933178--ForwardRawInput.zip (155 KB)

1 Like

[quote=“Tautvydas-Zilys”, post:11, topic: 856247]
Alright, the raw input forwarding API landed to 2021.2.0b16. Attached is the API documentation preview. Hopefully it allows you to use buffered raw input reads yourself and then forward them to Unity too :). Lastly, this should allow you to forward both mouse and keyboard events to Unity as it will also work with the new input system too.
[/quote]

Awesome! I like the use of IntPtr in the new API. When 2021.2.0b16 is available I’ll test it out. That’s great news it will work with the new input system as well. Until now, I’ve always advised users disable it, but now it should be possible to have it running alongside Rewired on Windows.

I'm one of those people who reported this problem within Rewired and was made aware of this thread. I didn't expect much but wow. Thank you so much to both of you, I don't get all the technical details but it seem that Rewired will get updated later on based on the fix and API made available by Unity

Awesome :)

1 Like

Is this fixed in b16?

[quote=“ZhavShaw”, post:14, topic: 856247]
Is this fixed in b16?
[/quote]

I have not made an update to Rewired to change how Raw Input works in Rewired to use the new API, so at the present time, no, it’s not fixed.

1 Like


Ahh, I downgraded according to your tested version.

[quote=“Tautvydas-Zilys”, post:11, topic: 856247]
Alright, the raw input forwarding API landed to 2021.2.0b16. Attached is the API documentation preview. Hopefully it allows you to use buffered raw input reads yourself and then forward them to Unity too :). Lastly, this should allow you to forward both mouse and keyboard events to Unity as it will also work with the new input system too.
[/quote]

I downloaded 0b16 and began implementing the Raw Input forwarding, but I’m seeing it was implemented using unsafe pointers. Because Rewired_Core.dll is a DLL and I only release one version of the DLL for each major release of Unity, I handle branching dependent on Unity preprocessor defines like this using a script included in the project that implements an interface which is defined in the DLL so it make function calls using the interface. The problem is, the script included in the project must make the call to UnityEngine.Windows.Input.ForwardRawInput which is unsafe. This requires Unity be set to compile allowing unsafe code. Based on my testing, this is not the default setting, which means when Rewired is installed, it will almost certainly have a compiler error on start.

The only workaround I can think of would be to create another DLL that includes nothing but the function call to UnityEngine.Windows.Input.ForwardRawInput, then call into that from my script to avoid the compiler errors. However, also this introduces the problem that I will now have to maintain a separate release of Rewired for Unity 2021.2. I am already currently maintaining 7 branches for the 7 major versions of Unity Rewired supports. I have made it a point to avoid having to maintain separate projects for minor Unity releases.

It would be helpful if there was another overload of the function that used IntPtr instead of native pointers.

Thanks for trying it out.

Do you have an .asmdef for your script? You could create one and allow unsafe code inside the .asmdef. That way you wouldn't need "allow unsafe code" to be turned on for the whole project.

I can add this. Let me know if you run into any more issues with the API.

Apologies about the unsafe pointers... I debated whether to use those or IntPtrs. I decided on the pointers because they had more type information so it would be harder to use them incorrectly. I didn't even think about unsafe code not being on by default.

I don't currently use ASMDEF files in Rewired due to a lot of issues they cause with installing integration addon-packs into the Rewired folder that need to reference content in Assembly-CSharp.dll (classes or functions defined in the supported asset scripts and therefore cannot be referenced from a separate assembly).

It looks to me like I could use an ASMDEF w/ unsafe enabled as a workaround by including just a single static class containing a wrapper function for ForwardRawInput that takes IntPtrs. Because the assembly is compiled by Unity, I can use the #if UNITY_2021_2_OR_NEWER to exclude this call for Unity 2021.1 without having to maintain separate 2021.1 and 2021.2 branches of Rewired. This wouldn't be possible using a normal DLL. Thanks for the suggestion.

Makes sense. It would also be acceptable to pass uint arrays for the header and data indices and an IntPtr for the data.

Okay, thanks!

@Tautvydas-Zilys

One question:

Is it okay to call ForwardRawInput multiple times per frame? I see your implementation expands the buffers if too many messages come in. I would like to avoid that, so I'm forwarding the events whenever the buffer gets full. I've tested with an event buffer size of 1 just to force it to be called multiple times, and it seems to work, but I just wanted to make sure this is okay.