Development Build runs in background even with "run in background" unticked.

Build & Deployment
1

What it says on the tin.

We’re currently trying to track down a problem that only occurs in builds, only occurs with PS4 controllers, and only occurs if the player minimises or defocuses the game for a few minutes before coming back to it. If they do this, the game locks up and has to be closed.

I had hoped that a development build wouild allow me to find out where it was locking up, but that’s not possible because development builds don’t suspend when defocused.

If anyone knows how I can work around this, or why having a PS4 controller connected would cause the lock-up, I’d be very much obliged!

2

You can work around it by calling Application.runInBackground = false on startup. We automatically enable runInBackground in development builds so that they don’t pause when you attach a profiler.

3

Ah! Fabulous - thank you so much.

4

Unfortunately, while this has made it possible to reproduce the bug in a development build, I can’t find any way of figuring out what the actual problem is. If I connect to the process while it’s locked up and break in, it’s in ‘external code’ with about 50 active threads and no indication of what or where it’s going wrong. Unplugging the PS4 controller unsticks it and allows the game to continue running.

Would it be possible for someone at Unity to take a look? This is going to be a blocker for us releasing, certainly on Switch (where it happens after the switch has been asleep for a bit), and there doesn’t seem to be anything I can do here to fix it. Thanks!

5

If, when you’re in that state, click “Debug” → Save Dump As… and send it to me, I could take a look.

6

Fantastic; will do!

7

I have a dump file to share - where’s the best place to put it?

Also, looking at the log file there’s nothing from before the lock-up, and the only thing after is:

Joystick disconnected (“Wireless Controller”).
<RI.Hid> Failed to create device file:
2 The system cannot find the file specified.

NOTE: The controller is listed as ‘wireless controller’ but is actually connected via USB (the computer’s bluetooth is turned off)

8

I’d suggest uploading it to some cloud storage (like dropbox or google drive) and send me the link. You can put it in a .zip/.7z file too - that will greatly reduce its size as dumps compress really well.

9

I’ve passed a link via a conversation; let me know if you need anything else.

10

I don’t think I received it I’m afraid.

11

Okay, thanks - I just started another conversation with the same link. Let me know if you get it!

12

Yes, got it now.

13

From what I can tell, it might be getting stuck in an infinite loop either in Rewired_Windows.dll or our API it calls (Unity - Scripting API: Windows.Input.ForwardRawInput). But I cannot tell for sure just by looking at a process snapshot during a single point in time. Could you capture another dump for me, wait for the game to continue being stuck for about 30 seconds after the first dump capture completes and then capture a second dump? That should help me identify what exactly it gets stuck on.

You could also try temporarily removing Rewired from your project to check whether that is what gets stuck.

14

Thanks for that. We DID try disabling Rewired and the problem DID go away, but obviously because that crippled the entire game and meant we weren’t polling input at all we couldn’t be sure whether the problem was in the DLL or in Input in general. I shall prepare the dumps ASAP. Thanks again!

15

This was fixed in Rewired 1.1.45.0. This was an edge case of the DualShock 4 and DualSense Extended Support code continuing to receive IMU input events from Raw Input while Unity prevents Rewired from updating and processing the event buffers because Update in Background is disabled.

16

Thanks for that - I shall test the update immediately!

17

Initially, updating to 1.1.45.0 seemed to fix the issue in Unity 2021.3.7f1. However, since updating to 2021.3.28f1, the problem has recurred. We can’t be entirely sure it’s related to the update, but we tested it pretty thoroughly at the time and couldn’t get it to happen.

I have a dump file from a development build:

https://drive.google.com/file/d/1Jp4KOD4IQ00PlfX_Wpr0Q1cx9ZeMaamr/view?usp=sharing

I’m planning to roll back to 7f1 and try another build. The problem is that we absolutely have to roll forward eventually to support console SDKs.

18

Minor Unity updates would have no impact at all on Rewired. Only in the case of a major update (2021 to 2022) would it impact Rewired and require installation of the matching Rewired version.

The 1.1.45.0 update implemented ring buffers for these events so there is no possibility of the same thing happening as before. If events continue to come in while the application is not in focus while Rewired is not allowed to update to deal with those events, the older events will simply be discarded when the new ones are added. In previous versions, a list was being used which was continuously expanding causing the memory footprint of the application to grow out of control.

Disable Enhanced Device Support and Dual Shock IMU events (the massive spamming of IMU events is what caused the buffers to grow out of control) will not be sent by Raw Input. Then, if your application still crashes, it’s most likely not being caused by Rewired.

19

Here’s the full callstack from the latest dump file:

UnityPlayer.dll!memcpy_repmovs()
UnityPlayer.dll!MemoryManager::ReallocateFallbackToAllocateDeallocate(void * ptr, unsigned __int64 size, unsigned __int64 align, const MemLabelId & label, AllocateOptions allocateOptions, const char * file, int line)
UnityPlayer.dll!MemoryManager::Reallocate(void * ptr, unsigned __int64 size, unsigned __int64 align, const MemLabelId & label, AllocateOptions allocateOptions, const char * file, int line)
UnityPlayer.dll!realloc_internal(void * ptr, unsigned __int64 size, unsigned __int64 align, const MemLabelId & label, AllocateOptions allocateOptions, const char * file, int line)
[Inline Frame] UnityPlayer.dll!dynamic_array_detail::dynamic_array_data::reallocate(void *)
[Inline Frame] UnityPlayer.dll!dynamic_array_detail::dynamic_array_data::reserve_impl(unsigned __int64)
UnityPlayer.dll!dynamic_array_detail::dynamic_array_data::resize_buffer(unsigned __int64 size, ResizePolicy resizePolicy, unsigned __int64 type_size, unsigned __int64 align)
UnityPlayer.dll!dynamic_array<unsigned char,0>::resize_buffer_nocheck(unsigned __int64 size, ResizePolicy resizePolicy)
[Inline Frame] UnityPlayer.dll!dynamic_array<unsigned char,0>::resize_buffer(unsigned __int64)
[Inline Frame] UnityPlayer.dll!dynamic_array<unsigned char,0>::resize_uninitialized(unsigned __int64 size, ResizePolicy)
UnityPlayer.dll!RawInputQueue::QueueInput(unsigned int * rawInputHeaderIndices, unsigned int * rawInputDataIndices, unsigned int indicesSize, unsigned char * rawInputData, unsigned int rawInputDataSize)
UnityPlayer.dll!ForwardRawInput(unsigned int * rawInputHeaderIndices, unsigned int * rawInputDataIndices, unsigned int indicesSize, unsigned char * rawInputData, unsigned int rawInputDataSize)
UnityPlayer.dll!Input_CUSTOM_ForwardRawInputImpl(void * rawInputHeaderIndices, void * rawInputDataIndices, unsigned int indicesCount, void * rawInputData, unsigned int rawInputDataSize)
UnityEngine.CoreModule.dll!UnityEngine.Windows.Input.ForwardRawInputImpl
UnityEngine.CoreModule.dll!UnityEngine.Windows.Input.ForwardRawInput
Rewired_Windows_Functions.dll!Rewired.Internal.Windows.Functions.ForwardRawInput
Assembly-CSharp.dll!Rewired.Utils.ExternalTools.WindowsStandalone_ForwardRawInput
Rewired_Windows.dll!gEPJIdTJwmJwoXPlEpTuwVGlPAjJ.LoHXgvXRgibliPxrEmxGBIIvHlco
Rewired_Windows.dll!gEPJIdTJwmJwoXPlEpTuwVGlPAjJ.JVmPHZFvNwKTBNQvnFdvvEMoBQvDA
Rewired_Windows.dll!gEPJIdTJwmJwoXPlEpTuwVGlPAjJ.ufXhCfVwREvFzeIKHBjAtIunzaCc
Rewired_Windows.dll!eiAEQUagfGTjXRjdiGwJDzSBPdYSA.ufXhCfVwREvFzeIKHBjAtIunzaCc
Rewired_Windows.dll!ZMqjCYehEKqIQMMhejVBKtVTTPMf.KzAQeNNAclKkdLhTuBkrNoRVPKzi
Rewired_Windows.dll!ZMqjCYehEKqIQMMhejVBKtVTTPMf.KzAQeNNAclKkdLhTuBkrNoRVPKzi
user32.dll!UserCallWinProcCheckWow()
user32.dll!DispatchMessageWorker()
[Inline Frame] UnityPlayer.dll!TranslateAndDispatchEventMsg(const tagMSG &)
UnityPlayer.dll!TranslateAndDispatchFilteredMessage(const tagMSG & msg)
UnityPlayer.dll!MainMessageLoop()
UnityPlayer.dll!UnityMainImpl(HINSTANCE__ * hInst, HINSTANCE__ * hPrev, wchar_t * szCmdLine, int nCmdShow)
UnityPlayer.dll!UnityMain(HINSTANCE__ * hInst, HINSTANCE__ * hPrev, wchar_t * szCmdLine, int nCmdShow)
[Inline Frame] NSGP.exe!invoke_main()
NSGP.exe!__scrt_common_main_seh()
kernel32.dll!BaseThreadInitThunk()
ntdll.dll!RtlUserThreadStart()

Looks like the same issue to me. It’s very weird that it doesn’t happen in Uinty 2021.3.7f1 but does in 2021.3.28f1. These are HID events that are coming in. In this particular dump file, 22 events are being forwarded.

20

I had not seen the previous dump in this thread, so the information I provided previously actually wasn’t relevant to this issue even though the symptoms and reproduction steps are the same. What’s happening here involves the forwarding of Raw Input events to Unity. This occurs immediately after Raw Input events are delivered to Rewired’s message window. If this is the same stack as before, the changes I made in 1.1.45.0 would not have fixed it. I changed how IMU events are stored to avoid buffers growing out of control which can sometimes lead to the application stalling out seemingly forever when the gigantic buffer is finally processed, but this does not involve buffering of actual Raw Input events (Raw Input events would become invalid at the end of the cycle). Those IMU events are just a bunch of structs created from the information contained in the Raw Input event. That code is unrelated to this.

The application being out of focus with or without Run in Background enabled would not have any effect on any of this Raw Input forwarding code because this code is not executing in the Unity Update function but rather the moment the event is delivered to the window. Unity is handling the Windows message pump for the application so it should run regardless of whether or not Run in Background is true. I don’t see how the steps you describe could lead to corruption of data or indices forwarded to Unity to result in an infinite loop when the data is processed.

The only thing I can do is attempt to reproduce it and log all the raw data and indices sent to Unity and see if something looks wrong in the data.