How to debug a hang after splash screen

About 5% of the time, my application will hang forever, showing a solid gray screen after the splash screen. This does not happen with the Blank HDRP template.

Why is this happening?

Some attempts:

  • Build with script debugging enabled. Attach debugger in vscode when it hangs.
  • Result: (unable to see the current stack information when pausing, debugger doesn’t appear to be working correctly.)
  • As a test to see if my debugging setup was broken, I added a while (true) {} loop to my startup to deliberately introduce a hang, within C#. Attaching the debugger, and hitting the pause button, it breaks immediately on the while (true) {} loop as expected. Could it be hanging in native code somewhere?
  • Look at log file.
  • Result: Nothing interesting. Running diff, the log files are almost exactly identical to the successful run with no hang, except some Vulkan device id.
  • Attach gdb debugger. sudo gdb -p 39277
  • Result: Below is the result of info threads command within gdb, plus the current stack trace on thread 1, using the bt command from gdb.
(gdb) info threads
  Id   Target Id                                           Frame
* 1    Thread 0x78a065b3ef80 (LWP 39277) "view.x86_64"     syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
  2    Thread 0x78a03ffff640 (LWP 39278) "Profiler.Dispat" syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
  3    Thread 0x78a03f7fe640 (LWP 39279) "AssetGarbageCol" syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
  4    Thread 0x78a03effd640 (LWP 39280) "AssetGarbageCol" syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
  5    Thread 0x78a03e7fc640 (LWP 39281) "AssetGarbageCol" syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
  6    Thread 0x78a03dffb640 (LWP 39282) "AssetGarbageCol" syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
  7    Thread 0x78a03d7fa640 (LWP 39283) "AssetGarbageCol" syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
  8    Thread 0x78a03cff9640 (LWP 39284) "AssetGarbageCol" syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
  9    Thread 0x78a03c7f8640 (LWP 39285) "AssetGarbageCol" syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
  10   Thread 0x78a03bff7640 (LWP 39286) "AssetGarbageCol" syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
  11   Thread 0x78a03b7f6640 (LWP 39287) "AssetGarbageCol" syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
  12   Thread 0x78a03aff5640 (LWP 39288) "AssetGarbageCol" syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
  13   Thread 0x78a03a7f4640 (LWP 39289) "AssetGarbageCol" syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
  14   Thread 0x78a039ff3640 (LWP 39290) "AssetGarbageCol" syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
  15   Thread 0x78a0397f2640 (LWP 39291) "AssetGarbageCol" syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
  16   Thread 0x78a038ff1640 (LWP 39292) "AssetGarbageCol" syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
  17   Thread 0x78a0387f0640 (LWP 39293) "AssetGarbageCol" syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
  18   Thread 0x78a037fef640 (LWP 39294) "AssetGarbageCol" syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
  19   Thread 0x78a0377ee640 (LWP 39295) "AssetGarbageCol" syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
  20   Thread 0x78a036fed640 (LWP 39296) "AssetGarbageCol" syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
  21   Thread 0x78a0367ec640 (LWP 39297) "AssetGarbageCol" syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
  22   Thread 0x78a035feb640 (LWP 39298) "SDLTimer"        __futex_abstimed_wait_common64 (private=<optimized out>, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x1b9e350)
    at ./nptl/futex-internal.c:57
  23   Thread 0x78a033fff640 (LWP 39320) "Finalizer"       __futex_abstimed_wait_common64 (private=<optimized out>, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x78a034bb74b8)
    at ./nptl/futex-internal.c:57
  24   Thread 0x78a033dfe640 (LWP 39321) "Debugger agent"  __libc_recv (flags=<optimized out>, len=11, buf=0x78a033dfc1e5, fd=16) at ../sysdeps/unix/sysv/linux/recv.c:28
  25   Thread 0x78a031e70640 (LWP 39322) "Job.Worker 0"    syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
  26   Thread 0x78a03166f640 (LWP 39323) "Job.Worker 1"    syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
  27   Thread 0x78a030e6e640 (LWP 39324) "Job.Worker 2"    syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
  28   Thread 0x789f97fff640 (LWP 39325) "Job.Worker 3"    syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
  29   Thread 0x789f977fe640 (LWP 39326) "Job.Worker 4"    syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
  30   Thread 0x789f96ffd640 (LWP 39327) "Job.Worker 5"    syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
  31   Thread 0x789f967fc640 (LWP 39328) "Job.Worker 6"    syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
  32   Thread 0x789f95ffb640 (LWP 39329) "Job.Worker 7"    syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
  33   Thread 0x789f957fa640 (LWP 39330) "Job.Worker 8"    syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
  34   Thread 0x789f94ff9640 (LWP 39331) "Job.Worker 9"    syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
  35   Thread 0x789f947f8640 (LWP 39332) "Job.Worker 10"   syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
  36   Thread 0x789f93ff7640 (LWP 39333) "Job.Worker 11"   syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
  37   Thread 0x789f937f6640 (LWP 39334) "Job.Worker 12"   syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
  38   Thread 0x789f92ff5640 (LWP 39335) "Job.Worker 13"   syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
  39   Thread 0x789f927f4640 (LWP 39336) "Job.Worker 14"   syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
  40   Thread 0x789f91ff3640 (LWP 39337) "Job.Worker 15"   syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
  41   Thread 0x789f917f2640 (LWP 39338) "Job.Worker 16"   syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
  42   Thread 0x789f90ff1640 (LWP 39339) "Job.Worker 17"   syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
  43   Thread 0x789f7ffff640 (LWP 39340) "Job.Worker 18"   syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
  44   Thread 0x789f7f7fe640 (LWP 39341) "Background Job." syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
  45   Thread 0x789f7effd640 (LWP 39342) "Background Job." syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
  46   Thread 0x789f7e7fc640 (LWP 39343) "Background Job." syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
  47   Thread 0x789f7dffb640 (LWP 39344) "Background Job." syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
  48   Thread 0x789f7d7fa640 (LWP 39345) "Background Job." syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
  49   Thread 0x789f7cff9640 (LWP 39346) "Background Job." syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
  50   Thread 0x789f7c7f8640 (LWP 39347) "Background Job." syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
  51   Thread 0x789f7bff7640 (LWP 39348) "Background Job." syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
--Type <RET> for more, q to quit, c to continue without paging--
  52   Thread 0x789f7b7f6640 (LWP 39349) "Background Job." syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
  53   Thread 0x789f7aff5640 (LWP 39350) "Background Job." syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
  54   Thread 0x789f7a7f4640 (LWP 39351) "Background Job." syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
  55   Thread 0x789f79ff3640 (LWP 39352) "Background Job." syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
  56   Thread 0x789f797f2640 (LWP 39353) "Background Job." syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
  57   Thread 0x789f78ff1640 (LWP 39354) "Background Job." syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
  58   Thread 0x789f787f0640 (LWP 39355) "Background Job." syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
  59   Thread 0x789f77fef640 (LWP 39356) "Background Job." syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
  60   Thread 0x789f777ee640 (LWP 39357) "BatchDeleteObje" syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
  61   Thread 0x789f76fed640 (LWP 39358) "Loading.AsyncRe" syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
  62   Thread 0x789f547ff640 (LWP 39360) "view.x86_64"     0x000078a061ee57f8 in __GI___clock_nanosleep (clock_id=clock_id@entry=0, flags=flags@entry=0, req=0x789f547fd550, rem=0x789f547fd540)
    at ../sysdeps/unix/sysv/linux/clock_nanosleep.c:78
  63   Thread 0x789f53ffe640 (LWP 39361) "[vkrt] Analysis" __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x789f53ffc570, op=393, expected=0, futex_word=0x789f2c000f5c)
    at ./nptl/futex-internal.c:57
  64   Thread 0x789f537fd640 (LWP 39362) "[vkcf] Analysis" __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x789f20000f58) at ./nptl/futex-internal.c:57
  65   Thread 0x789f52ffc640 (LWP 39363) "view.x86_64"     __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x789f24010f68) at ./nptl/futex-internal.c:57
  66   Thread 0x789f527fb640 (LWP 39364) "view.x86_64"     __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x789f527f9470, op=393, expected=0, futex_word=0x789f18000f58)
    at ./nptl/futex-internal.c:57
  67   Thread 0x789f1f7ff640 (LWP 39365) "Vulkan Submissi" syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
  68   Thread 0x789f1effe640 (LWP 39366) "UnityGfxDeviceW" 0x000078a061f18bcf in __GI___poll (fds=0x789f159487c0, nfds=1, timeout=10) at ../sysdeps/unix/sysv/linux/poll.c:29
  69   Thread 0x789f1e7fd640 (LWP 39368) "threaded-ml"     0x000078a061f18bcf in __GI___poll (fds=0x789ef800a3b0, nfds=3, timeout=986) at ../sysdeps/unix/sysv/linux/poll.c:29
  70   Thread 0x789f5059c640 (LWP 39369) "Audio PulseAudi" __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x324f71c) at ./nptl/futex-internal.c:57
  71   Thread 0x789f5746c640 (LWP 39370) "Audio Stream Th" 0x000078a061ee57f8 in __GI___clock_nanosleep (clock_id=clock_id@entry=0, flags=flags@entry=0, req=req@entry=0x789f5746a5b0, rem=rem@entry=0x0)
    at ../sysdeps/unix/sysv/linux/clock_nanosleep.c:78
  72   Thread 0x789ee9596640 (LWP 39399) "Thread Pool Wor" __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x78a034bb216c) at ./nptl/futex-internal.c:57
(gdb) bt
#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1  0x000078a064afe83e in UnityClassic::Baselib_SystemFutex_Wait(int*, int, unsigned int) () from /home/brian/View-Shipping/View-DevMono/UnityPlayer.so
#2  0x000078a064851976 in GfxDeviceWorker::WaitForSignal() () from /home/brian/View-Shipping/View-DevMono/UnityPlayer.so
#3  0x000078a063d7bc11 in GfxDeviceClient::FinishRendering() () from /home/brian/View-Shipping/View-DevMono/UnityPlayer.so
#4  0x000078a0637a1cf0 in CleanupEngine() () from /home/brian/View-Shipping/View-DevMono/UnityPlayer.so
#5  0x000078a0637c8ce4 in PlayerCleanup(bool) () from /home/brian/View-Shipping/View-DevMono/UnityPlayer.so
#6  0x000078a063d16566 in PlayerMain(int, char**) () from /home/brian/View-Shipping/View-DevMono/UnityPlayer.so
#7  0x000078a061e29d90 in __libc_start_call_main (main=main@entry=0x2010e0 <main>, argc=argc@entry=1, argv=argv@entry=0x7fff0ed05fa8) at ../sysdeps/nptl/libc_start_call_main.h:58
#8  0x000078a061e29e40 in __libc_start_main_impl (main=0x2010e0 <main>, argc=1, argv=0x7fff0ed05fa8, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7fff0ed05f98)
    at ../csu/libc-start.c:392
#9  0x0000000000201029 in _start ()

Some details
Unity engine version: 2022.3.34f1
Scripting backend: Mono (but also happens on IL2CPP)
Render pipeline: HDRP
OS version: Ubuntu 22.04
GPU: Nvidia RTX 4070Ti
Driver version: 535.183.01

What I am looking for: Are there any tools I can use to debug this? Maybe additional gdb commands I can type to get a better understanding of what the program is hanging on? Any advice would be very helpful.