Random crashes (seems to be related to renderer)

Hi,

We’ve been getting random crashes on some play tester’s machines and we’ve been struggling to debug. it happens when the player works around in a room and they all crash on the same line in the stacktrace. we were able to locate the crash source using winDBG

Microsoft (R) Windows Debugger Version 10.0.22415.1002 AMD64
Copyright (c) Microsoft Corporation. All rights reserved.


Loading Dump File [C:\Users\Leon\Desktop\Crashes\Crash_2021-08-17_014728425\crash.dmp]
User Mini Dump File: Only registers, stack and portions of memory are available


************* Path validation summary **************
Response                         Time (ms)     Location
Deferred                                       srv*
Symbol search path is: srv*
Executable search path is:
Windows 10 Version 19043 MP (16 procs) Free x64
Product: WinNt, suite: SingleUserTS Personal
Edition build lab: 19041.1.amd64fre.vb_release.191206-1406
Machine Name:
Debug session time: Mon Aug 16 21:47:28.000 2021 (UTC - 4:00)
System Uptime: not available
Process Uptime: 0 days 0:07:00.000
................................................................
.......................................
This dump file has an exception of interest stored in it.
The stored exception information can be accessed via .ecxr.
(a38.5f80): Access violation - code c0000005 (first/second chance not available)
For analysis of this file, run !analyze -v
ntdll!NtWaitForMultipleObjects+0x14:
00007ffd`db9cd974 c3              ret
0:011> !analyze -v
ERROR: FindPlugIns 8007007b
*******************************************************************************
*                                                                             *
*                        Exception Analysis                                   *
*                                                                             *
*******************************************************************************

*** WARNING: Unable to verify checksum for mono-2.0-bdwgc.dll
*** WARNING: Unable to verify timestamp for nvwgf2umx.dll
*** WARNING: Unable to verify timestamp for atidxx64.dll
*** WARNING: Unable to verify timestamp for amdihk64.dll
*** WARNING: Unable to verify timestamp for graphics-hook64.dll

KEY_VALUES_STRING: 1

    Key  : AV.Fault
    Value: Execute

    Key  : Analysis.CPU.mSec
    Value: 4515

    Key  : Analysis.DebugAnalysisManager
    Value: Create

    Key  : Analysis.Elapsed.mSec
    Value: 15761

    Key  : Analysis.Init.CPU.mSec
    Value: 405

    Key  : Analysis.Init.Elapsed.mSec
    Value: 10664

    Key  : Analysis.Memory.CommitPeak.Mb
    Value: 256

    Key  : Timeline.Process.Start.DeltaSec
    Value: 420

    Key  : WER.OS.Branch
    Value: vb_release

    Key  : WER.OS.Timestamp
    Value: 2019-12-06T14:06:00Z

    Key  : WER.OS.Version
    Value: 10.0.19041.1

    Key  : WER.Process.Version
    Value: 9.18.944.0


CONTEXT:  (.ecxr)
rax=0000000000000000 rbx=0000000000030013 rcx=000001e975f55d60
rdx=000000000000000b rsi=000001e975af2908 rdi=000000341cdff980
rip=0000000000000001 rsp=000000341cdff4f8 rbp=000000341cdff850
r8=0000000000030013  r9=0000000000000001 r10=00007ffd743911f0
r11=000000341cdff570 r12=0000000000000000 r13=0000000000000012
r14=000001e950c10010 r15=000001e97543f9a0
iopl=0         nv up ei pl nz na pe nc
cs=0033  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00010202
00000000`00000001 ??              ???
Resetting default scope

EXCEPTION_RECORD:  (.exr -1)
ExceptionAddress: 0000000000000001
   ExceptionCode: c0000005 (Access violation)
  ExceptionFlags: 00000000
NumberParameters: 2
   Parameter[0]: 0000000000000008
   Parameter[1]: 0000000000000001
Attempt to execute non-executable address 0000000000000001

PROCESS_NAME:  Keyword.exe

EXECUTE_ADDRESS: 1

FAILED_INSTRUCTION_ADDRESS:
+0
00000000`00000001 ??              ???

ERROR_CODE: (NTSTATUS) 0xc0000005 - The instruction at 0x%p referenced memory at 0x%p. The memory could not be %s.

EXCEPTION_CODE_STR:  c0000005

EXCEPTION_PARAMETER1:  0000000000000008

EXCEPTION_PARAMETER2:  0000000000000001

STACK_TEXT:
00000034`1cdff4f8 00007ffd`733368d7     : 00000000`0100010f 00000000`0100010f 00000000`00030013 00000000`00000000 : 0x1
00000034`1cdff500 00000000`0100010f     : 00000000`0100010f 00000000`00030013 00000000`00000000 ff000201`00000001 : UnityPlayer!BatchRenderer::RenderBatch+0x77
00000034`1cdff508 00000000`0100010f     : 00000000`00030013 00000000`00000000 ff000201`00000001 00080000`000800ff : 0x100010f
00000034`1cdff510 00000000`00030013     : 00000000`00000000 ff000201`00000001 00080000`000800ff 00000000`00000000 : 0x100010f
00000034`1cdff518 00000000`00000000     : ff000201`00000001 00080000`000800ff 00000000`00000000 000001e9`00000001 : 0x30013


FAULTING_SOURCE_LINE:  C:\buildslave\unity\build\Runtime\Camera\RenderLoops\BatchRenderer.cpp

FAULTING_SOURCE_FILE:  C:\buildslave\unity\build\Runtime\Camera\RenderLoops\BatchRenderer.cpp

FAULTING_SOURCE_LINE_NUMBER:  367

FAULTING_SOURCE_CODE:
No source found for 'C:\buildslave\unity\build\Runtime\Camera\RenderLoops\BatchRenderer.cpp'


SYMBOL_NAME:  unityplayer!BatchRenderer::RenderBatch+77

MODULE_NAME: UnityPlayer

IMAGE_NAME:  UnityPlayer.dll

STACK_COMMAND:  ~11s ; .ecxr ; kb

FAILURE_BUCKET_ID:  SOFTWARE_NX_FAULT_c0000005_UnityPlayer.dll!BatchRenderer::RenderBatch

OS_VERSION:  10.0.19041.1

BUILDLAB_STR:  vb_release

OSPLATFORM_TYPE:  x64

OSNAME:  Windows 10

IMAGE_VERSION:  2020.3.6.35766

FAILURE_ID_HASH:  {e97d9fd0-5b5b-1def-fa2f-1fa6daf03059}

Followup:     MachineOwner
---------

we don’t have access to the source file, so this is where our investigation halts.

the machine that produced this crash is fairly highend(AMD Ryzen 9 4900H NVIDIA RTX 2060 16 GB RAM).
We’ve found that turning off Graphics Job in player settings help, but not always. the crash still happens. i also don’t think that’s a fix.

the project is fairly big and since it happens randomly i couldn’t come up with a reproducible project. but let me know if there are additonal info i can provide to assist tracking down the issue.

Can you run the “k” command to print the callstack in WinDBG?

this is what i get:

 # Child-SP          RetAddr               Call Site
00 00000034`1cdfe048 00007ffd`d937cab0     ntdll!NtWaitForMultipleObjects+0x14
01 00000034`1cdfe050 00007ffd`d937c9ae     KERNELBASE!WaitForMultipleObjectsEx+0xf0
02 00000034`1cdfe340 00007ffd`7320329d     KERNELBASE!WaitForMultipleObjects+0xe
03 00000034`1cdfe380 00000000`00000000     UnityPlayer!winutils::ExternalCrashHandler::HandleCrash+0x26d [C:\buildslave\unity\build\PlatformDependent\Win\ExternalCrashHandler.cpp @ 278]

however not sure if it is actually on the thread you are interested in.

anyways, i’ve attached the dmp folder which should tell more.

7424201–908648–Crash_2021-08-17_014728425.7z (96.3 KB)

Would you be able to capture a full crash dump? The automatic dumps that get generated are stripped down for reduced size, but they don’t contain full crash information, unfortunately. You can generate a full dump by configuring Windows Error Reporting: Collecting User-Mode Dumps - Win32 apps | Microsoft Learn

Use dumpType = 2.

thanks for the instructions, i’ll work on getting them

apologize in advance for large file size - the crash was easier to repro if we turn up texture resolution settings, hence it is over 10GB in size(1.4G after compression). i’ve uploaded to google drive here:
https://drive.google.com/file/d/1sQ3WW8mfMfz8m_xzYFAqo1t8JzGzOq3F/view?usp=sharing

No need to apologize. It’s expected that full size dumps are large. Let me look at it.

1 Like

Can you try disabling the SRP batcher (Unity - Manual: Scriptable Render Pipeline Batcher) and to see if the issue goes away?

we are not using scriptable render pipeline, so the option isnt available.

Would you be able to file a bug report? Unfortunately the dump doesn’t immediately make it clear to me how Unity gets into this situation, and it’s likely the issue will have to bounce between different teams to figure out (and we can’t exactly bounce a forum post).

understood, i will do this tonight.