Crashes in Open GLES graphics rendering on OnePlus mobiles

[LemanColt]

Hi!
We see multiple crash reports through the Unity Cloud Diagnostics occuring on OnePlus 6, 6T, 7T, 7Pro and 7TPro mobiles.
The crash seems to be in the graphics rendering area. And we have been trying to dig into this the past few days without being able to figure out why it crashes
We are using Unity 2019.2.17f1. OpenGLES 2 and 3 enabled.

Would greatly appreciate help on this topic. Anyone that have any clues? Or anything we should try?

Stack trace:

Native StackTrace:

Thread 0 (crashed)
0 libunity.so 0x00000073844bf30c CrashReporting::Android::SignalHandler(int, siginfo*, void*) (CrashReporter.cpp:354)
1 libunity.so 0x0000007384851c6c NativeRuntimeException::signal_handler(int, siginfo*, void*) (NativeRuntimeException.cpp:757)
2 app_process64 0x0000005c01b88dcc
3 [vdso] 0x00000074854b8690
4 libc.so 0x000000748108c4cc
5 libunity.so 0x0000007384902e24 ApiGLES::ClearBufferSubData(unsigned int, gl::BufferTarget, long, long) (ApiGLES.cpp:2288)
6 libunity.so 0x00000073848e4714 BufferManagerGLES::AcquireBuffer(unsigned long, DataBufferGLES::BufferUsage, bool) (DataBuffersGLES.cpp:344)
7 libunity.so 0x00000073848e3a3c BufferGLES::EnsureBuffer(DataBufferGLES*&, unsigned long, DataBufferGLES::BufferUsage) (BuffersGLES.h:224)
8 libunity.so 0x00000073848f1764 GfxDeviceGLES::SetComputeBufferData(GfxBuffer*, void const*, unsigned long, unsigned long) (BuffersGLES.h:44)
9 libunity.so 0x0000007384a5ef6c GfxDevice::UpdateComputeSkinPoseBuffer(GfxBuffer*, MatrixArrayJobOutput*) (SharedObject.h:167)
10 libunity.so 0x00000073845d97f4 GfxDeviceWorker::RunCommand(ThreadedStreamBuffer&) (GfxDeviceWorker.cpp:0)
11 libunity.so 0x00000073845da5b4 GfxDeviceWorker::RunExt(ThreadedStreamBuffer&) (GfxDeviceWorker.cpp:528)
12 libunity.so 0x00000073845d3048 GfxDeviceWorker::RunGfxDeviceWorker(void*) (GfxDeviceWorker.cpp:492)
13 libunity.so 0x00000073847a0a84 Thread::RunThreadWrapper(void*) (Thread.cpp:81)
14 libc.so 0x00000074810f3894
15 libc.so 0x0000007481091b70
16 . 0x0000000000000000

[florianpenzkofer]

Since the crash happens in GPU skinning a workaround could be to turn that off.

Sounds like cannot reproduce the crash yourself?

[Wielok]

Hi
We are experiecing similar bug and we can’t find the way to fix it or even reproduce it appears randomly. We tried turning off GPU skinning, diffrent unity versions 2019.3.5f, 2019.3.15f and 32 bit builds. This is our stack trace from unity dashboard:

1 libunity.so 0x0000007f612dd218 ApiGLES::ClearBufferSubData(unsigned int, gl::BufferTarget, long, long)
2 libunity.so 0x0000007f612bdb24 BufferManagerGLES::AcquireBuffer(unsigned long, DataBufferGLES::BufferUsage, bool)
3 libunity.so 0x0000007f612bcd9c BufferGLES::EnsureBuffer(DataBufferGLES*&, unsigned long, DataBufferGLES::BufferUsage)
4 libunity.so 0x0000007f612caef0 GfxDeviceGLES::SetComputeBufferData(GfxBuffer*, void const*, unsigned long, unsigned long)
5 libunity.so 0x0000007f61436c14 GfxDevice::UpdateComputeSkinPoseBuffer(GfxBuffer*, MatrixArrayJobOutput*)
6 libunity.so 0x0000007f6176fae8 GfxDeviceWorker::RunCommand(ThreadedStreamBuffer&)
7 libunity.so 0x0000007f61771818 GfxDeviceWorker::RunExt(ThreadedStreamBuffer&)
8 libunity.so 0x0000007f61768cd0 GfxDeviceWorker::RunGfxDeviceWorker(void*)
9 libunity.so 0x0000007f611d5f9c Thread::RunThreadWrapper(void*)

Did you find a way to fix it?
Greetings
Wielok

[tessellation]

@Wielok

Hi
We are experiecing similar bug and we can’t find the way to fix it or even reproduce it appears randomly. We tried turning off GPU skinning, diffrent unity versions 2019.3.5f, 2019.3.15f and 32 bit builds. This is our stack trace from unity dashboard:

We are experiencing the same issues in Unity 2018.4.25f1, reported by users on Galaxy S9 Android 10. @Wielok did you find that turning off GPU skinning fixed the problem?

[ImpossibleRobert]

I got the same in 2020.1 on Android just right now, when quickly hovering over many different UI canvases in VR. It happens always @florianpenzkofer if you need an APK to debug, I have one.

When deactivating graphics jobs, it still happens but the backtrace changes to:

2020.08.21 22:21:54.242 12393 12499 Error CRASH Version ‘2020.1.3f1 (cf5c4788e1d8)’, Build type ‘Development’, Scripting Backend ‘mono’, CPU ‘armeabi-v7a’
2020.08.21 22:21:54.242 12393 12499 Error CRASH Build fingerprint: ‘oculus/vr_monterey/monterey:7.1.1/NGI77B/821490.13210.0:user/release-keys’
2020.08.21 22:21:54.242 12393 12499 Error CRASH Revision: ‘0’
2020.08.21 22:21:54.242 12393 12499 Error CRASH ABI: ‘arm’
2020.08.21 22:21:54.242 12393 12499 Error CRASH Timestamp: 2020-08-21 22:21:54+0200
2020.08.21 22:21:54.242 12393 12499 Error CRASH pid: 12393, tid: 12499, name: Thread-8 >>> com.wetzold.travrsal <<<
2020.08.21 22:21:54.242 12393 12499 Error CRASH uid: 10153
2020.08.21 22:21:54.242 12393 12499 Error CRASH signal 11 (SIGSEGV), code 1 (SEGV_MAPERR), fault addr 0x0
2020.08.21 22:21:54.242 12393 12499 Error CRASH Cause: null pointer dereference
2020.08.21 22:21:54.242 12393 12499 Error CRASH r0 00000000 r1 89fd11c0 r2 00000100 r3 00000000
2020.08.21 22:21:54.243 12393 12499 Error CRASH r4 00000180 r5 00000000 r6 89fd1180 r7 00000180
2020.08.21 22:21:54.243 12393 12499 Error CRASH r8 6ae09f10 r9 cb97f790 r10 ce5d7080 r11 0000001f
2020.08.21 22:21:54.243 12393 12499 Error CRASH ip f0c2a624 sp cb97f578 lr d5e49e70 pc f0bb8858
2020.08.21 22:21:54.243 12393 12499 Error CRASH
2020.08.21 22:21:54.243 12393 12499 Error CRASH backtrace:
2020.08.21 22:21:54.243 12393 12499 Error CRASH #00 pc 00017858 /system/lib/libc.so (memcpy+108) (BuildId: fde26210202025ebbef572bc1778c23b)
2020.08.21 22:21:54.243 12393 12499 Error CRASH #01 pc 008a8e6c /data/app/com.wetzold.travrsal-2/lib/arm/libunity.so (GfxDeviceGLES:rawBuffersBatchMode(GfxBatchHeader const&)+180) (BuildId: d54fd529171d29e7111eaa7645905909ccd6153b)
2020.08.21 22:21:54.243 12393 12499 Error CRASH #02 pc 00e2c82f /data/app/com.wetzold.travrsal-2/lib/arm/libunity.so (GfxDeviceWorker::RunCommand(ThreadedStreamBuffer&)+21478) (BuildId: d54fd529171d29e7111eaa7645905909ccd6153b)
2020.08.21 22:21:54.243 12393 12499 Error CRASH #03 pc 00e2cf13 /data/app/com.wetzold.travrsal-2/lib/arm/libunity.so (GfxDeviceWorker::RunExt(ThreadedStreamBuffer&)+26) (BuildId: d54fd529171d29e7111eaa7645905909ccd6153b)
2020.08.21 22:21:54.243 12393 12499 Error CRASH #04 pc 00e2cedb /data/app/com.wetzold.travrsal-2/lib/arm/libunity.so (GfxDeviceWorker::Run()+98) (BuildId: d54fd529171d29e7111eaa7645905909ccd6153b)
2020.08.21 22:21:54.243 12393 12499 Error CRASH #05 pc 00e27207 /data/app/com.wetzold.travrsal-2/lib/arm/libunity.so (GfxDeviceWorker::RunGfxDeviceWorker(void*)+2) (BuildId: d54fd529171d29e7111eaa7645905909ccd6153b)
2020.08.21 22:21:54.243 12393 12499 Error CRASH #06 pc 0073b53b /data/app/com.wetzold.travrsal-2/lib/arm/libunity.so (Thread::RunThreadWrapper(void*)+342) (BuildId: d54fd529171d29e7111eaa7645905909ccd6153b)
2020.08.21 22:21:54.243 12393 12499 Error CRASH #07 pc 00047d83 /system/lib/libc.so (__pthread_start(void*)+22) (BuildId: fde26210202025ebbef572bc1778c23b)
2020.08.21 22:21:54.243 12393 12499 Error CRASH #08 pc 0001a035 /system/lib/libc.so (__start_thread+6) (BuildId: fde26210202025ebbef572bc1778c23b)
2020.08.21 22:22:00.293 12393 12393 Error CRASH other thread is trapped; signum = 11

[florianpenzkofer]

@ImpossibleRobert your call stack looks like it might be caused by SRP Batcher. DrawBuffersBatchMode is only used when SRP batcher is enabled. You can try to turn it off in the SRP settings asset to confirm this.

It’s probably not related to crash of the original poster.

I can’t easily debug APKs (and our QA won’t be able to confirm the fix and we can’t backport the fix), please create a bug report with repro project.

[florianpenzkofer]

@Wielok @tessellation Regarding the crash in ClearBufferSubData. I had a look at the Unity code and this could happen if glMapBuffer returns NULL. The most likely issue would be that we are running out of address space, which would make sense if it only happens in 32bit builds. Is there anything in logcat that would indicate that we are close to out-of-memory?

[tessellation]

@florianpenzkofer

@Wielok @tessellation Regarding the crash in ClearBufferSubData. I had a look at the Unity code and this could happen if glMapBuffer returns NULL. The most likely issue would be that we are running out of address space, which would make sense if it only happens in 32bit builds. Is there anything in logcat that would indicate that we are close to out-of-memory?

I switched to CPU-based skinning and the crashes went away. So perhaps the GPU is running out of memory?

[florianpenzkofer]

Could also be some kind of leak in our GPU skinning implementation.

[tessellation]

@florianpenzkofer

Could also be some kind of leak in our GPU skinning implementation.

Possibly. Let me give you some more information about how my app uses skinning. We currently don’t have any 3D characters, but we do have 2D characters and we use the older Anima2D system, not the new Unity 2D bone system. I’m guessing under the covers they both use the engine’s basic skinning system, which probably doesn’t care if the resulting mesh is 2D or 3D. Regardless, the mesh isn’t dynamic or anything and is relatively simple compared to typical 3D characters. We also usually only have 1 skinned character on the screen at any one time, so it’s not anything taxing to the system as far as I can tell.

[PatrickKa]

I’m also seeing this crash happening seemingly randomly. The app ran for less than 10 minutes. It started to drop frames and then freezed.

Unity 2018.4.23f1
IL2CPP, 64 Bit
Built-in render pipeline
Dev build
GPU Skinning enabled
VR, Oculus Quest 1

Log:

10-12 17:34:47.672 32597 32643 W Adreno-GSL: <gsl_ldd_control:553>: ioctl fd 55 code 0xc040094a (IOCTL_KGSL_GPU_COMMAND) failed: errno 35 Resource deadlock would occur
10-12 17:34:47.672 32597 32643 W Adreno-GSL: <log_gpu_snapshot:462>: panel.gpuSnapshotPath is not set.not generating user snapshot
10-12 17:34:47.673 2296 31685 I Vsync : Vsync 42365: 2 x 13.93 ms (71.82 Hz, at 749155.687)
10-12 17:34:47.674 2296 31685 I Clocks : SetSchedFifo( tid=32671, pri=1 ) succeeded
10-12 17:34:47.675 2296 31685 I Clocks : SetSchedFifo( tid=2529, pri=1 ) succeeded
10-12 17:34:47.685 32597 32643 D Unity : NativeCrashSerializer::EndReport() Success!
10-12 17:34:47.685 32597 32643 E CRASH : signal 11 (SIGSEGV), code 1 (SEGV_MAPERR), fault addr 0000000000000000
10-12 17:34:47.685 32597 32643 E CRASH : *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
10-12 17:34:47.685 32597 32643 E CRASH : Build type ‘Development’, Scripting Backend ‘il2cpp’, CPU ‘arm64-v8a’
10-12 17:34:47.685 32597 32643 E CRASH : Build fingerprint: ‘oculus/vr_monterey/monterey:7.1.1/NGI77B/904120.17180.0:user/release-keys’
10-12 17:34:47.685 32597 32643 E CRASH : Revision: ‘0’
10-12 17:34:47.686 32597 32643 E CRASH : pid: 32597, tid: 32643, name: Thread-5 >>> com.comp.name <<<
10-12 17:34:47.686 32597 32643 E CRASH : x0 0000000000000000 x1 0000000000000000 x2 0000000000000640 x3 0000000000000004
10-12 17:34:47.686 32597 32643 E CRASH : x4 0000000000000000 x5 0000000000000040 x6 000000000000003f x7 0000000000000000
10-12 17:34:47.686 32597 32643 E CRASH : x8 0000000000000000 x9 0000000000000004 x10 0000000000000001 x11 0000000000000000
10-12 17:34:47.686 32597 32643 E CRASH : x12 0000000000000005 x13 00000000000060a1 x14 00000000000000b7 x15 0000007f93b18810
10-12 17:34:47.686 32597 32643 E CRASH : x16 0000007f786b7f40 x17 0000007f93a4ff40 x18 00000000000000b6 x19 0000000000000680
10-12 17:34:47.686 32597 32643 E CRASH : x20 0000000000000002 x21 0000000000000fc2 x22 0000007f687cfe98 x23 0000007f487fd1b0
10-12 17:34:47.686 32597 32643 E CRASH : x24 0000000000000001 x25 0000007f687c51e8 x26 0000007f55fc02b0 x27 0000007f67200040
10-12 17:34:47.686 32597 32643 E CRASH : x28 f3860c71e6e6fb69 x29 0000007f66fff290 x30 0000007f77b4aa60
10-12 17:34:47.686 32597 32643 E CRASH : sp 0000007f66ffef20 pc 0000007f93a500a8 pstate 0000000040000000
10-12 17:34:47.686 32597 32643 E CRASH :
10-12 17:34:47.686 32597 32643 E CRASH : backtrace:
10-12 17:34:47.717 32597 32665 I VrApi : FPS=24,Prd=45ms,Tear=3,Early=0,Stale=3,VSnc=1,Lat=1,Fov=0,CPU4/GPU=4/4,2304/670MHz,OC=FF,TA=0/0/0,SP=N/N/N,Mem=1554MHz,Free=990MB,PSM=0,PLS=0,Temp=27.0C/0.0C,TW=0.00ms,App=0.00ms,GD=0.00ms,CPU&GPU=27.64ms,LCnt=1,GPU%=1.00,CPU%=0.13(W0.30),DSF=1.00
10-12 17:34:47.757 32597 32643 E CRASH : #00 pc 00000000000004ec [vdso] ()
10-12 17:34:47.758 32597 32643 E CRASH : #01 pc 000000000001b0a4 /system/lib64/libc.so (memset+356)
10-12 17:34:47.758 32597 32643 E CRASH : #02 pc 0000000001275a5c /data/app/com.comp.name-1/lib/arm64/libunity.so (_ZN7ApiGLES18ClearBufferSubDataEjN2gl12BufferTargetEll+44)
10-12 17:34:47.758 32597 32643 E CRASH : #03 pc 000000000128784c /data/app/com.comp.name-1/lib/arm64/libunity.so (_ZN17BufferManagerGLES13AcquireBufferEmN14DataBufferGLES11BufferUsageEb+564)
10-12 17:34:47.758 32597 32643 E CRASH : #04 pc 00000000012862b0 /data/app/com.comp.name-1/lib/arm64/libunity.so (_ZN10BufferGLES12EnsureBufferERP14DataBufferGLESmNS0_11BufferUsageE+104)
10-12 17:34:47.758 32597 32643 E CRASH : #05 pc 0000000001292eb8 /data/app/com.comp.name-1/lib/arm64/libunity.so (_ZN13GfxDeviceGLES20SetComputeBufferDataEP9GfxBufferPKvmm+132)
10-12 17:34:47.758 32597 32643 E CRASH : #06 pc 0000000000f70b10 /data/app/com.comp.name-1/lib/arm64/libunity.so (_ZN9GfxDevice27UpdateComputeSkinPoseBufferEP9GfxBufferP20MatrixArrayJobOutput+72)
10-12 17:34:47.758 32597 32643 E CRASH : #07 pc 00000000006bc5b8 /data/app/com.comp.name-1/lib/arm64/libunity.so (_ZN15GfxDeviceWorker10RunCommandER20ThreadedStreamBuffer+25236)
10-12 17:34:47.758 32597 32643 E CRASH : #08 pc 00000000006bd2e0 /data/app/com.comp.name-1/lib/arm64/libunity.so (_ZN15GfxDeviceWorker6RunExtER20ThreadedStreamBuffer+40)
10-12 17:34:47.758 32597 32643 E CRASH : #09 pc 00000000006bd2ac /data/app/com.comp.name-1/lib/arm64/libunity.so (_ZN15GfxDeviceWorker3RunEv+136)
10-12 17:34:47.758 32597 32643 E CRASH : #10 pc 00000000006b5f8c /data/app/com.comp.name-1/lib/arm64/libunity.so (_ZN15GfxDeviceWorker18RunGfxDeviceWorkerEPv+4)
10-12 17:34:47.758 32597 32643 E CRASH : #11 pc 0000000000b6fb5c /data/app/com.comp.name-1/lib/arm64/libunity.so (_ZN6Thread16RunThreadWrapperEPv+668)
10-12 17:34:47.758 32597 32643 E CRASH : #12 pc 0000000000076454 /system/lib64/libc.so (_ZL15__pthread_startPv+204)
10-12 17:34:47.758 32597 32643 E CRASH : #13 pc 000000000001e204 /system/lib64/libc.so (__start_thread+16)

What is this about?

Adreno-GSL: <gsl_ldd_control:553>: ioctl fd 55 code 0xc040094a (IOCTL_KGSL_GPU_COMMAND) failed: errno 35 Resource deadlock would occur

Kinda feels like a deadlock that freezes the application. Might be very related.

[PatrickKa]

@florianpenzkofer

Could also be some kind of leak in our GPU skinning implementation.

You can see in the log that at the time of the crash almost 1 GB of memory was still available and the app was only running like 10 minutes. This doesn’t look like a memory related crash at all.

[florianpenzkofer]

@PatrickKa Can you please create a bug report for this?

[spatialx]

We are also seeing this issue; most recently on Unity 2020.1.1f1 on Oculus Quest (1 & 2) but have also seen on 2019.3 and 2019.4

Native StackTrace:

Thread 0 (crashed)
0   libc.so                            0x0000007fabbd50a8 <system symbols missing>
1   libunity.so                        0x0000007f8c166b68 ApiGLES::ClearBufferSubData(unsigned int, gl::BufferTarget, long, long)
2   libunity.so                        0x0000007f8c146290 BufferManagerGLES::AcquireBuffer(unsigned long, DataBufferGLES::BufferUsage, bool)
3   libunity.so                        0x0000007f8c15435c GfxDeviceGLES::SetComputeBufferData(GfxBuffer*, void const*, unsigned long, unsigned long)
4   libunity.so                        0x0000007f8c2b6a00 GfxDevice::UpdateComputeSkinPoseBuffer(GfxBuffer*, MatrixArrayJobOutput*)
5   libunity.so                        0x0000007f8c6eb64c GfxDeviceWorker::RunCommand(ThreadedStreamBuffer&)
6   libunity.so                        0x0000007f8c6ec8a4 GfxDeviceWorker::RunExt(ThreadedStreamBuffer&)
7   libunity.so                        0x0000007f8c6e4b64 GfxDeviceWorker::RunGfxDeviceWorker(void*)
8   libunity.so                        0x0000007f8c05fb7c Thread::RunThreadWrapper(void*)
9   libc.so                            0x0000007fabc30454 <system symbols missing>
10  libc.so                            0x0000007fabbd8204 <system symbols missing>

[spatialx]

Filed a bug report, but doesn’t really have any more info than what’s above.

https://fogbugz.unity3d.com/default.asp?1296327_9ro5mh7paoorootd

[eusebium]

Any updates on this? This still happens on 2023.2.18f1. Setting GPU Skinning to CPU is a workaround.

[chad-yu]

@eusebium

Any updates on this? This still happens on 2023.2.18f1. Setting GPU Skinning to CPU is a workaround.

Wow… This amazingly worked for me… How did you find this workaround? Did you literally just go through different settings and try to make builds?