Slow Uploading Symbol to Cloud Diagnostics Step

RaventurnPatrick · November 6, 2024, 2:20pm

Hi,
Will you make the symbol upload async? i.e. so we can already deploy / upload the build while the symbols are being uploaded. This step regularly takes a few minutes (90% of the build time)

its_neil · November 6, 2024, 2:31pm

Hi there, cloud diagnostics symbol upload is already done async, do you have an editor.log file so we can see where the bottleneck may be?

RaventurnPatrick · November 6, 2024, 3:14pm

We are currently still using Unity 2022 LTS, will Unity 6 perform this step after the build has been deployed to the device and the .aab has been created?

Relevant lines from log:

Creation of the 7z symbols (happens before deployment)

[BUSY       7s] 7z .../Library/Bee/Android/Prj/IL2CPP/symbols.zip
[BUSY      17s] 7z .../Library/Bee/Android/Prj/IL2CPP/symbols.zip
[BUSY      27s] 7z .../Library/Bee/Android/Prj/IL2CPP/symbols.zip
[3063/3125 32s] 7z .../Library/Bee/Android/Prj/IL2CPP/symbols.zip
7-Zip (z) 24.08 (x64) : Copyright (c) 1999-2024 Igor Pavlov : 2024-08-11
Scanning the drive:
6 files, 935808368 bytes (893 MiB)
Creating archive: .../Library/Bee/Android/Prj/IL2CPP/symbols.zip

The Uploading symbols task in this case only took around 30sec (but sometimes it can hang for much longer)
Also the task is executed before the deployment or apk is created

Android PostProcess task "Cloud Diagnostics Symbol Upload" took 451.3268 ms
time="2024-11-06T16:08:58+01:00" level=info msg="Running Usymtool..." 
time="2024-11-06T16:08:58+01:00" level=info msg="Command line: C:/Program Files/Unity/Hub/Editor/2022.3.51f1/Editor/Data\Tools\usymtool.exe -symbolPath .../Library/Bee/Android/Prj/IL2CPP/Gradle\unityLibrary\symbols -log ..\AppData\Local\Unity\Editor\symbol_upload.log -il2cppOutputPath .../Library/Bee/Android/Prj/IL2CPP/Gradle/../Il2CppBackup/il2cppOutput -il2cppFileRoot ...\Library/Bee/artifacts/Android/il2cppOutput/cpp" 
time="2024-11-06T16:08:58+01:00" level=info msg="Usymtool executable path: C:/Program Files/Unity/Hub/Editor/2022.3.51f1/Editor/Data\Tools\usymtool.exe" 
time="2024-11-06T16:08:58+01:00" level=info msg="Usymtool OS/Arch: windows/amd64" 
time="2024-11-06T16:08:58+01:00" level=info msg="Working directory: ..." 
time="2024-11-06T16:08:58+01:00" level=info msg="Log file path: ...\AppData\Local\Unity\Editor\symbol_upload.log" 
time="2024-11-06T16:08:58+01:00" level=info msg="LZMA_PATH: C:/Program Files/Unity/Hub/Editor/2022.3.51f1/Editor/Data\Tools\lzma.exe" 
time="2024-11-06T16:08:58+01:00" level=info msg="USYM_UPLOAD_URL_SOURCE: https://perf-events.cloud.unity3d.com/url" 
time="2024-11-06T16:08:58+01:00" level=info msg="USYM_UPLOAD_AUTH_TOKEN: (present, redacted)" 
time="2024-11-06T16:08:58+01:00" level=info msg="symbolPath: .../Library/Bee/Android/Prj/IL2CPP/Gradle\unityLibrary\symbols" 
time="2024-11-06T16:08:58+01:00" level=info msg="il2cppOutputPath: .../Library/Bee/Android/Prj/IL2CPP/Gradle/../Il2CppBackup/il2cppOutput" 
time="2024-11-06T16:08:58+01:00" level=info msg="il2cppFileRoot: ...\Library/Bee/artifacts/Android/il2cppOutput/cpp" 
time="2024-11-06T16:08:58+01:00" level=info msg="Scanning IL2CPP output for C# line numbers..." 
time="2024-11-06T16:09:00+01:00" level=info msg="Done scanning IL2CPP output for C# line numbers" 
time="2024-11-06T16:09:00+01:00" level=info msg="Found 6 files to check, scanning for binary images now..." 
time="2024-11-06T16:09:00+01:00" level=info msg="Found 6 binary images to process (not all files contain a binary image, and some contain multiple images)" 
time="2024-11-06T16:09:00+01:00" level=info msg="Log for folder symbol_upload_root:
...\Library\Bee\Android\Prj\IL2CPP\Gradle\unityLibrary\symbols\arm64-v8a\libil2cpp.so
	arm64	04d895642f88705e8cfc678d543a506eaa45b441
...\Library\Bee\Android\Prj\IL2CPP\Gradle\unityLibrary\symbols\arm64-v8a\libmain.so
	arm64	a691e4189d17bd781167ccf0a89bdfc238a3bea2
...\Library\Bee\Android\Prj\IL2CPP\Gradle\unityLibrary\symbols\arm64-v8a\libunity.so
	arm64	94e43f644eff5e47
...\Library\Bee\Android\Prj\IL2CPP\Gradle\unityLibrary\symbols\armeabi-v7a\libil2cpp.so
	arm	baac31d1f02b3782a680c466bea7bd3b4262adc7
...\Library\Bee\Android\Prj\IL2CPP\Gradle\unityLibrary\symbols\armeabi-v7a\libmain.so
	arm	6ae746ab24465f5527c3e6e2e68d2dbfcbd96844
...\Library\Bee\Android\Prj\IL2CPP\Gradle\unityLibrary\symbols\armeabi-v7a\libunity.so
	arm	7c1e597a113841e9
" 
time="2024-11-06T16:09:00+01:00" level=info msg="File with UUID 7c1e597a113841e9 already exists." 
time="2024-11-06T16:09:00+01:00" level=info msg="Signed URL is: https://storage.googleapis.com/unity-cs-common-prd-crash-symbols/9eac01df-561b-4f24-a3aa-04affb9f32ae/04d895642f88705e8cfc678d543a506eaa45b441.usym.lzma?..." 
time="2024-11-06T16:09:00+01:00" level=info msg="File with UUID a691e4189d17bd781167ccf0a89bdfc238a3bea2 already exists." 
time="2024-11-06T16:09:00+01:00" level=info msg="File with UUID 6ae746ab24465f5527c3e6e2e68d2dbfcbd96844 already exists." 
time="2024-11-06T16:09:00+01:00" level=info msg="File with UUID 94e43f644eff5e47 already exists." 
time="2024-11-06T16:09:00+01:00" level=info msg="Signed URL is: https://storage.googleapis.com/unity-cs-common-prd-crash-symbols/..."

I assume creating + uploading a nearly 1GB large symobls file is the culprit here

its_neil · November 6, 2024, 3:55pm

Thanks for that, is it possible to share the full editor.log? The symbols upload here took only 0.5seconds.

Also, the symbol generation before APK is by design in Unity 2022.3 and below. Unity 6 uses Gradle to generate the symbols and make the build and it does this in parallel.

RaventurnPatrick · November 7, 2024, 10:03am

Thanks for the fast response,
I sent the logs via pm due to privacy reasons

its_neil · November 8, 2024, 11:39am

Thanks, I’ve had a look at them. Give me some time and I can circle back with you

its_neil · November 11, 2024, 10:27am

So I have confirmed with the Cloud Diagnostics team that everything is done async, and the only way to improve this is to either have faster internet, reduce the footprint of your symbols so there is less to generate/upload or switch off Cloud Diagnostics if you’re not doing a store build.

Tomas1856 · November 11, 2024, 10:54am

P.S this might help too

Having timestamps to log entries might help you determine where Unity is talking the most time.

RaventurnPatrick · November 12, 2024, 3:31pm

Thanks for the response. Solutions that would improve the experience for us in Unity 6:

Add flag to Build Settings to Enable/Disable Cloud Diagnostics Symbol Upload
Better: Upload Symbols to Cloud Diagnostics but deploy & run build on device even when it is not yet finished

Regarding the footprint of symbols - is there a way to reduce the symbol size? Maybe could you link to the docs?

Tomas1856 · November 12, 2024, 4:38pm

See here - Unity - Manual: Android symbols, symbols can be pretty big if you have Debugging symbols selected, selecting Public symbols should lower the size of symbols.

RaventurnPatrick · November 20, 2024, 10:30am

We found out that when Unity Services are enabled, the Symbols are always built as “Debug” thus they are large. I guess however that is necessary so Unity Cloud Diagnostics can show stack traces with line numbers?

Tomas1856 · November 20, 2024, 11:02am

Correct.

P.S This was considered as bug in Unity 6, and UCD no longer forces debugging symbols, and just follows Debug symbol level setting in build setting window., but left untouched in earlier Unity versions.

UnityElishaS · November 20, 2024, 4:31pm

On the subject of saving build time during development: Cloud Diagnostics can simply be disabled outright for any builds that do not require crash reporting.

Whether or not it is an appropriate action depends on your development needs, but disabling Cloud Diagnostics will save on build time for those builds because the symbol upload process will not be attempted. Cloud Diagnostics can be appropriately re-enabled for any build(s) that strongly benefit from crash reporting, such as a public release for example.