Long Jenkins Builds: Build Shader Compiler Thread Starvation Warning

Hello,

We’re running into an issue with 1/3 build machines that we have setup with Jenkins for our Unity Project. The short summary is we are spammed with thread starvation warnings during the Unity Shader Compilation of the build and it is significantly increasing our build times (30mins → 4h) and sometimes hits the 4h time out limit and fails the build.

The other 2 PCs haven’t had this issue, or at least not nearly to this extent.
The new PC is more powerful than the other two so we don’t believe the issue lie with our CPU being below spec.

NOTES:

  • We’re using the latest AmplifyShader for our project (1.9.2.0) and the issue started occurring when upgrading to Unity 2022. We’ve upgraded since the initial major upgrade and are now on Unity 2022.3.20f1 and we still have the issue.
  • The issue is isolated to our Android Addressables Builds (i.e. Quest & Pico).
  • When we run the build via command line on the Build machine directly, rather than via Jenkins, we do not have this issue, which makes us wonder if it’s an issue with SSH.
  • We get the same UnauthroizedAccessException you’ll see in the logs on our other PC builds and they don’t have any issue.

Below is are the warning logs, and the logs before and after them.
Any and all advice on what to try would be greatly appreciated.

Start importing Packages/com.unity.xr.picoxr/Assets/Resources/Controller/PICO 4 L.prefab using Guid(853840c5196b2486a97e1785f3a68575) Importer(-1,00000000000000000000000000000000)  -> (artifact id: 'dd7eff224b41e8b25b323c9c23d08226') in 0.003238 seconds
...
Launched and connected shader compiler UnityShaderCompiler.exe after 0.01 seconds
Launched and connected shader compiler UnityShaderCompiler.exe after 0.01 seconds
Launched and connected shader compiler UnityShaderCompiler.exe after 0.01 seconds
Launched and connected shader compiler UnityShaderCompiler.exe after 0.01 seconds
Launched and connected shader compiler UnityShaderCompiler.exe after 0.01 seconds
Launched and connected shader compiler UnityShaderCompiler.exe after 0.02 seconds
Launched and connected shader compiler UnityShaderCompiler.exe after 0.02 seconds
Launched and connected shader compiler UnityShaderCompiler.exe after 0.02 seconds
Launched and connected shader compiler UnityShaderCompiler.exe after 0.02 seconds
Launched and connected shader compiler UnityShaderCompiler.exe after 0.01 seconds
Launched and connected shader compiler UnityShaderCompiler.exe after 0.02 seconds
Launched and connected shader compiler UnityShaderCompiler.exe after 0.02 seconds
Launched and connected shader compiler UnityShaderCompiler.exe after 0.02 seconds
Launched and connected shader compiler UnityShaderCompiler.exe after 0.02 seconds
Launched and connected shader compiler UnityShaderCompiler.exe after 0.02 seconds
Launched and connected shader compiler UnityShaderCompiler.exe after 0.02 seconds
Launched and connected shader compiler UnityShaderCompiler.exe after 0.02 seconds
Launched and connected shader compiler UnityShaderCompiler.exe after 0.02 seconds
[Licensing::Client] Successfully resolved entitlement details

UnauthorizedAccessException: Access to the path "C:\Jenkins\workspace\lkabout-Pico-Addressables_rel5.0\Unity\Assets\AmplifyShaderEditor\Plugins\Editor\Templates\TemplateMenuItems.cs" is denied.
  at System.IO.FileStream..ctor (System.String path, System.IO.FileMode mode, System.IO.FileAccess access, System.IO.FileShare share, System.Int32 bufferSize, System.Boolean anonymous, System.IO.FileOptions options) [0x0019e] in <dc753a1061284f8e971ee88ee4826eee>:0
  at System.IO.FileStream..ctor (System.String path, System.IO.FileMode mode, System.IO.FileAccess access, System.IO.FileShare share, System.Int32 bufferSize, System.IO.FileOptions options) [0x00000] in <dc753a1061284f8e971ee88ee4826eee>:0
  at (wrapper remoting-invoke-with-check) System.IO.FileStream..ctor(string,System.IO.FileMode,System.IO.FileAccess,System.IO.FileShare,int,System.IO.FileOptions)
  at System.IO.StreamWriter..ctor (System.String path, System.Boolean append, System.Text.Encoding encoding, System.Int32 bufferSize) [0x00055] in <dc753a1061284f8e971ee88ee4826eee>:0
  at System.IO.StreamWriter..ctor (System.String path) [0x00008] in <dc753a1061284f8e971ee88ee4826eee>:0
  at (wrapper remoting-invoke-with-check) System.IO.StreamWriter..ctor(string)
  at AmplifyShaderEditor.IOUtils.SaveTextfileToDisk (System.String shaderBody, System.String pathName, System.Boolean addAdditionalInfo) [0x0003c] in C:\Jenkins\workspace\lkabout-Pico-Addressables_rel5.0\Unity\Assets\AmplifyShaderEditor\Plugins\Editor\Utils\IOUtils.cs:607
  at AmplifyShaderEditor.TemplatesManager.CreateTemplateMenuItems () [0x002cb] in C:\Jenkins\workspace\lkabout-Pico-Addressables_rel5.0\Unity\Assets\AmplifyShaderEditor\Plugins\Editor\Templates\TemplatesManager.cs:794
  at AmplifyShaderEditor.TemplatePostProcessor.OnPostprocessAllAssets (System.String[] importedAssets, System.String[] deletedAssets, System.String[] movedAssets, System.String[] movedFromAssetPaths) [0x002a3] in C:\Jenkins\workspace\lkabout-Pico-Addressables_rel5.0\Unity\Assets\AmplifyShaderEditor\Plugins\Editor\Templates\TemplatePostProcessor.cs:160
  at (wrapper managed-to-native) System.Reflection.RuntimeMethodInfo.InternalInvoke(System.Reflection.RuntimeMethodInfo,object,object[],System.Exception&)
  at System.Reflection.RuntimeMethodInfo.Invoke (System.Object obj, System.Reflection.BindingFlags invokeAttr, System.Reflection.Binder binder, System.Object[] parameters, System.Globalization.CultureInfo culture) [0x0006a] in <dc753a1061284f8e971ee88ee4826eee>:0

(Filename: Assets/AmplifyShaderEditor/Plugins/Editor/Utils/IOUtils.cs Line: 607)

warn: Microsoft.AspNetCore.Server.Kestrel[22]
      As of "03/28/2024 17:11:07 +00:00", the heartbeat has been running for "00:00:02.6095541" which is longer than "00:00:01". This could be caused by thread pool starvation.
warn: Microsoft.AspNetCore.Server.Kestrel[22]
      As of "03/28/2024 17:32:26 +00:00", the heartbeat has been running for "00:00:01.7100941" which is longer than "00:00:01". This could be caused by thread pool starvation.
warn: Microsoft.AspNetCore.Server.Kestrel[22]
      As of "03/28/2024 17:33:49 +00:00", the heartbeat has been running for "00:00:01.1047536" which is longer than "00:00:01". This could be caused by thread pool starvation.
warn: Microsoft.AspNetCore.Server.Kestrel[22]
      As of "03/28/2024 17:43:53 +00:00", the heartbeat has been running for "00:00:01.1725176" which is longer than "00:00:01". This could be caused by thread pool starvation.
warn: Microsoft.AspNetCore.Server.Kestrel[22]
      As of "03/28/2024 17:49:11 +00:00", the heartbeat has been running for "00:00:01.0193822" which is longer than "00:00:01". This could be caused by thread pool starvation.
warn: Microsoft.AspNetCore.Server.Kestrel[22]
      As of "03/28/2024 17:50:15 +00:00", the heartbeat has been running for "00:00:01.1089309" which is longer than "00:00:01". This could be caused by thread pool starvation.
warn: Microsoft.AspNetCore.Server.Kestrel[22]
      As of "03/28/2024 17:54:31 +00:00", the heartbeat has been running for "00:00:01.2296653" which is longer than "00:00:01". This could be caused by thread pool starvation.
warn: Microsoft.AspNetCore.Server.Kestrel[22]
      As of "03/28/2024 17:54:53 +00:00", the heartbeat has been running for "00:00:01.1386848" which is longer than "00:00:01". This could be caused by thread pool starvation.
warn: Microsoft.AspNetCore.Server.Kestrel[22]
      As of "03/28/2024 18:04:24 +00:00", the heartbeat has been running for "00:00:01.0194366" which is longer than "00:00:01". This could be caused by thread pool starvation.
warn: Microsoft.AspNetCore.Server.Kestrel[22]
      As of "03/28/2024 18:09:14 +00:00", the heartbeat has been running for "00:00:01.0193139" which is longer than "00:00:01". This could be caused by thread pool starvation.
warn: Microsoft.AspNetCore.Server.Kestrel[22]
      As of "03/28/2024 18:12:59 +00:00", the heartbeat has been running for "00:00:01.1175371" which is longer than "00:00:01". This could be caused by thread pool starvation.
warn: Microsoft.AspNetCore.Server.Kestrel[22]
      As of "03/28/2024 18:14:10 +00:00", the heartbeat has been running for "00:00:01.2309732" which is longer than "00:00:01". This could be caused by thread pool starvation.
warn: Microsoft.AspNetCore.Server.Kestrel[22]
      As of "03/28/2024 18:22:48 +00:00", the heartbeat has been running for "00:00:01.1403372" which is longer than "00:00:01". This could be caused by thread pool starvation.
warn: Microsoft.AspNetCore.Server.Kestrel[22]
      As of "03/28/2024 18:23:00 +00:00", the heartbeat has been running for "00:00:01.5006161" which is longer than "00:00:01". This could be caused by thread pool starvation.
warn: Microsoft.AspNetCore.Server.Kestrel[22]
      As of "03/28/2024 18:23:53 +00:00", the heartbeat has been running for "00:00:01.1099464" which is longer than "00:00:01". This could be caused by thread pool starvation.
warn: Microsoft.AspNetCore.Server.Kestrel[22]
      As of "03/28/2024 18:28:18 +00:00", the heartbeat has been running for "00:00:01.1401322" which is longer than "00:00:01". This could be caused by thread pool starvation.
warn: Microsoft.AspNetCore.Server.Kestrel[22]
      As of "03/28/2024 18:28:22 +00:00", the heartbeat has been running for "00:00:01.5300483" which is longer than "00:00:01". This could be caused by thread pool starvation.
warn: Microsoft.AspNetCore.Server.Kestrel[22]
      As of "03/28/2024 18:34:42 +00:00", the heartbeat has been running for "00:00:01.2902343" which is longer than "00:00:01". This could be caused by thread pool starvation.
warn: Microsoft.AspNetCore.Server.Kestrel[22]
      As of "03/28/2024 18:36:20 +00:00", the heartbeat has been running for "00:00:01.1700687" which is longer than "00:00:01". This could be caused by thread pool starvation.
warn: Microsoft.AspNetCore.Server.Kestrel[22]
      As of "03/28/2024 18:37:35 +00:00", the heartbeat has been running for "00:00:01.1429231" which is longer than "00:00:01". This could be caused by thread pool starvation.
warn: Microsoft.AspNetCore.Server.Kestrel[22]
      As of "03/28/2024 18:38:03 +00:00", the heartbeat has been running for "00:00:01.0486363" which is longer than "00:00:01". This could be caused by thread pool starvation.
warn: Microsoft.AspNetCore.Server.Kestrel[22]
      As of "03/28/2024 18:38:41 +00:00", the heartbeat has been running for "00:00:01.2005954" which is longer than "00:00:01". This could be caused by thread pool starvation.
warn: Microsoft.AspNetCore.Server.Kestrel[22]
      As of "03/28/2024 18:38:45 +00:00", the heartbeat has been running for "00:00:01.1104822" which is longer than "00:00:01". This could be caused by thread pool starvation.
Launched and connected shader compiler UnityShaderCompiler.exe after 0.09 seconds
warn: Microsoft.AspNetCore.Server.Kestrel[22]
      As of "03/28/2024 18:49:54 +00:00", the heartbeat has been running for "00:00:02.1600695" which is longer than "00:00:01". This could be caused by thread pool starvation.
warn: Microsoft.AspNetCore.Server.Kestrel[22]
      As of "03/28/2024 18:52:32 +00:00", the heartbeat has been running for "00:00:01.0200120" which is longer than "00:00:01". This could be caused by thread pool starvation.
warn: Microsoft.AspNetCore.Server.Kestrel[22]
      As of "03/28/2024 18:55:45 +00:00", the heartbeat has been running for "00:00:01.1100593" which is longer than "00:00:01". This could be caused by thread pool starvation.
warn: Microsoft.AspNetCore.Server.Kestrel[22]
      As of "03/28/2024 18:55:52 +00:00", the heartbeat has been running for "00:00:01.2236298" which is longer than "00:00:01". This could be caused by thread pool starvation.
warn: Microsoft.AspNetCore.Server.Kestrel[22]
      As of "03/28/2024 18:56:27 +00:00", the heartbeat has been running for "00:00:11.1903708" which is longer than "00:00:01". This could be caused by thread pool starvation.
warn: Microsoft.AspNetCore.Server.Kestrel[22]
      As of "03/28/2024 18:57:15 +00:00", the heartbeat has been running for "00:00:01.0493128" which is longer than "00:00:01". This could be caused by thread pool starvation.
warn: Microsoft.AspNetCore.Server.Kestrel[22]
      As of "03/28/2024 18:58:49 +00:00", the heartbeat has been running for "00:00:01.4106083" which is longer than "00:00:01". This could be caused by thread pool starvation.
warn: Microsoft.AspNetCore.Server.Kestrel[22]
      As of "03/28/2024 18:59:25 +00:00", the heartbeat has been running for "00:00:01.0805486" which is longer than "00:00:01". This could be caused by thread pool starvation.
warn: Microsoft.AspNetCore.Server.Kestrel[22]
      As of "03/28/2024 19:00:27 +00:00", the heartbeat has been running for "00:00:01.2595080" which is longer than "00:00:01". This could be caused by thread pool starvation.
warn: Microsoft.AspNetCore.Server.Kestrel[22]
      As of "03/28/2024 19:01:21 +00:00", the heartbeat has been running for "00:00:01.0502368" which is longer than "00:00:01". This could be caused by thread pool starvation.
warn: Microsoft.AspNetCore.Server.Kestrel[22]
      As of "03/28/2024 19:02:27 +00:00", the heartbeat has been running for "00:00:01.0491844" which is longer than "00:00:01". This could be caused by thread pool starvation.
warn: Microsoft.AspNetCore.Server.Kestrel[22]
      As of "03/28/2024 19:07:01 +00:00", the heartbeat has been running for "00:00:01.0799049" which is longer than "00:00:01". This could be caused by thread pool starvation.
warn: Microsoft.AspNetCore.Server.Kestrel[22]
      As of "03/28/2024 19:07:35 +00:00", the heartbeat has been running for "00:00:01.2567111" which is longer than "00:00:01". This could be caused by thread pool starvation.
warn: Microsoft.AspNetCore.Server.Kestrel[22]
      As of "03/28/2024 19:08:46 +00:00", the heartbeat has been running for "00:00:01.6781932" which is longer than "00:00:01". This could be caused by thread pool starvation.
warn: Microsoft.AspNetCore.Server.Kestrel[22]
      As of "03/28/2024 19:08:52 +00:00", the heartbeat has been running for "00:00:05.3998677" which is longer than "00:00:01". This could be caused by thread pool starvation.
warn: Microsoft.AspNetCore.Server.Kestrel[22]
      As of "03/28/2024 19:10:37 +00:00", the heartbeat has been running for "00:00:04.4988149" which is longer than "00:00:01". This could be caused by thread pool starvation.
warn: Microsoft.AspNetCore.Server.Kestrel[22]
      As of "03/28/2024 19:11:15 +00:00", the heartbeat has been running for "00:00:02.6999341" which is longer than "00:00:01". This could be caused by thread pool starvation.
warn: Microsoft.AspNetCore.Server.Kestrel[22]
      As of "03/28/2024 19:11:54 +00:00", the heartbeat has been running for "00:00:01.1404744" which is longer than "00:00:01". This could be caused by thread pool starvation.
Start importing Packages/com.unity.visualeffectgraph/Editor/Templates/SimpleRibbon.vfx using Guid(c071abb088c76054bb603238f21bce1c) Importer(-1,00000000000000000000000000000000) DisplayProgressbar: Compiling Packages/com.unity.visualeffectgraph/Editor/Templates/SimpleRibbon.vfx
-> (artifact id: '168a911b3b78d6ec7cc0fe9b33d85d84') in 0.498033 seconds
...

Best,
JP