I’m currently upgrading a large-scale Unity project from version 2021.3 to 6000.1.6f1, and I’ve encountered a major issue with build times. On 2021.3, a full build of the project would take around 8 hours. After moving to 6000.1, the same project has been building for over 200 hours and still hasn’t completed.
This upgrade is being done with the objective of using the latest advancements on shader stripping that I read about in the beginning of the year, when we were having a problem after launching the dame where it would seem that out shader were being completely loaded in memory with all variants combinations at the start of the game, before the first screen, resulting in 5gb or memory being allocated with just shaders. This is something we addressed by cutting down shader properties but it didn’t felt like it never were perfect (only the necessary variants being loaded).
For this new build there haven’t been any significant structural changes to the project itself between these versions—same shaders, same assets, same platform targets. The only major variable is the Unity version.
Has anyone else experienced this? Are there any known changes in the way this version handles asset import, shader compilation, or build pipelines that could explain this?
Any insights would be appreciated. At the moment, this is making it extremely difficult to move forward with testing and release planning on consoles.
Okay, so first I need to say this: 8 hours by itself is sheer and absolute madness! I’ve worked with projects both Unity and proprietary C/C++ of sizes reaching 100+ GB in assets and none of them, ever, compiled or built for more than an hour.
Not saying that this cannot happen, but I highly suspect there might just be something utterly wrong here to begin with.
Is the build machine adequate? It might run out of RAM and starts thrashing the disk for instance. Or the disk might be a literal spinning drive. Or the build machine has an active antivirus without excluding the project, build output, unity install, and temp folders.
Could be simply an exorbitant amount of custom shaders, or something that causes the shaders to be compiled to a gazillion variants. It couldn’t hurt to scan Unity Resources for particularly Shader Graph and shader compilation optimization steps if you haven’t done so yet.
It could be worthwhile to run the Profiler in edit mode while making that 8 hours build to see where most of the time is spent unless it’s always and has always been mostly the shader compilation process.
Yep, the machine is quite capable machine with a i7-9700K and 16Gb of ram and a RTX2070 Super. The machine doesn’t have any active work doing alongside with the build and no antivirus.
The 8 hour compilation was “fine”, it was manageable, what I don’t understand is the jump from 8 to 200 plus hours after the unity 6 update. We do have a good amount of custom shaders as our game have a lot of light dependent mechanics
I am sure that there is something utterly wrong with it, my plan is to figure out what exactly.
The majority of the time does seems to be shader compilation. We have a few shaders that are based on the URP Lit shaders with our spin on it.
It would be great if we could get any hints of what to look for, even if it is to come back to the 8 hour long mark for the build
16 GB memory may be inadequate, especially for a “large scale” project. Check if the machine isn’t running low on memory while making builds, ie with Task Manager. If memory usage ends up >= 80% you ought to at least double the installed memory, better more as in-memory caching of files can provide additional speed bumps. For developer systems these days we all recommend installing at least 32 GB of RAM as the absolute minimum.
The sudden jump from 8 to 200 hours may be reflecting on that because previously, the system may have hit a bottleneck every once in a while, but now it may be hitting it more frequently perhaps due to increased memory usage in the build process.
The i7 is also something to check on because that CPU came out in late 2018, which means it is along with the rest of the hardware not up to today’s standards. Specifically in terms of PCIe bandwidth and memory speeds. If the project is even on a spinning hard disk, or a cheap and crippled SSD (they tend to offer high speeds only for short periods of time) that can also contribute to slow builds.
For now, my gut feeling is that the hardware is what’s primarily slowing down your builds.
I know the PC ain’t the best of the best, for that we were well aware that it could be better if that computer were stronger, but we been seeing this behavior in all our personal machines that are way stronger, the difference is that on those we can’t just leave it for days building. It is honestly feeling like there is something wrong; either we did something wrong, or there is something wrong. Because the one machine that was building in 8 hours is now taking forever.
This week we will investigate it even further but honestly we are kind of lost on what we could do
I mean with 8 hours builds it’s a long shot but you could run the Profiler in edit mode and make a build. If possible, trim down what you have so that it builds within a fraction of that time. This is assuming the culprit is the same.
It still kind of sounds like the long build times have accrued over time and never got looked at but you’re past the point where it’s still manageable to analyze. Kind of like the frog in the boiling pot problem.
Going back to an earlier version could also help, perhaps you can even find that between one and another months the build times made a sudden jump. Though I suppose that would have been noticed. But maybe there’s half an hour here, and another half an hour there, and then you might be able to find commonalities such as having added a ton of new materials, changes in shader graphs, things like that.
I’m sure there’s a problem on top when the 8 to 200 jumps up, which may be a regression in Unity. But I’m more concerned with those 8 hours to begin with.
I also wonder, are these 8 hours incremental builds or clean builds? Is it an IL2CPP build? Did you make changes to Player Settings that might affect build times, such as code stripping and optimizations?
Sorry for the delayed response, I had a problem on my personal computer that took a while to resolve
The duration of those builds are mainly shader compilation, the rest of the earlier steps are fairly quick to complete. For some reason, it seems that the newer version is compppiling way more shader variants than before, even for shadergraphs that don’t have any other keyword added to them.
I’ve done a test today, I ran a build on the older version and in fact, the older version is compiling WAY less variants
A single Shader Graph that was compiling around 160 variants on 2021.3 is now compiling 61k variants.
This explains my problem, that being said, I STILL dont have any clues to why this is since both projecs are marked with the same project settings about shader stripping
This explains the encrease in time, but WHY is my shaders compiling so much MORE shader variants on the new version?
Ignoring the unusual increase in shader variants that sounds about right. All of my projects spend the majority of their build time compiling shader variants. Unity should be spawning one copy of the shader compiler per thread. It’s one of the reasons I’m on a 5950X (16C/32T). I can’t imagine being on a 9700K (8C/8T).
I mean, it wasn’t great, 8 hours on a IL2CPP build wasn’t good, but we did it overnight so it wasn’t the end of the world, but the version upgrade adding this much more variants is making it impossible to let the rest of the team migrate to the newer version where building is impossible
I’ve also tried implementing IPreprocessShaders to have a bit more control over the variants but I don’t seem to be having any effect as the code seems to be ignored during build; I am trying to just log shader compilation while skiping the compilation itself but everything is compiling anyway and my log file nor being created, which makes me feel like my IPreprocessShaders is being ignored on build
How can even a shadergraph with no user-created keywords be compiling to so many variants?
Compiling shader "Shader Graphs/shader_ilhaCemiterio" pass "Universal Forward" (fp)
Full variant space: 4529848320
After settings filtering: 15728640
After built-in stripping: 61440
After scriptable stripping: 61440
Processed in 0.70 seconds
starting compilation...
From around 174 to 61440 is a crazy amount to increase, and this is one of the smaller shaders, so no wonder why the build is taking so long
