With URP/HDRP, building with a standard number of materials results in a massive massive number of potential shader variants, with the building taking literally hours (With Gigiya taking around 13 hours I believe?). This was/is not the case with builtin.
What is the technical reason for this issue, and what are the plans to solve it?
Hi Josh, thank you for your question. This is on top of our minds.
We are aware the shader variant explosion is a problem for productions, and it’s a hard problem to solve. This was also a big issue highlighted by Gigaya and we know it’s impacting several other projects as well. This is something that multiple teams are working on to improve. I also acknowledge we can always do better. So let me share some thoughts on why this is a hard problem to solve and what we are doing.
Why was this not a problem in Built-in RP?
Built-in RP would also have reached the scalability issues on shader explosion if it had continue developing graphics features in the same pace we are adding for SRPs. The scalability limitations and some of the problems around uber shaders are hinted here in this blog post by Aras.
Why does build time keeps increasing for URP?
URP is a pipeline rapidly in development, we are on a way to overcome built-in RP in every way. We are landing several features in each release. In 2023 we are close in feature parity with built-in RP feature while also offering a lot more. In some platforms, whenever we add a feature we face a choice: Do we regress runtime performance or do we add more variants? It’s a hard balance that we try to best reason with every feature for all the platforms we support. In the past we tried to optimize and turn some of the variants into branches but identified runtime regressions in some platforms.
Why do users get a regression in build time even if they don’t use new features in their project?
This could happen for a combination of reasons.
The shader stripping system is not ready to handle scalability at this amount of variants. There’s a cost in the shader stripper as it processes every variant, and it could be that there are multiple C# shader strippers in a single project. There’s a massive improvement on this system by pre-filtering variants, in our test case it’s a 100x reduction in variant processing for the C# shader stripping for URP Lit shader. Developers will discuss the backport of this improvement to older versions of Unity.
The current shader stripping and Unity project cooking is too manual and relies a lot on users to understand the settings and optimize it themselves. We don’t prevent settings to be changed at runtime. We have to be careful If we strip features by default or have some implicit static analysis heuristic then this could cause situations that users have features being stripped unwillingly from the build. The impact of that conservative stripping we have is that we end up adding many variants that are not really needed in the build in most cases.
We also lack API for users to fine tune the balance of runtime vs build time performance, by f.ex selecting which variants they would like to turn into a branch. We also lack API to explicitly strip some of the features.
A few things that we are doing to improve this:
We have been landing improvements to shader systems in Unity. The shader pre-filtering work is one of them, we also recently landed some improvement to reduce shader memory footprint by allowing users to control how shaders are evicted from memory. In URP every release we do a pass to look for potential optimizations that we might have missed, f.ex in 2021.2 we did this, and in 22.2 we did already some other work for PRs. There’s an opportunity here to improve and be better at catching up these optimization opportunities at PR level instead.
We can communicate better to users the known issues / expectations when they update to a newer version and define on a release acceptance criteria when it comes to build time regression from one LTS to the other. We are defining these release checklist and guidelines for our upcoming 2023 release, which is the one we are working now. However I just want to make it clear that we support LTS so any improvements that we find in this area we are considering to backport to older versions.
This is a challenge for release and testing. We support 25 platforms and several graphics APIs. Quite frequently an approach that works well in one platform might not in the other when it comes to decide between shader keyword or branching. We have already some performance test automation for URP, but we are expanding that to run on more platforms and capture more metrics, such as build times.
We have added more documentation on shader stripping [1][2][3][4]. We currently have documentation in progress to document shader keyword in URP.
The above list is not the end of it. We will need to keep getting feedback and iteratively improving it.
I think before official can figure out the truly smart solution to really fix it, the much more practical path for official to do is come out the great force shader stripping tooling first that u can force turn off the shader keyword u are not required. And when u wrongly turn off the shader keyword u can see it goes wrong like turn into pink immediately before building player runtime build. I afraid the solutions u propose will just only end up only speed up a little bit. What I expect is I dun need to take hours to get new player runtime time even for the first time build.
Strict Shader Variant Matching was introduced in 2022.1.0a7, and can be used to fallback on a pink debug shader when a requested variant is missing, logging an error in the console detailing the variant and its keywords.
We shared some useful information on recent and upcoming improvements to Shader Variant Management in the following Shaders forum thread, and will keep updating as we move forward with additional improvements!
https://forum.unity.com/threads/improvements-to-shader-build-time-and-runtime-memory-usage.1305615/
I see. I would like to see the proper shader tooling happening soon as I dun really want to a lot of time to write a shader stripping tooling myself which way too time consuming. And also is there any latest shader stripping code sample I can refer to for now since the sample at blog post is no longer able to download anymore?
Dynamic branching instead of keyword was alluded here already, but just for completeness, I’d like to mention that there is a new #pragma dynamic_branch directive. I can’t find the forum post on this, but it’s hidden in the doc somewhere. I think the version is 2023. Correct me if I’m wrong.
edit: version is 2022.1.0a15 as per the post below.
Since processing a large amount of variants for potential stripping and compilation can still take a long time, and so we are introducing the Variant Prefiltering improvement that @phil_lira mentioned above.
Prefiltering allows to perform early exclusion of shader keywords based on prefiltering rules (driven by RP settings), resulting in less variants to process thus significantly reducing shader processing time.
With prefiltering, the stripping would also look a bit different, and so we are now updating the documentation to reflect and will soon include new meaningful examples.
I think I provide more contexts about the tooling I expecting. Until now from wat I see shader stripping is happening fully auto mode. I would like to have great tooling window to able to inspect all the shader variant at my production project and then strip it manually. I think unity china ady implemented similar tooling that I mention. Maybe official can consider to merge their tooling into both URP and HDRP package.
dynamic_branch keywords were introduced in 2022.1.0a15, to abstract over dynamic branching in shaders.
See more detail in: https://docs.unity3d.com/2022.1/Documentation/Manual/SL-MultipleProgramVariants.html
Keep in mind that before 2022.1, you can also implement dynamic branching traditionally via if statements:
https://docs.unity3d.com/2021.3/Documentation/Manual/shader-branching.html
One improvement we are investigating, is allowing to define runtime usage flags for certain RP features, while specifying when its ok to use dynamic branching for certain features. Features controlled at runtime should generally use ‘multi_compile’ keywords, but it may be acceptable to use ‘dynamic_branch’ keywords on certain platforms (thus generating less variants). Features not controlled at runtime generally should use ‘shader_feature’ keywords, which are stripped out based on material usage, thus producing less variants.
As part of the larger improvements to Shader Variant Management, we plan to provide a meaningful Editor tool that will help to easily understand the project’s shader variant usage. This will allow you to view which shaders and keywords generate the most variants included the build, and potentially also the runtime usage of variants (which variants are requested, missing or unused at runtime).
This would help with more effective authoring of conditional shader features, as well as help perform more effective variant stripping. We will share more about our plans and provide additional detail as soon as we can!
This would be very useful, though I wonder how it should be best exposed? And if people will generally understand the side effects? For instance, in Rock Band we massively lowered our variant count by switching some of our lighting variants to branches. It was a small loss in performance, but massively reduced variants. However, on a lower end platform it would not have been the best choice. Further, conditional branches on the GPU have to be handled with care especially in regards to texture samples, and you can accidentally break the quad sharing of texture samples if your not careful- something a lot competent shader authors might not be aware of. And the built time stripping vs. runtime switching is a pretty different paradigm when it comes to dynamic runtime changes.
We are still exploring how to introduce such an improvement, and make sure users are aware of the tradeoffs - but as you highlighted dynamic branching in shaders should be handled with care, and with platform considerations taken into account. Sensible defaults and an opt-in approach could definitely help.
As a note, as of 2021.3 we officially support conditional declaration of shader keywords in platform conditionals (to declare ‘dynamic_branch’ / ‘multi_compile’ / ‘shader_feature’ based on the target platform)
Another potential optimization in my mind would be to scan all materials in the project - those would be the variants the project needs, correct? And then allowing for a way to add additional variants to be included when projects use runtime variants. (Armchair programming of course)
The new changes do sound like a good step in the direction, and I’m very glad it’s on the top of the list.
Devs often keep doing dev/release builds to test other code/performance issues. We really can’t wait 2-4 hours for each game build.
Would it be possible to do some runtime shader compiling, if devs don’t care about those performance drops? And just add a watermark in a corner of such game builds, to make sure nobody accidentally publishes such builds publicly?
And then only wait for those long hours for builds where the shaders would be pre-compiled?
I’m guessing all of the code for compilation is squarely in the editor space and would be really hard / impossible to do in builds
This is something we are investigating, to be able to provide API to allow a multi compile to be either turned into a dynamic branch of be compiled as shader variant.
‘shader_feature’ keywords address this, as we will not include their resulting shader variants in the build if no material in the project enable these. However, many render pipeline settings can be controlled at runtime and thus we need to include their possible variants, as they may be requested at runtime. As @phil_lira mentioned above, we could allow you to flag whether certain features can utilize dynamic branching instead, to reduce the amount of compiled variants.
With improvements to our build time shader processing, we will pre-filter unnecessary multi_compile keywords, so this should significantly reduce variant processing time though!
In CryEngine and Unreal they do it over the network. You keep the editor running and the build requests new shader variants when needed, which are compiled on the editor and sent back to the build.
Hey guys, we changed our project from Built-in to URP in the past months for only now discover that the build time is a huge problem! In the Built-in our entirely game compiled in less than 3 hours for Windows, and now we waited for over 70 hours to have only 50% of the game compiled. I was looking for some bug report and I found this thread, which is really disappointed.
If the build time is so bad this should be something much more visible. We did a trial before porting the entire game but obvious with just a parcel of the game where this problem wasn’t so noticiable.
But I’m still wondering if there is a bug with shader chache, there isn’t any shader cache?
I looked several times to the progress bar and I saw the same shader over and over again being compiled. I tried that with Unity 2021.3.11f1 and then I upgrade to 2021.3.15f1 and I had the same results.
Our game uses addressables that builds extremely slow but it’s only about 1.5gb and then we have currently 20 DLCs that shares a lot of shaders but seems like none of them was shared in compilation. If by any chance the chache worked we can deal with that because the next builds would be much faster.
The result build time in our case is about 50x worse then on built-in. If this is in fact the expected result for each build then SRP shouldn’t be an option so widely encoraged right now.
Your best bet is to manually strip the variants and passes you don’t use - URP just has a ton of shader keywords by default that are setup to be dynamic. For instance, stripping deferred rendering if you’re not using it, as well as various other options.
And yeah, the marketing department did no favors here by directing the sheep over the cliff.