URP 120k+ Shader Varients

[FlightOfOne]

I have been struggling with building for the last 4 days. Why is it compiling 120K Shader varients? Is this normal?

[Matt-Cranktrain]

I’m seeing this post a lot more recently ( here’s one from a couple weeks ago ) - it’s a problem I’ve run into, and continue to struggle with, and my thoughts/solutions in that thread are still the same.

Please share what works for you in fixing it, because some of the suggestions in older threads involving this problem I suspect are out of date.

I would love if Unity would give some visibility in the question of where all these shader variants come from. Yes, shader keywords * number of graphics APIs (like OpenGl and Vulkan) but to have actual numbers about which where each set of tens of thousands of variants comes from would be nice!

[FlightOfOne]

@Matt-Cranktrain

I’m seeing this post a lot more recently ( here’s one from a couple weeks ago ) - it’s a problem I’ve run into, and continue to struggle with, and my thoughts/solutions in that thread are still the same.

Please share what works for you in fixing it, because some of the suggestions in older threads involving this problem I suspect are out of date.

I would love if Unity would give some visibility in the question of where all these shader variants come from. Yes, shader keywords * number of graphics APIs (like OpenGl and Vulkan) but to have actual numbers about which where each set of tens of thousands of variants comes from would be nice!

I haven’t had any luck. Been trying to get a build out for the last 4 days without any success, goes for like 10-25 hours and get stuck (so it seems). I will post if I find anything relevant.

[Matt-Cranktrain]

Have you tried removing Vulkan and Metal from the list of Graphics APIs in the Player settings? That is the only thing that dropped the Shader Variant count low enough to finish a build in a reasonable time (20-30 minutes).

(I’m on Unity 2021.2b, but I was experiencing the same thing with Unity 2020 as well.)

Edit: Also, what is your project like? Do you have lots of Shader Graphs? Shaders from Assets on the store? Or is this literally a new Unity standard URP project?

[adamgolden]

When I upgraded from 2021.2 alpha to beta (initially), it was impossible to get a build to complete for the same reason (too many variants). I had to learn / integrate use of Shader Variant Collections. Here’s a thread that has more info on how to reference ONLY the variants you actually use in the project. Just ignore the parts about WebGL and missing/pink shaders - solves that kind of stuff too but unrelated.

WebGl no UI Toolkit visible (BETA 6)

In my experience with missing or pink materials in WebGL, it’s often been a shader not being included in the build. This might not be the issue here, but worth noting in case that’s all it is / for anyone else it might help. The easiest way to see if it’s related to the build missing a shader (and quickly resolve the problem, if so) is by creating a Shader Variant Collection. To do this, open Project Settings->Graphics tab, scroll to the bottom. It will say for example “100 shaders tracked, 250…

[Matt-Cranktrain]

@adamgolden

When I upgraded from 2021.2 alpha to beta (initially), it was impossible to get a build to complete for the same reason (too many variants). I had to learn / integrate use of Shader Variant Collections. Here’s a thread that has more info on how to reference ONLY the variants you actually use in the project. Just ignore the parts about WebGL and missing/pink shaders - solves that kind of stuff too but unrelated.
https://discussions.unity.com/t/851819/6

Hang on, unless I’m very mistaken (and happy to be corrected if so) setting Shader Variant Collections in the Graphics tab like you’ve suggested has no effect on what happens when you click ‘Build’. Andy Maloney wrote a big, helpful blog post on this last year - the point of these assets is to control when Shaders load in at runtime. So instead of instantiating a prefab, with a new Material/Shader and having a hitch in the gameplay at runtime, you can specify to load a ShaderVariantCollection at the beginning of the application, or even on a per-level basis, as Andy writes about.

Having looked into this already, and just in case I was wrong about this, I actually have previously done what you’ve suggested; I have an Asset assigned in the Preloaded Shaders - which lists “33 shaders, 64 variants”… and this has zero impact on what happens on build time which lists the same tens of thousands of shader variants, the same massive amount that FlightOfOne was experiencing in his original post.

[adamgolden]

@Matt-Cranktrain

Hang on, unless I’m very mistaken (and happy to be corrected if so) setting Shader Variant Collections in the Graphics tab like you’ve suggested has no effect on what happens when you click ‘Build’.

Not saying it’s the same for you or OP. In my case, it was what happened when I clicked build (too many variants errors preventing it from completing) that led me to even learning about variant collections in the first place. I first had to remove almost all of the “Always Included Shaders” from the list I had going (which worked in alphas and previously), because including them that way was where “Too Many Variants” turned out to be coming from. Unfortunately, just removing them led to lots of pink of course, and my investigation of how to force shaders to be included without just dropping them into that and without “too many variants” issues or pink led to collections. I’m not currently including any via graphics tab (…I now create and call WarmUp on them at runtime), but it was a reasonably quick/easy solution which worked for me when I first ran into the issue. I hope it helps someone else anyway - good luck

[FlightOfOne]

@Matt-Cranktrain

Have you tried removing Vulkan and Metal from the list of Graphics APIs in the Player settings? That is the only thing that dropped the Shader Variant count low enough to finish a build in a reasonable time (20-30 minutes).

(I’m on Unity 2021.2b, but I was experiencing the same thing with Unity 2020 as well.)

Edit: Also, what is your project like? Do you have lots of Shader Graphs? Shaders from Assets on the store? Or is this literally a new Unity standard URP project?

I just downgraded to 2020.16, see if that helps and started building again. I will try the Vulkan setting if this fails again. Thanks!

As far custom shaders, I use LUX (which I deleted most of it except what I need, like 3 remain), Vegetation engine ( author told me he only has 2 variations in there), and microsplat. This is it. I do not have many shader graphs either.

My very first build worked fine, though it took close to 8 hours.

Maybe I am on a fool’s errand trying to stick to LTS. I would upgrade to 2021 but I am afraid it would break microsplat or TVE.

[FlightOfOne]

@adamgolden

When I upgraded from 2021.2 alpha to beta (initially), it was impossible to get a build to complete for the same reason (too many variants). I had to learn / integrate use of Shader Variant Collections. Here’s a thread that has more info on how to reference ONLY the variants you actually use in the project. Just ignore the parts about WebGL and missing/pink shaders - solves that kind of stuff too but unrelated.
https://discussions.unity.com/t/851819/6

Thanks for this, I will give it a try, I am willing to try just about anything at this point just to get one successful build out. Will post you guys on what happens.

[FlightOfOne]

@adamgolden

Not saying it’s the same for you or OP. In my case, it was what happened when I clicked build (too many variants errors preventing it from completing) that led me to even learning about variant collections in the first place. I first had to remove almost all of the “Always Included Shaders” from the list I had going (which worked in alphas and previously), because including them that way was where “Too Many Variants” turned out to be coming from. Unfortunately, just removing them led to lots of pink of course, and my investigation of how to force shaders to be included without just dropping them into that and without “too many variants” issues or pink led to collections. I’m not currently including any via graphics tab (…I now create and call WarmUp on them at runtime), but it was a reasonably quick/easy solution which worked for me when I first ran into the issue. I hope it helps someone else anyway - good luck

By how much did this improve your build time?

[FlightOfOne]

@adamgolden

When I upgraded from 2021.2 alpha to beta (initially), it was impossible to get a build to complete for the same reason (too many variants). I had to learn / integrate use of Shader Variant Collections. Here’s a thread that has more info on how to reference ONLY the variants you actually use in the project. Just ignore the parts about WebGL and missing/pink shaders - solves that kind of stuff too but unrelated.
https://discussions.unity.com/t/851819/6

so, this is what I see. I definitely do not see 120,000+ variants, haha.

[adamgolden]

@FlightOfOne

By how much did this improve your build time?

No idea - at the time, I was mostly concerned with being able to build ever again at all after the update . I liked the “Always Included Shaders” thing, it was so hands-free compared to what I’m doing now. Previously I would warm up a shader whenever just by exposing a renderer in the scene (i.e. display for a frame somewhere out of sight with mesh bounds to 99999 to prevent culling), still have to for a few random things I can’t seem to be warm otherwise (I focus on WebGL mostly and that’s effectively single threaded… warming of shaders, large textures and meshes is pretty much unavoidable). If you just include a big variant collection with lots of shaders in Graphics tab, the startup time of your app/game will be slower vs. “always included” since those weren’t being warmed up during startup - so it’s worth reading over related API re WarmUp with custom collections, so if you want more control you can do it at runtime whenever you want (during loading bars, etc.), but depending on what you’re doing it could be fine and less to think about just having it at once at startup.

@FlightOfOne

so, this is what I see. I definitely do not see 120,000+ variants, haha.

Nope - so you would just Save To Asset, then drop that asset into a Preloaded Shaders slot. Also I notice you have Automatic for Lightmap and Fog modes, maybe try setting them to just what you need instead. You could also play with your “Instancing Variants” choice, though I don’t know enough about to offer more than that. Another thing to try (make sure you have a backup first of course)… is removing all of the “Always Included Shaders” after you have a collection with what’s used of them added to “Preloaded Shaders”.

[DonCornholio]

I am not 100% sure if this solves your problem, but least your build times can be massively reduced by using this shader stripping tool :

GitHub - SixWays/UnityShaderStripper: Modular utilities for shader stripping to...

With that you can manage what shaders get built and are put into the build.But you have to be thorough so you don’t miss anything you need. I have been using it for every project for almost two years now. It should be part of Unity to be honest. One thing you have to know though, from my experience it only properly works with shader_feature and multi_compile and not the new ones with _local at the end or _fragment.
You can also just comment out shader_feature and multi_compile directives in every shader which you don’t need.

[FlightOfOne]

@DonCornholio

I am not 100% sure if this solves your problem, but least your build times can be massively reduced by using this shader stripping tool :
https://github.com/SixWays/UnityShaderStripper
With that you can manage what shaders get built and are put into the build.But you have to be thorough so you don’t miss anything you need. I have been using it for every project for almost two years now. It should be part of Unity to be honest. One thing you have to know though, from my experience it only properly works with shader_feature and multi_compile and not the new ones with _local at the end or _fragment.
You can also just comment out shader_feature and multi_compile directives in every shader which you don’t need.

Thank you. I will take a look.

[FlightOfOne]

@adamgolden

No idea - at the time, I was mostly concerned with being able to build ever again at all after the update . I liked the “Always Included Shaders” thing, it was so hands-free compared to what I’m doing now. Previously I would warm up a shader whenever just by exposing a renderer in the scene (i.e. display for a frame somewhere out of sight with mesh bounds to 99999 to prevent culling), still have to for a few random things I can’t seem to be warm otherwise (I focus on WebGL mostly and that’s effectively single threaded… warming of shaders, large textures and meshes is pretty much unavoidable). If you just include a big variant collection with lots of shaders in Graphics tab, the startup time of your app/game will be slower vs. “always included” since those weren’t being warmed up during startup - so it’s worth reading over related API re WarmUp with custom collections, so if you want more control you can do it at runtime whenever you want (during loading bars, etc.), but depending on what you’re doing it could be fine and less to think about just having it at once at startup.

Nope - so you would just Save To Asset, then drop that asset into a Preloaded Shaders slot. Also I notice you have Automatic for Lightmap and Fog modes, maybe try setting them to just what you need instead. You could also play with your “Instancing Variants” choice, though I don’t know enough about to offer more than that. Another thing to try (make sure you have a backup first of course)… is removing all of the “Always Included Shaders” after you have a collection with what’s used of them added to “Preloaded Shaders”.

I tried all this and more but no luck. I am going to create a clean project.

[adamgolden]

@FlightOfOne

I tried all this and more but no luck. I am going to create a clean project.

You’ve tried deleting Library folder and let assets reimport? It’s not the same as just reimport all, that has fixed all kinds of random issues for me …I do it routintely once a week or so, this week it had grown to almost 40gb. After delete and create a build its less than 10gb, and things work more smoothly all around. I know it’s a common advice you probably tried a bunch of times already, just mentioning in case you haven’t and for anyone else.

[FlightOfOne]

@adamgolden

You’ve tried deleting Library folder and let assets reimport? It’s not the same as just reimport all, that has fixed all kinds of random issues for me …I do it routintely once a week or so, this week it had grown to almost 40gb. After delete and create a build its less than 10gb, and things work more smoothly all around. I know it’s a common advice you probably tried a bunch of times already, just mentioning in case you haven’t and for anyone else.

Yeah, deleting the the library has become a routine for me too (because of reasons you said above), do it every few days. And yeah, already tried :(.

[FlightOfOne]

First of all thank you very, very much for those tried/helped, really appreciate it -I learned a lot (including unrelated to building) these past few days, which helped also helped improve my performance.

It seems my 5 days of extreme bad luck and deadlock is over. I was finally able to do a successful build.

Very special thank to @adamgolden !! You saved me big time -you got me on the right track and is what ended up fixing the problem.

As a side effect (your other recommendations) it greatly improved my load times and of course the build time went down to like nothing (compared to 8 to 27hrs+). Also reduced my build size. Setting “Instancing Variants” to “Strip All” and using Preload is what did it.

I did notice shaders missing but I think I just need to add them in and everything should be ok.

I am not sure if there any other re-precautions to having these settings this way and if you know any please do share.

Here are my settings, hope it helps someone else.

[adamgolden]

@FlightOfOne

I did notice shaders missing but I think I just need to add them in and everything should be ok.

If nothing else seems to work for getting a specific shader to be included - you can try adding a material reference to a component somewhere in a scene you’re including in the build (i.e. drop a few into a public Material[ ] whatevers; ). …just be cautious of any textures being assigned to it or they’ll increase build size being included as well. Adding a Shader reference doesn’t work the same - it needs to be assigned to a referenced Material. Hope that helps get your stragglers sorted - good luck!

[FlightOfOne]

I finally fixed it for good. I created a clean project and transferred it to the new project manually. Painful, but I took that time to clean out the project. Something was probably corrupt. I would be surprised if something did not corrupt because of how many times unity hangs up (have to end task) crash.