I have a few different questions here. Compiling shader variants has practically ground progress on our project to a halt. I’ve added a script via the Shader Variant API (not sure of official name - but the IPreprocessShaders interface), to inspect what variants it is building. Through this, it has become apparent that it is building over 1 million variants. Sometimes, this takes hours. Strangely, sometimes it takes about 20-30 minutes. Either way, it’s killing us on this project.
I’ve investigated in more depth and have some findings:
a) There are many many instances of the same shader+keyword configuration - sometimes over 900. I printed out all values I could and everything matches on each of those unique instances.
b) In a sample project, I was able to reproduce this by simply adding a new material referencing the Standard shader - as long as the material was being used in a scene. With 1 material, I got 2 instances of each unique configuration. With 2 materials, I got 3 instances, and this then grew linearly as more materials were added.
c) I tried using the shader variant API to ignore all unique variants after the first one it saw. This didn’t work - faster builds, but everything in-game was magenta.
I have 3 main questions:
Why does Unity increase the number of variants linearly with each material use? Surely it only needs one copy of each possible configuration of the shader+keywords at runtime, so why is it building many many times more than it needs to?
Why does Unity build shader variants EACH time the game is built. I’m building for PS4 (in this example), and each time I build the window pops up saying it’s building shader variants. Nothing changed since last time I built so why isn’t this work cached and reused for subsequent builds on the same platform?
What, if anything, can we do to fix this so we’re no longer basically blocked on this project. Unity was meant to save us money by improving our iteration times but at this point it has ground us to a halt.
P.S. I have read through this thread already ( Compiling Shader Variants Taking Ages ) and tried everything suggested. Nothing works and there’s no response from Unity in the thread.
Are you using a shader variant collection asset? If not, Unity will iterate over every material in the project and ask to compile every single variant. My understanding is it is not actually recompiling them every time, but checking the disk first and skipping if found. However on modern PCs it probably takes longer to access the disk than to just compile the variant and re-save it every time, especially if you’re not running your project off of a fast SSD.
I do believe this is an area of active work over at Unity. They’ve been seeing issues with shader compile times for LWRP and HDRP builds, in large part because they’ve internally finally moved to having internal groups building larger projects that better reflect real world use cases than the hyper focused demos of the past.
Each time a material is encountered, we need to make sure the shader variant is actually compiled. What you see in the dialog that says “I’m building shader variants” is not necessarily actually building shader variants. If a variant has been compiled already, it’s getting it from the shader cache in the library folder.
I suppose what’s happening when you do your own IPreprocessShader and remove variants that you think are identical, it will think that it doesn’t need to reference this variant, and doesn’t tell the material, where to look for its shader (or something similar).
When you see a duplicate, it doesn’t mean it gets recompiled.
When your build takes hours instead of 20-30 minutes, this means that something triggered shader recompilation, for all shaders or just some shaders. This can happen if you update Unity (but it’s not always the case), if you modify an include file, or modify shader source.
Ok, good to know that it’s not actually compiling them - but it’s mostly a moot point as it’s still taking an extremely long time, regardless - so I’d still like advice on how we can improve it.
Furthermore, the numbers we are seeing just seem pretty insane. I’ll give an example - prior to doing some pretty aggressive stripping in my IPreprocessShader, the profiler was showing memory usage on PS4 under the “Unity” category as 1.4GB. After some aggressive stripping, the profiler now shows 0.6GB under the “Unity” category. This implies, of course, that 800MB of memory was being used up by shaders, or some other bookkeeping that is tied to shaders. This is pretty crazy, especially given that it’s the “default” behaviour and I’ve had to write this IPreprocessShader to get it back under control.
Does this seem out of whack to you, @aleksandrk ? Does it sound like a bug, or is this what people should be expecting to see? For the record, this is a VR project with Single Pass Stereo on - so I understand that increases the number of shader variants… but… 800MB??
Maybe we have to create a ShaderVariantSettings.asset file, that contains the properties to skip all shaders? (I don’t know. All I saw was an option to skip all shaders.)
@tinto22 I’m not sure about PS4
I’ll let the team that works on it know about this thread.
Many shaders come with this line: #pragma multi_compile _ UNITY_SINGLE_PASS_STEREO STEREO_INSTANCING_ON STEREO_MULTIVIEW_ON, which quadruples the amount of shader variants. There is some code that strips unnecessary variants, but I’m not sure, what the end result is. So, in theory, it could mean a lot of extra memory.
Hi @sambickley , we are experiencing really slow build times on PC too, so it’s not PS4 specific.
For the memory issue I mentioned above, though, I’d need to confirm on PC as I was working on PS4 when I discovered it. The problem is that to determine that, I’ll need to do hours-long builds. I’ll try to do that today anyway.
While the issue is more pronounced on PS4, it does still happen on PC. Here are the results of my tests today, all PC/Standalone builds:
Build 1:
Graphics Settings - Built-in Shader Settings. All dropdowns set to “Built-in shader”
Graphics Settings - Shader Stripping Lightmap Modes. All checked/ticked.
IPreprocessShader - No stripping at all
Memory Profiler Results
GfxDriver - 1.02GB
Unity - 1.06GB
Build 2:
Graphics Settings - Built-in Shader Settings. All dropdowns set to “No support”, except Screen Space Shadows
Graphics Settings - Shader Stripping Lightmap Modes. All checked/ticked, except “Realtime Non-Directional” and “Realtime Directional”
IPreprocessShader - No stripping at all
Memory Profiler Results
GfxDriver - 0.57GB
Unity - 0.88GB
Build 3:
Graphics Settings - Built-in Shader Settings. Same as build 2
Graphics Settings - Shader Stripping Lightmap Modes. Same as build 2
IPreprocessShader - Aggressive stripping, including stripping out all use of the Standard shader, which we don’t use directly (we have a lot of custom surface shaders).
Honestly if you did your research properly, this is resolved by creating proper shader variant collections, turning off “optimise mesh data” etc etc. We went from 4 hour shader compilation + build time down to 11 minutes for huge project.
But to be honest if this is enough to make you switch engine, I guarantee game dev is not for you because you will run into difficult to solve issues in any engine eventually. Thats what game development is, problem solving.
Good luck but as a user of both unreal and unity, I think your comments about unreal vs unity are optimistic at best and show you have not used unreal in depth or for very long. You will run into a whole set of other problems once you realise how difficult it is to make anything other than a few types of games, and the mess / nightmare that blueprints is and being unable to do certain things without it.
I am certain I will see both of you back here in a month or so.
Rubbish. Unity very much is game engine, unless you think 3d rendering frameworks contain physics, audio, pathfinding + AI tools, timelines, scripting etc etc.
No reason to be so condescending. I have worked for many years with Unity and it’s always been a struggle to fix what Unity should solve out of the box. Unreal won’t be drag&drop-easy as Unity tries to always imply in their shiny 1-minute trailers but at least I have the source of the engine and many, many more (integrated!) tools Unity is lacking of.
I don’t have time to argue about engines with you. Think whatever you want but I am done with Unity. If it works for you, great. Unity is missing lots of editor tools and basic functionalities for years and they obviously seem to prioritize adding more unfinished features instead of finishing and making existing tools usable on a production level.
I’m here because my game takes longer to compile after upgrading, but a few posts here were off-topic. I do agree with @MadeFromPolygons_1 about the point that game dev is about problem-solving. Personally, I use both engines and have spent over 8 yrs on Unity, if there’s an engine way better than its competitors, that would be an easy choice. The real problem of Unity is not what features are not good enough, it’s lack of docs. When Unity says something is “Preview” or “Not production-ready”, they mean it, but people usually don’t understand clearly about what prevents them to do so.
Regarding this topic, I think @aleksandrk explained it well. What we need is transparency and best practice guideline, black box is never OK for enterprise productive.
