Stripping scriptable shader variants using ShaderVariantCollections?

I’m currently trying to implement shader variant stripping into our project - and the most obvious approach seemed to be using ShaderVariantCollections for that purpose. This would be a very simply workflow:

  • Create all the ShaderVariantCollections for the project by playing the game in the editor, using Settings / Graphics / Save to Asset (after clearing / tracking the shaders and shader variants). Relevant documentation.
  • Write an implementation of IPreprocessShaders.OnPreprocessShader() that uses those ShaderVariantCollections to strip out any shader keywords that are not used.

This could probably also be really useful to completely remove shaders from the project that are not used in any of the ShaderVariantCollections.

Unfortunately, it turns out that the Unity APIs that are available to Unity users don’t cover that use case. ShaderVariantCollections has shaderCount and variantCount, methods to add, remove or clear all variants, or test if a given variant is already in there - but no way to access the actual shaders or shader variants.

The ShaderVariantCollectionInspector accesses this information but I don’t see a way to access this from our own scripts, e.g. a custom implementation of IPreprocessShaders.OnPreprocessShader().

1 Like

@hippocoder and/or @aliceingameland could you ping Christophe about this? I have a rather hacky solution working now by taking ShaderVariantCollectionInspector and adding a button into the custom inspector that basically creates my own custom ShaderVariantCollection that has all the info available that I need. But it would be nice to have a more solid approach (I could post about it on the alpha-list - but it’s not really alpha-related, so I felt it’s more appropriate to post it here). It really seems to me that something built-in that’s based on ShaderVariantCollections would probably be the most elegant solution most of the time but I might be completely missing something.

Also, it turns out that stripping scriptable shader variants doesn’t help with the maximum number of shader keywords (256). So, one thing that I feel is also missing with this approach is a way to tell Unity to globally drop specific shader keywords right at the start of shader processing (and ideally even in the editor runtime). Something like “ignore shader keywords” in platform-specific project graphics settings might do the trick. It is often not feasible to change the shader files.

It gets worse: I removed my IPreprocessShaders implementation to have a new “reference build” where everything is included. But the build-time is still 35 minutes (before, a build was almost two hours), and it seems that some of the shaders variants are still missing. I have deleted ShaderCache and ShaderCache.db, as well as the ScriptAssemblies just to be sure. Also, I don’t get the log statements from my IPreprocessShaders implementation - but it very much looks as if I’m no longer getting those variants anymore.

Are the results from IPreprocessShaders cached somewhere?

Have you ever solved this problem? I have a similar idea and am clearing the variants collection for some shaders but they end up in the build anyway, wondering if anything is cached or not.

Our implementation for this is :

using System;
using System.Collections.Generic;
using System.IO;
using Agens;
using UnityEditor;
using UnityEditor.Build;
using UnityEditor.Callbacks;
using UnityEditor.Rendering;
using UnityEngine;
using UnityEngine.Rendering;
#if !AGENS_NO_SHADER_STRIPPING
class ShaderVariantsStripperProject : IPreprocessShaders
{
     
    private ShaderStrippingSetting shaderStrippingSetting;
    private static StreamWriter shaderStripLogFile;
    public ShaderVariantsStripperProject()
    {
        var shaderStripSettings = Resources.LoadAll<ShaderStrippingSetting>("");
        if (shaderStripSettings.Length == 0)
        {
            Debug.LogWarning("Did not find shader stripping settings asset , please create one. Compiling all shaders");
        }
        else
        {
            if (shaderStripSettings.Length > 1)
            {
                Debug.LogWarning("Warning More than one shader strip setting asset -- only using first found");
            }
            shaderStrippingSetting = shaderStripSettings[0];
        }
    }
    public int callbackOrder { get { return (int)ShaderVariantsStripperOrder.Project; } }
    public bool KeepVariant(Shader shader, ShaderSnippetData snippet, ShaderCompilerData shaderVariant)
    {
        bool resultKeepVariant = true;
        if (shaderStripLogFile == null)
        {
            string timeStamp = String.Format("{0}d_{1}m__{2}h_{3}m",
                DateTime.Now.Day, DateTime.Now.Month,
                DateTime.Now.Hour, DateTime.Now.Minute);
            shaderStripLogFile = new StreamWriter("Stripped_ShaderLog_" + timeStamp + ".txt");
        }
     
        var shaderKeywords = shaderVariant.shaderKeywordSet.GetShaderKeywords();
        string[] keywords = new string[shaderKeywords.Length];
        int i = 0;
        foreach (var shaderKeyword in shaderVariant.shaderKeywordSet.GetShaderKeywords())
        {
            keywords[i]=shaderKeyword.GetKeywordName();
            i++;
        }
        ShaderVariantCollection.ShaderVariant variant = new ShaderVariantCollection.ShaderVariant(shader,snippet.passType,keywords);
        if (shaderStrippingSetting != null)
        {
#if UNITY_IOS || UNITY_OSX
        resultKeepVariant = shaderStrippingSetting.collectionOfShadersToKeepMetal.Contains(variant);
#elif UNITY_STANDALONE_WIN
        resultKeepVariant = shaderStrippingSetting.collectionOfShadersToKeepPC.Contains(variant);
#endif
        }
        if (!resultKeepVariant)
        {
            string prefix = "not keepeing VARIANT: " + shader.name + " (";
            if (snippet.passName.Length > 0)
                prefix += snippet.passName + ", ";
            prefix += snippet.shaderType.ToString() + ") ";
            string log = prefix;
            for (int labelIndex = 0; labelIndex < keywords.Length; ++labelIndex)
                log += keywords[labelIndex] + " ";
            shaderStripLogFile.Write(log + "\n");
        }
     
        return resultKeepVariant;
    }
    public void OnProcessShader(
        Shader shader, ShaderSnippetData snippet, IList<ShaderCompilerData> shaderVariants)
    {
     
        int inputShaderVariantCount = shaderVariants.Count;
        for (int i = 0; i < shaderVariants.Count; ++i)
        {
            bool keepVariant = KeepVariant(shader, snippet, shaderVariants[i]);
            if (!keepVariant)
            {
                shaderVariants.RemoveAt(i);
                --i;
            }
        }
        if (shaderStripLogFile != null)
        {
            float percentage = (float)shaderVariants.Count / (float)inputShaderVariantCount * 100f;
            shaderStripLogFile.Write("STRIPPING(" + snippet.shaderType.ToString() + ") = Kept / Total = " + shaderVariants.Count + " / " + inputShaderVariantCount + " = " + percentage + "% of the generated shader variants remain in the player data\n");
        }
    }
     [PostProcessBuild(1)]
    public static void OnPostprocessBuild(BuildTarget target, string pathToBuiltProject) {
        if (shaderStripLogFile != null)
        {
            shaderStripLogFile.Close();
            shaderStripLogFile = null;
        }
     
    }
}
#endif

It seems to be working, we went from a memory footprint of 200 MB of shaderlab to 50 MB , which seems much more sane. However, the process of stripping the shaders seems to be very slow. 30 minutes added build time or so for metal.

The next thing I am going to try is to use the log file of what we are stripping out - because maybe there’s a configuration of the project that can be made, to not have Unity bundle that many variants to begin with.

Also, we are going to try to use the ShaderVariant collection to generate a more effecient strip file.
The hunch: it might be faster to use the shaderVariants file to generate a list of shaders&variants that probably should be stripped, and then directly strip based on that list - instead of calling shaderVariantCollection.Contains - which probably is the slow part of this mix.

I’ll report once I know more.

Update : We might be seeing similar caching based behaviour. Don’t really know what’s going on.

Update : also - we were returning Package in the shader stripping order, so that’s why it was so slow, I’ve updated the code to change it into Project

3 Likes

Something broke in our project big time after having the stripper run. The iOS build just crashes on startup - it seems to go to a shader - that is looking for a fallback and then into lza-decompress-code which finally crashes.

I’m deleting the Library folder now and rebuilding without stripping code - hope to have a build that runs again. It would be really nice to get the shader stripping features working. The variant explosion is so annoying (I understand why shaders are built like this though), especially when it’s not needed for the final product.

It would really be great if someone from UT could chime in. I still think this is a great and useful feature - but not if it breaks things the way it currently seems to break things.

1 Like

I would also be very interested in this. It’s just not feasible to have ~100MB of shader code in an Android build when I could define exactly which variants I actually want to use.

We got it quite stable now and it has reduced our memory footprint by maybe 250 MB - our build times are good as well. I will write a blog post and share our code soon. I think maybe our project was configured in a very unfortunate way in regards to shader variants, because our team is small and no one had complete view over these things until we had to figure out a way to reduce them.

Also the weird issues we experienced at the beginning are no longer an issue - we have since upgraded unity as well as done some changes to our code : we are using the 2018.4 LTS version.

4 Likes

Any news on this? ShaderVariantCollections still don’t expose their variant lists.

2 Likes

Hi there ! Any Update on this? We’re a small team as well and would love to see your solution to this.

Hey there ! I just found this amazing asset on github :

This will basically give you all the control and flexibility you need and is very easy to use.
I don’t know if this is the solution havchr promised but, and i still would love to read that blogpost, but this solves my problems so far :slight_smile:

1 Like

I was about to finally post my code but I would rather suggest using SixWays UnityShaderStripper instead - it is much better.

1 Like

Hi guys!

What about of preventing stripping? I need the same, but in another direction. All shader keywords must not be stripped if they are used by my svc.
Because I’ve stucked with a bug, when I build shaders from scene in another bundle, shader variants just being stripped. But If I build them to 1 bundle, all is fine and actually single scene bundle size is bigger than scene+dependent shaders bundle(but it supposed to be vice versa).

One more thing. Could anybody explain, why IPreprocessShaders.OnProcessShader
callback is being invoked few times for the same shader? Looks like first time it is being invoked with the highest number of varaints, and then number is reducing. I suppose it should invokes only once per shader.

Hi @havchr … Do you have the next class ShaderStrippingSetting to try to use your quoted example.
.collectionOfShadersToKeepMetal.Contains(variant)
and manipulate the resource

Thanks!!

1 Like

I would heavily advice you using SixWays UnityShaderStripper instead - it is much better. https://github.com/SixWays/UnityShaderStripper

1 Like

Thanks!

For me, the OnProcessShader is not called for all shaders. Who has any thoughts, why could this be? Is there a limit on the number of processed shaders?

@ZooxSoft it should be called on everything that goes into the build.
Which Unity version do you use?

I believe OnProcessShaders is called for all shaders Unity think will be used in the project. It will behave differently if you have Strip All, Strip Unused or Strip None enabled.

See this picture : 7568842--937102--upload_2021-10-13_8-26-11.png
(From this article)