Shader keyword system improvements in 2021.2 alpha

Unity Engine
,
1

Hello Unity users!

TL;DR:

Unity 2021.2.0a16 changes the way shader keywords work and removes fixed limits on the number of keywords used both per shader and globally.

First, the long-awaited part: fixed keyword limits are gone. There can now be up to 4.294.967.294 (2^32 - 2) global keywords per project. Local keywords are limited to 65534 (2^16 - 2) per shader or compute shader.

Next, we introduce a concept of keyword spaces.

Each shader has its own local keyword space, with no distinction between local or global keywords that existed before. All keywords declared in a shader are local. [EDITED on 08.06.2021] This, in turn, means that #pragma multi_compile and #pragma multi_compile_local do the same thing; the same is true for #pragma shader_feature and #pragma shader_feature_local. We will deprecate the pragmas with “_local” suffix in the future. The “_local” suffix in #pragma multi_compile and #pragma shader_feature directives determines whether the global keyword state can override the local shader keyword. The first appearance of a shader keyword wins in case of a conflict. [END OF EDIT]
Local keyword space is formed from all keywords declared in the shader, all keywords declared in passes that are added with UsePass functionality and all keywords from the shaders in the fallback chain. If a keyword in a fallback or in UsePass has the same name as a keyword in the main shader, it’s treated as the same keyword and counts only once. Unity automatically adds several builtin keywords to each local keyword space (UNITY_SINGLE_PASS_STEREO, STEREO_CUBEMAP_RENDER_ON, STEREO_MULTIVIEW_ON and STEREO_INSTANCING_ON). Any keywords added by shader compiler access plugins are also automatically added to each local keyword space.
For compute shaders the local keyword space is formed from all keywords declared in the compute shader.
The local keyword limit mentioned above applies to keywords in a single local keyword space.

Global keyword space is completely separate. Prior to rendering keywords from the global keyword space are converted to local keyword space. The conversion is based on keyword names. Enabling a global keyword FOO means that any shader or compute shader that has FOO in its local keyword space will have this keyword enabled.

A note on performance: local keyword spaces maintain a mapping from the global keyword space. When a keyword is added to the global space, the mapping gets updated on the first conversion from the global space to the local space. For best performance, add all global keywords as early as possible.

C# API changes.

We keep the current string-based C# API intact with some minor improvements: Shader.DisableKeyword and Shader.IsKeywordEnabled no longer create a global keyword if it doesn’t exist.

We also added a new API that is faster and has a clear separation between local and global keywords. Material, ComputeShader, Shader, CommandBuffer and ShaderKeywordSet classes have been extended to work with the new API - methods that accept instances of LocalKeyword and GlobalKeyword structs. Shader and ComputeShader also got a new member keywordSpace that can be used to query the local keyword space of a particular Shader or ComputeShader.
GlobalKeyword also has an explicit static method to create a new GlobalKeyword. The constructor does not create a new keyword if it doesn’t exist.
Shader.EnableKeyword and CommandBuffer.EnableShaderKeyword create a new global keyword if it doesn’t exist.
Shader.DisableKeyword, CommandBuffer.DisableKeyword, Material.DisableKeyword, Material.EnableKeyword and ComputeShader counterparts have no effect if a keyword doesn’t exist. Shader, Material and ComputeBuffer IsKeywordEnabled methods return false for keywords that do not exist.
We also added Shader.enabledGlobalKeywords and Shader.globalKeywords to query the enabled global keywords and all existing global keywords, respectively.

Stay tuned for more updates!

(EDITED on 04.06.2021)
We added new API to set shader keyword state directly. Material.SetKeyword, Shader.SetKeyword, ComputeShader.SetKeyword and CommandBuffer.SetKeyword are available from 2021.2.0a19 onward.

26 Likes
2

Known issues

  • Filtering objects in the scene hierarchy or opening Prefab variants from the hierarchy spams errors in the console (EDIT: fixed in 2021.2.0a17)
  • GI debug views are not rendering anything (EDIT: fixed in 2021.2.0a18)
  • Draw calls are more expensive (EDIT: fixed in 2021.2.0b3)
  • Material is marked as dirty if enabling a keyword that is already enabled or disabling a keyword that is already disabled (EDIT: fixed in 2021.2.0a18)

These issues are already fixed and will appear in a later alpha version.

2 Likes
3

Ah, only 65534 local variants? 640k is enough for anyone, huh?

(Rushes off to design a shader for 65535 local keywords)

7 Likes
4

Word!

1 Like
5

Glad to see this has been addressed, it’s been showing its age for quite a long while now.

1 Like
6

Good news thanks :slight_smile:

Please add ability to disable keyword in editor so trying to enable it for material will lead into some pinky-cyan shader.
Thus we can control usage of shader variants right in editor

1 Like
7

Not sure I understand. Do you mean “some list of keywords in the Editor that are not allowed to be turned on / show a warning when they get turned on”?

8

Mostly right :slight_smile:

Show warning is good but most important is pink shader to understand that this option is not allowed
e.g. LitUberShader I will disable NormalMaps and if someone add normal map in any material it will become pink, or pinky-cyan to distinguish from error shader, and ok there can be warning in log or/and in material inspector that this option was disallowed in project.

With this tool we can easily create set of shaders with variants that allowed in project and easily manage project ShaderVariant Count. For now we only can write strip scripts make build to test amount of shader variants used (usually 180000 just for Lit) and to see pink shaders that try to use stripped variants. With this tool this can be easily done in editor and may be strip Lit Uber Shader to just 10 variant used in project and all in Editor.

9

@JesOb we’ll think about this

1 Like
10

I think, in practice, these changes mean naming conventions should change from what most people use.

If my understanding is correct, there really aren’t global keywords from the shaders perspective anymore- only local. And when some code calls Shader.EnableKeyword, it sets this as a global keyword and sets this for every shader, regardless of if the intention was for that feature to be local or global.

Personally I would have preferred that these remain distinct spaces, so someone calling Shader.EnableKeyword to set a global keyword would not affect my local shader keywords. While global keywords are useful, most keywords in the shaders I write and use tend to want to be local, and are expressed as options on a per material basis for the user, not a game wide basis.

But that said, if this is the new system, I think it means wanting to prefix your keyword naming conventions to avoid collisions, which is kind of the opposite of what people did before local keywords (reusing common keyword names to avoid bloating the number of keywords). In fact, I’d suggest we simply start putting LOCAL or GLOBAL in the name of the keyword to mark it’s intension. Sure, if someone sets _LOCAL_NORMALMAP via Shader.EnableKeyword, they will still be setting it as global and changing it for every material anyway, but at least they can infer they are breaking the rules that way, instead of accidentally doing it because the intention is not clear.

This does make me wonder, if I have something like:

#pragma shader_feature _ _FOO _BAR

And I set _BAR on the material, but call Shader.EnableKeyword(“_FOO”), who wins? The code is only compiled for _FOO or BAR, not both, and the mixing of global and local space means that either the code has a priority feature, or both will end up getting set…

11

Yes, that’s correct.

This would leave some interesting side-effects. Suppose you have a shader that declares a keyword FOO inside a shader_feature_local and it has a fallback that has the same keyword inside a shader_feature. Should it be affected by the global keywords then? Should only the keyword from the fallback be affected?

That said, we’ll consider this, as it’s a behaviour change indeed.

Both get enabled and then either variant can be picked. I think it’s mentioned in the manual, but I can’t find it right now :slight_smile: The keyword system worked this way for ages, and I’d like to improve that as well.

12

Yes, because you explicitly declared that it’s control is provided by the material, not the global namespace. To me this seems more sensible, because it’s exactly what you told it to do. While the new system is clearly workable, it actually treats global keywords as local keyword overrides, NOT it’s own global keyword registry.

I’m also guessing that it doesn’t work bidirectionally - for instance, you might think you could use this new system in your game to disable all normal maps with Shader.DisableKeyword(“_NORMALMAP”); But since that just removes it from the global registry, it’s going to default back to whatever the material says instead of disabling the keyword globally, because there’s not really a clear on/off in the keyword system, rather defined or not. So with this system, you’re not really enabling and disabling keywords, but rather overriding them and not overriding them, globally. That is not at all clear from the function naming or description of global/local used. If global vs. local is explicit, then both the code in the shader and in the C# layer is clear and explicit about what it means.

6 Likes
13

That’s correct as well. But that’s how it used to work before the introduction of shader_feature_local :slight_smile:

14

wait, so even if you declare a feature as local it gets overwritten by global setting?

15

yes, that’s how the new system currently works.
But you said it already:

16

Or are we already talking about different “local”? :slight_smile:
Right now (2021.2.0a16+) the behaviour is like this: enabled global keywords will enable a keyword with the same name in all local keyword spaces, regardless of whether it’s declared as shader_feature_local, multi_compile_local, shader_feature or multi_compile.

17

Great news! One request, I think nearly every project has this function:

void SetKeywordEnabled(string keyword, bool enabled) {
  if (enabled) {
    Shader.EnableKeyword(keyword);
  } else {
    Shader.DisableKeyword(keyword);
  }
}

And variations for materials etc. Would be good to have as an actual official API

2 Likes
18

Noob question and forgive me if I missed this somewhere but, does this have any impact on materials and how they handle unused keywords?

For example the current LTS default behavior where if you try one shader and then switch to another very different shader on the same material, you need to manually clean up the disabled/inapplicable fields to avoid issues in some cases.

Would these changes also mean all local keywords are applied again from scratch or something like that?

19

Right; that’s the change I am not fond of, because I only see it causing issues. Ideally I’d prefer to define where a keyword is expected to be set, so that a local keyword is actually local, and not local only until some script somewhere that I didn’t write sets it globally but didn’t realize my shader uses the same named keyword.

otherwise I’ll start naming everything v defensively to minimize this chance “_BS_LOCAL_MYFEATURE”. Workable, certainly, but we have modern conventions like namespaces so we don’t need to do stuff like this anymore. If we are going to keep the overriding behavior than I would suggest we change terminology to match; keywords, and global keyword overrides, or something like that.

3 Likes
20

@pvloon it’s coming later :slight_smile:

@firstuser it will keep the keywords that exist in the new shader and drop those that don’t.

@jbooth_1 many projects rely on global keywords overriding material state, so we won’t change that.
Thanks for bringing this to my attention - we didn’t consider this particular behaviour (being able to define keywords that cannot be overridden by global keywords) important.

2 Likes