New shader preprocessor

Hello Unity alpha users!

2020.1.0a15 is out and it brings a new shader preprocessor (experimental). You can enable it by passing a command line option “-force-new-shader-pp” to the Editor executable.
EDITED on 09.01.2020: As of 2020.1.0a18 you can enable it per project via Project Settings → Editor Settings → Caching Preprocessor checkbox.
EDITED on 03.05.2020: As of 2020.2.0a8 it’s possible to override preprocessor in the shader inspector on a per shader basis.
EDITED on 04.06.2020: As of 2020.2.0a13 Caching Preprocessor is now the default option. It is still possible to turn it off either in project settings or on a per-shader basis.

It’s going to be faster (performance improvements coming in 2020.1.0a16, shader compilation time is reduced by 5-25%), comes with new features and follows the C standard more thoroughly.

New features

Show preprocessed source
Shader inspector now has a checkbox “Preprocess only” which toggles between compiling shader variants and showing preprocessed source. The checkbox is only active for the new preprocessor.

Conditionals can now affect #pragma directives
#pragma directive parsing is now done using the new preprocessor, which means preprocessor conditionals can be used to influence, which #pragma directives are selected.

Restrictions:
#pragma directives are not searched for in include files (this would increase shader compilation time significantly).

Conditionals will affect #pragma directives if the expression depends only on:

  • Any custom #define
  • The following platform keywords: SHADER_API_MOBILE, SHADER_API_DESKTOP, UNITY_NO_RGBM, UNITY_USE_NATIVE_HDR, UNITY_FRAMEBUFFER_FETCH_AVAILABLE, UNITY_NO_CUBEMAP_ARRAY
  • Unity version macro

The list may be expanded in the future.

Accurate include dependency tracking
The same rules that are used for conditionals affecting #pragma directives are now applied to include dependency tracking. This reduces the chance of shader reimport when some include file changes.

Support for #pragma once directive
Instead of writing the regular include guards

#ifndef MY_ASWESOME_INCLUDE_FILE_INCLUDED
#define MY_ASWESOME_INCLUDE_FILE_INCLUDED
<actual include code here>
#endif //MY_ASWESOME_INCLUDE_FILE_INCLUDED

It’s now possible to just write

#pragma once
<actual include code here>

Support for #pragma warning directive
#pragma warning behaves the same as #pragma error, but issues a warning.

New #include_with_pragmas directive (from 2020.1.0a18)
This is a new directive introduced for the new preprocessor. It is exactly the same as the standard #include directive, but it is not ignored during shader import time, which makes it possible to put shader snippet parameters (#pragma directives) in an include file.
This directive has to be used explicitly (e.g. if you use #include_with_pragmas inside a file included using a regular #include directive, it won’t be checked).

Differences in behaviour with the previous preprocessor

  • Octal number recognition is deferred to the compiler unless the number is part of a preprocessor conditional expression.
  • No number truncation to 64 bits happens in the preprocessor unless the number is part of a preprocessor conditional expression.
  • String with escape characters are now handled correctly.
  • #line directive can use non-10-base integers as line number
  • #line directive arguments will be macro-expanded if they don’t match or <Integer + String> pattern
  • FILE and LINE macros can be used.
  • “\\n” is not treated as a space anymore.
  • “\\n” can now be used anywhere in the code, for example “#\\nd\\ne\\nf\\ni\\nn\\ne\\n A\\n 1\n” is equivalent to “#define A 1\n”
  • When a macro is redefined, the new preprocessor issues a warning if the definition is different and uses the last definition.
  • defined(x) can now come as a result of macro expansion
  • Macros with 0 parameters are now supported. Expamle: #define A() B
  • Self-referring macros are now handled correctly. Example: #define A 1 + A
  • Empty macro arguments are now supported. Example: #define A(x, y)… A(,)
  • Commas inside brackets (“[ ]”) and braces (“{}”) will now be treated as a delimiter between macro arguments.
  • Generic preprocessing numbers are now recognized as such.
  • Self-expanding macros are now handled correctly.
  • In general, macro expansion now follows the C standard.
  • Ill-formed #undef directives are no longer ignored when the first argument matches .
  • Full stringizing support.
  • Concatenation works within macros without arguments. Example: #define A x ## y
  • Invalid concatenation now issues a warning.
  • Macro parameters are not macro-expanded when used for concatenation or stringizing.
  • Character constants are now treated as integers.
  • No support for % operator in conditionals.
  • Shift operators are now supported in conditionals.
  • #include directive arguments are now macro-expanded if they don’t match the regular format.
  • When the new preprocessor encounters a macro name without parameters no substitution happens (previously resulted in an error).

Restrictions (EDITED on 20.12.2019)

The following platforms are supported starting from Unity 2020.1.0a17:

  • Xbox One
  • PS4

The following shader types are supported starting from Unity 2020.1.0a17:

  • Compute shaders

The following shader types are not yet supported:

  • Ray tracing shaders
  • Surface shaders

These will gradually become available in the upcoming alpha releases.

Thanks for reading this far :slight_smile:
Stay tuned!

10 Likes

Greetings Unity alpha users!

Unity 2020.1.0a17 is out, and brings Xbox Ona and PS4 support as well as Compute shader support.
We also introduced UNITY_OLD_PREPROCESSOR macro, which is defined only for the old preprocessor.

More to follow! :slight_smile:

1 Like

Maybe also update the initial post “Restriction” section?

1 Like

@Peter77 updated, thanks for the suggestion!

1 Like

Thanks for this - really exciting. Will it eventually cover performance improvements or is it mostly going to be a quality of life authoring thing?

No complaints though, quality of life is a very nice thing in shader land :slight_smile:

1 Like

As it’s only the preprocessor changing, the shaders should be the same, so runtime performance shouldn’t be affected :slight_smile:
The new preprocessor should not be slower. I’ve seen anything between 1% and 25% faster import and compilation time when using the new one, but I haven’t seen it perform slower than the old system. If you find a case where it is slower, please report a bug.

1 Like

It’s awesome that #pragma’s will finally be conditionally compilable (although the only use case I have left, that I haven’t ended up with good workarounds for, is: including multiple vert/frags within a single file).

When you say:

… how bad is it, and why can’t we make that decision ourselves?

Also … any chance for conditional-compile of passes? :slight_smile: (currently I’m scouring the forums for best practices on ways to clean-up code that has divergent use of multiple passes, and finding only ugly hacks :(. On the plus side, it led me to this thread, and it’s good to see what’s coming in 2020.1, even if I won’t be able to use it for another year or two once everyone else has upgraded)

Great news!

Could you elaborate on this one? Was this not possible before? For instance this works in 2019.2:

#define DEBUG defined(DEBUG_MODE) && !defined(RENDERING_CAMERA)
// and later:
#if DEBUG
// code
#endif

Do you have any suggestions on how to toggle different debug modes of a compute shader from script?

In a normal shader I’m able to define keywords and then toggle that keyword on a Material by script. Something similar would be super useful for compute shaders as well. Currently I have to enable debug modes by uncommenting #defines in the shader. I can use bool properties, but that’s bad for performance and isn’t compiled out of the shader when not needed.

And just after posting this I’ve seen that multi compile for compute shaders is coming in 2020.1 as well! Compute shaders will be even more powerful! Thanks a lot for working on this!

something like

#if SHADER_API_DESKTOP
#pragma vertex DesktopVert
#else
#pragma vertex MobileVert
#endif

should work now :slight_smile:

too bad to allow this directly (several times longer shader compilation)

we’re rolling the feature out step by step, just wait a bit :wink:

the change would be too drastic, so unlikely

some more advanced macros did not work properly with this, can’t say from the top of my head, what exactly didn’t work.

4 Likes

Happy New Year everyone!

2020.1.0a18 is out and it bring two more updates:

  • New preprocessor can now be enabled/disabled per project (Project Settings → Editor Settings → Caching Preprocessor checkbox)
  • #include_with_pragmas directive ( @a436t4ataf this is the thing you were asking for ;))

I updated the original post with details.
Stay tuned!

2 Likes

Will it be on by default in a new 2020 version?

@ROBYER1_1 the plan is to make it default during the 2020 cycle, most likely during the 2020.2 alpha.

It would be super useful if the new preprocessor also worked outside of the HLSL blocks. Primarily in the Properties block, but also while setting up render states. Using an include and not having to constantly copy-paste all those shared properties would be a huge improvement.

1 Like

Well, one doesn’t need a preprocessor for that. Just some additional syntax to allow such behaviour :slight_smile:

@aleksandrk This preprocessor (on 2020.1.0a24) seems to fall over looking for Xbox Shaders and Blit shaders? This happened when I activated the preprocessor and reopened my project. It has 2 shader graphs in. I can try to reproduce on a smaller repro but I was using Universal RP at this commit https://github.com/Unity-Technologies/ScriptableRenderPipeline/commit/1799d2ceeba0a7ea257eb6cfa6335651e489a3c8

Shader error in 'Hidden/Universal Render Pipeline/Blit': Couldn't open include file 'Packages/com.unity.render-pipelines.xboxone/ShaderLibrary/API/XBoxOne.hlsl'. at Common.hlsl(159)

Shader error in 'Universal Render Pipeline/Lit': undeclared identifier 'unity_Builtins0' at SpaceTransforms.hlsl(12) (on d3d11)

Shader error in 'Universal Render Pipeline/Nature/SpeedTree8': undeclared identifier 'unity_Builtins1' at SpaceTransforms.hlsl(12) (on d3d11)

Shader error in 'Graphs/MeshRenderer/MeshRenderer-ColorRim': undeclared identifier 'unity_Builtins0' at SpaceTransforms.hlsl(12) (on d3d11)

It all goes away if I disable that new preprocessor and hit play. Also causes shadergraph to be unusable while erroring out before I disable the preprocessor.

@ROBYER1_1 a fix is coming for the “Can’t open include file” issue, I’ll post here whan it’s available in 2020.1 :slight_smile:
If you can make a small repro for the others, would be great!

@ROBYER1_1 a fix for “Couldn’t open include file” will be available in 2020.1.0b2.

Thankyou for this, I will get round to submitting a bug report for the other shader issues next week. Have had a lot of shader unrelated XR bugs to report during our development!

Any word on when 2020.1.0b2 will hit?