Unity is adding a new "DXC" HLSL compiler backend option

Hey! So Unity has been using Microsoft's "FXC" HLSL compiler (plus our own HLSLcc when targeting Vulkan/Metal/OpenGL) for years. However FXC has a bunch of issues, like no support for some modern GPU features, extremely slow compile times for some shaders, and so on.

Over the past year we've been chipping away at integrating Microsoft's "new" HLSL compiler, called "DXC". Right now in 2021.2 alpha builds it kinda mostly works, with a bunch of caveats and gotchas. On the plus side, it does support modern GPU features (like wave/simd-group functions), and overall is about 4x faster at compiling shaders than the old compiler was.

If you have a bunch of shader code written or are just interested in playing around with it, we'd love your feedback! The new compiler is completely opt-in per-shader.

Here's a long google doc: DXC shader compiler integration, 2021.2 - with instructions how to use it via a #pragma in a shader, what works (DX12, Vulkan, Metal), what does not work (DX11, GL), what are the various HLSL syntax bits that were supported by the old compiler but not by the new one, and so on.

At this point due to various gotchas (see the doc above) we would not recommend using DXC in production, but testing your shader codebases with it, or playing around with it would be much appreciated.

8 Likes

I tried to use #pragma require WaveMultiPrefix to enable shder model 6.5 features.
I'm using Unity2020.2.0b14, URP 11;
The gfx backend is DX12, on windows platform.

But these following error occurred:

  • Shader Compiler Socket Exception: Terminating shader compiler process
  • Shader Compiler: Compile StampVertexGenerator.compute - StampVertexGenerator_SpawnStampVertices: Internal error communicating with the shader compiler process. Please report a bug including this shader and the editor log.

The document says FXC doesn't support ray tracing shaders, but then how did Unity compiled them up to now?

1 Like

[quote=“Neto_Kokku”, post:3, topic: 835265]
The document says FXC doesn’t support ray tracing shaders, but then how did Unity compiled them up to now?
[/quote]
Unity already used DXC just for the raytracing shaders.

[quote=“JiangBaiShi”, post:2, topic: 835265]
I tried to use #pragma require WaveMultiPrefix to enable shder model 6.5 features.
I’m using Unity2020.2.0b14, URP 11;
The gfx backend is DX12, on windows platform.

But these following error occurred:

  • Shader Compiler Socket Exception: Terminating shader compiler process
  • Shader Compiler: Compile StampVertexGenerator.compute - StampVertexGenerator_SpawnStampVertices: Internal error communicating with the shader compiler process. Please report a bug including this shader and the editor log.
    [/quote]

We could not reproduce this on 2020.2.0b14.3668 by just adding #pragma require WaveMultiPrefix to any shader. Could you file a bug report with the exact shader as the error message mentions, preferably from your exact your version of Unity.

Note that our DXC backend has been updated multiple times since 2020.2.0b14 so the crash is likely fixed in latest 2021.2

However, in 2021.2 we do now observe that #pragma require WaveMultiPrefix produces “error: validator version 1,4 does not support target profile.” which looks like a bug with DXC or with the Windows SDK. We will look into this more.

1 Like

This is both exciting and terrifying. Having barycentrics available is something I've often hacked around with baking colors into the mesh, being able to get them natively will be really huge. And a 12x compile speedup on a ton of shaders is amazing.

Currently I'll have to adapt things for a few issues present in the doc. Better Shaders uses VFace instead of SV_IsFrontFace, but that should be an easy switch. The one which is harder is being able to write to a global or cbuffer variable - I use that pattern a lot with Stackables, as coupled with a define it makes it easy for different stackables to leave optional scratchpad data for other stackables to use. However, I can come up with some kind of blackboard concept that builds a struct that gets passed between stackables to get around this I think. To be clear, this means you cannot write to a variable in the global namespace or in the PerMaterial CBuffer from within the shader, correct? Also, another doc mentions a -gec flag - is there a downside to using this flag to maintain backwards compatibility? And could it be enabled somehow from the shader?

The move to a #pragma require system over shader targets also sounds great - but can the requirements not be inferred from parsing the code? IE: If I use barycentrics, then the requirement is known. (Though I suppose this would require understanding the state of various conditional compiles, which could be involved).

tex2D is extremely common in shaders and apparently not allowed anymore (and I assume all the old variants as well). That seems like a huge one, especially for people who are quickly porting surface shaders to Better Shaders, since switching every texture sample around is a lot more work (it's almost copy/paste now). Perhaps adding warnings about it to the current compiler would be a good thing to get people to start moving now?

Given the current state in that doc, it does seem like it'll be a few years before the old compiler can be pulled. Still, I'd like to get BetterShaders in compliance sooner rather than later, as the community using it is still pretty small.

[quote=“jbooth_1”, post:6, topic: 835265]
Having barycentrics available is something I’ve often hacked around with baking colors into the mesh, being able to get them natively will be really huge
[/quote]
FWIW barycentrics do require GPU (and the graphics API) to support it, so for example they will never work on DX11 :frowning:

Yeah, this means you can’t write into any cbuffer (including the implicit globals cbuffer, i.e. loose global variables).

“-gec” flag has downsides in that while it enables this “allow writing into cbuffers” behavior, it disables some of more recent DXC features, so it’s kinda neither here nor there.

The problem is that you need to pass the “which shader target model?” to the DXC (and FXC) compiler before starting compilation, so it’s either “compile twice, once to max supported feature set to find the features used, then into the minimal featureset that still has all the features”, or to do a text-based search through the shader for what is used or not (and text based searches are extremely brittle).

That’s actually not a problem! We emulate it on DXC just like we emulate lack of it in some console shader compilers. If you look at HLSLSupport.cginc in recent Unity versions, there’s like

// DXC no longer supports DX9-style HLSL syntax of sampler2D, tex2D and friends.
// Emulate those using our own small structs & functions that have a combined sampler & texture.
#if defined(UNITY_COMPILER_DXC) && !defined(DXC_SAMPLER_COMPATIBILITY)
#define DXC_SAMPLER_COMPATIBILITY 1
struct sampler1D            { Texture1D t; SamplerState s; };
struct sampler2D            { Texture2D t; SamplerState s; };
struct sampler3D            { Texture3D t; SamplerState s; };
struct samplerCUBE          { TextureCube t; SamplerState s; };

float4 tex1D(sampler1D x, float v)              { return x.t.Sample(x.s, v); }
float4 tex2D(sampler2D x, float2 v)             { return x.t.Sample(x.s, v); }
// ...

And then the rest of the code can use it and things work out fine.

Yes quite likely some time until we can pull it. It will have to happen at some point though, which is exactly why we thought it would be good to give a heads up on “hey DXC is coming!”.

[quote=“Aras”, post:7, topic: 835265]
FWIW barycentrics do require GPU (and the graphics API) to support it, so for example they will never work on DX11 :frowning:
[/quote]

Damn, so stuffing colors for the next 8 years or so. One day we’ll have nice things…

[quote=“Aras”, post:7, topic: 835265]
That’s actually not a problem! We emulate it on DXC just like we emulate lack of it in some console shader compilers. If you look at HLSLSupport.cginc in recent Unity versions, there’s like
[/quote]

Great. I switched over from VFace to SV_isFrontFace this morning- does that have any issue on older APIs? I chose VFace because it was the older semantic and potentially supported on more platforms, but wasn’t sure if SV_isFrontFace was.

Which reminds me, it would be very nice if these abstraction layers were included for switch/xbox/ps4 - I suspect they are not for legal reasons, but they are all over the net, even in Unity’s own public github hosted projects. So I’ve managed to hack together my own version for Better Shaders from multiple sources, but I can’t know if they are really correct or updated because I don’t have hardware access and the sources are from random versions of Unity. (I’m pushing users to write stuff using the newer semantics, so make the same API available on Built In, because it’s a much better abstraction layer than the sorta “This is what we needed to wrap” version in Built In)

But perhaps a solution is just to include them into the engine, so I can reference them there regardless of which render pipeline is used or which consoles are installed. Since SRPs are now tied to Unity versions, having a way to path to them in the engine would make sense- and then when the abstractions are updated with new consoles or whatever, everything keeps working and is updated automatically. Right now I’m stuffing them into every pass of a shader for the built in pipeline because it’s the only way to keep the file portable (since I don’t want to require any include files be installed). So that would save about 8k lines of cruft if I could just #include them from somewhere.

[quote=“jbooth_1”, post:8, topic: 835265]
VFACE
[/quote]
If you look at the compiled shaders, Unity’s been replacing VFACE with SV_IsFrontFacing and adding a ternary to get a -1.0 or +1.0 value for while because GLSL has never supported VFACE. GLSL used gl_FrontFacing from the start, which is equivalent to SV_IsFrontFacing. hlsl2glslfork changed that back in 2014. It’s really only Direct3D 9.0 that used VFACE.

1 Like

[quote=“jbooth_1”, post:8, topic: 835265]
it would be very nice if these abstraction layers were included for switch/xbox/ps4 - I suspect they are not for legal reasons
[/quote]
Yup, various bits of various NDA platforms can’t be included everywhere easily.

[quote=“Aras”, post:10, topic: 835265]
Yup, various bits of various NDA platforms can’t be included everywhere easily.
[/quote]

Right, and that’s understandable - but if I could include a file that existed in the engine (not in the packages) which included all the various platforms that are installed on a machine, then it wouldn’t matter. Right now, because all of that stuff is shipped in packages, I have to put together my own version of this stuff for the built in pipeline because it doesn’t exist as an include in that pipeline (well, it has it’s own version- but the macro’s are not the same or complete, so you’d end up having to write different code for SRPs vs. standard again). Since SRPs are going to be shipping as part of Unity now, would I be able to include those files from srp core even if the user doesn’t have an SRP installed/active?

[quote=“TLukas”, post:5, topic: 835265]
We could not reproduce this on 2020.2.0b14.3668 by just adding #pragma require WaveMultiPrefix to any shader. Could you file a bug report with the exact shader as the error message mentions, preferably from your exact your version of Unity.

Note that our DXC backend has been updated multiple times since 2020.2.0b14 so the crash is likely fixed in latest 2021.2

However, in 2021.2 we do now observe that #pragma require WaveMultiPrefix produces “error: validator version 1,4 does not support target profile.” which looks like a bug with DXC or with the Windows SDK. We will look into this more.
[/quote]
Hi, Sorry for the late reply, I tried to upgrade Unity from 2020.2.0b14 to 2020.3.4f1c1, and upgraded my graphics dirver & windows OS to the newest version. However this problem still presents in my project:
I tested both on Quadro M2000 and GTX 1080, both produces “error: validator version 1,4 does not support target profile” when wavemultiprefix is used in shader code. Can I solve this issue by alternating my windows SDK version? Or I can only wait for you guys to fix it:(

This is pretty neat, I'm looking forward to trying it out.

Any estimates on how long until it's ready for production? I'm working on a project which is still a while away from release, and is targeting console and PC. So I'd have plenty of time to catch any bugs that pop up.

Will this make it possible to support proper sampler states in shaders, with the full range of features such as anisotropy, border modes, additional filter modes (Such as min/max, which are handy for Hi-Z generation) etc? There's not really a reason to couple textures and sampler states anymore, right? (Performance-wise, it's also better to re-use sampler states within a shader)

eg:

SamplerState _LinearSampler
{
    Filter = MIN_MAG_MIP_LINEAR;
    AddressU = Border;
    AddressV = Border;
    BorderColor = float4(1, 0, 0, 1);
};

Some other features such as being able to sample a read-only depth buffer, stream-out, adjacency topology, and dual source blending are also noticeably absent and would be handy for advanced graphics features, and I'm sure HDRP can utilize some of them too.

[quote=“Arycama”, post:13, topic: 835265]
Will this make it possible to support proper sampler states in shaders
[/quote]
No, that’s completely unrelated to the shader compiler backend. So this will not change by itself just because the compiler is different.

All of these are also not related to the shader compiler being FXC or DXC. If someone from the graphics teams will decide to expose them, then great. But none of them are related to the shader compiler.

1 Like

I finally got around to try this as well, and while I get this to work:

fixed4 frag (v2f i, float3 PerspectiveBaryWeights : SV_Barycentrics) : SV_Target

moving the SV_Barycentrics to the v2f struct

struct v2f
{ 
  float4 vertex : SV_POSITION;
  float2 uv : TEXCOORD0;
  float3 PerspectiveBaryWeights : SV_Barycentrics;
};

throws a shader error
Shader error in 'Unlit/NewUnlitShader': Semantic SV_Barycentrics is invalid for shader model: vs Use /Zi for source location. (on d3d11)

Is this expected? I'm on DX12 here right now.

[quote=“fherbst”, post:15, topic: 835265]
Is this expected? I’m on DX12 here right now.
[/quote]
It looks like DXC just does not like that – it’s trivially reproducible outside of Unity (see shader playground: http://shader-playground.timjones.io/1e89247ebc1bb12902aed1baa18b3256). You could file an improvement request for DXC folks to be able to do this I guess, https://github.com/microsoft/DirectXShaderCompiler/issues

1 Like

Ah, thanks. Guess my understanding was wrong then, I thought semantics could be moved between frag parameters and a parameter struct "by design".

Posted an issue here: https://github.com/microsoft/DirectXShaderCompiler/issues/3729
Also, reported a Unity crash, not sure if that's "URP is crashing" or DXC-specific though: (Case 1332972)

And here's DXC in Shader Graph :) See https://twitter.com/hybridherbst/status/1388147666106847237

7094788--845128--210430-164239051-NewShaderCompiler_-_SampleScene_-_Windows,_Mac,_Li.jpg

2 Likes

Although I don't think this is shader compiler related, @fherbst you should be able to wrap the corresponding line in a macro, eg:

struct v2f
{
  float4 vertex : SV_POSITION;
  float2 uv : TEXCOORD0;
#ifdef SHADER_STAGE_FRAGMENT
  float3 PerspectiveBaryWeights : SV_Barycentrics;
#endif
};

Similar macros exist for the other shader stages (Vertex, Domain, Hull, Compute) as detailed in the "Shader Stage Being Compiled" section of: https://docs.unity3d.com/Manual/SL-BuiltinMacros.html

The DX11 documentation has some more info on which semantics are supported as inputs/outputs for each shader stage: https://docs.microsoft.com/en-us/windows/win32/direct3dhlsl/dx-graphics-hlsl-semantics

1 Like

@Aras Are there any plans to export PBD for DXC compiled compute shaders shaders? In both NVIDIA Nsight and Microsoft's PIX it's not seemingly possible to step-by-step debug compute shaders from a Unity project with shaders compiled with DXC. Thanks for exposing the new functionality by the way! It's tremendously useful.

[quote=“AquaGeneral”, post:19, topic: 835265]
@Aras Are there any plans to export PBD for DXC compiled compute shaders shaders? In both NVIDIA Nsight and Microsoft’s PIX it’s not seemingly possible to step-by-step debug compute shaders from a Unity project with shaders compiled with DXC. Thanks for exposing the new functionality by the way! It’s tremendously useful.
[/quote]
Thank you for bringing this to our attention. I’m also guessing it doesn’t work with non-compute shaders as well? Looking at the code it seems like we might need to add extra compiler flags to embed PDB into the debug data but this is true for both compute and graphics shaders. We’ll investigate this shortly.

3 Likes