Reverse engineering UnityShaderCompiler. Code inside

Unity Engine
1

I’m not sure if this is the right place to post this, but I’d like to share with you a little something I’ve been working on.

GitHub - hamish-milne/UnityShaderCompiler: Interface and experiments with Unity's shader compiler

What you can find here is a .net interface to UnityShaderCompiler.exe, the new ‘service process’ introduced in 4.5 to replace the old CGBatch.exe, which, while much slower, did have the ability to be used outside of the Unity editor. This interface allows the same functionality in newer versions, and the output, like its predecessor, can be used (I believe!) in the Material constructor to load the shader directly into the game at runtime.

Why exactly would you need something like this?
Since it uses .net, you could use it from within the editor, at runtime, or outside it altogether. You could, in theory, do things like:

  • Creating and compiling shaders from your own code, rather than having to manually copy the files over
  • Creating shaders programatically at runtime
  • Direct access to all the input and output from the compiler, errors, stats etc.
  • Manual optimizations of the compiled code
  • Compile externally from another program, such as an IDE (this was what motivated me personally)

It’s niche, I’ll admit, and though I’ve found a few posts talking about offline compilation pre and post 4.5, I’m not sure if anyone outside of myself will find this useful. But hey, I’m a programmer, half of this was just for the fun of it :wink:

How did I manage it?
It was certainly tricky. There is zero documentation on the internal communication between Unity and the compiler. Finding the method wasn’t too tricky; the //./pipe/ prefix in the command (which I sniffed with procmon) was a dead giveaway: named pipes. It was easy enough to create a little interceptor tool in C#, which, with typical hacky elegance, we simply replace the original compiler executable with. The interceptor starts up its own named pipe server, connects to the real compiler and records all transactions to a log file.

Working out the message length was a bit of a pain. Essentially, every command sent either way ends in a newline, which is fine, but there’s no indication of how many lines to expect. So in fact, I had to reverse-engineer each command somewhat before I could know how many lines to read and write at each end. That code is in UnityShaderCompilerIntercept, and I’ve basically finished with it now.

Once I had a transcript, it was easy enough to work intuitively from there after compiling a variety of shaders on different platforms. Though, since I really have very little idea about how shaders work, some things still remain a mystery.

One example is ‘getPlatforms’. That command is run as soon as Unity starts up, and it returns 13 numbers. But the numbers it outputs seem to be the same on every system I’ve tried. I even made a tool to spoof the output (GetPlatformsTest) to see if changing them had any effect on Unity at all, but they did not. So I’m stumped. If anyone wants to help me out, compile and run the main project on your system and, if the output is different from “-3085, 0, 0, 1, 0, 2, 8, 1, 1, 2, 10, 0, 0” I’d be very interested to hear!

There are other ID numbers, values, semantics etc., particularly in the ‘bindings’ found before DirectX assembly, that I don’t really understand fully, and they’re all marked in the code.

Where to go from here?
Make a single function that takes some source code, and outputs the full compiled shader code for every platform. This shouldn’t take too long, as soon as I can properly parse the parameter that is used to determine what platforms are compatible.

Following that, I could see about making a full ‘spoofer’ so I can adjust some of the compiler’s output and see how Unity responds.

If you’re still reading, congrats! You’ve suffered through many paragraphs of rambling. If you know more than me about shaders (which you probably do) and want to help out, I could really use some assistance in deciphering various numbers.

Thanks for reading! I’ll stick around here for a while if anyone wants to know anything.

EDIT: Aha! Turns out the platform parameter is just a bitwise OR of all the available platforms, with the bits corresponding to the ID numbers I found earlier.

2 Likes
2

Hello hammil. I’m very interested in your work here. My goal with my project is to allow modders to submit customer shaders for use in my game, which requires the ability to compile shader code.

From your post here, it seems like you’re working on something I can quite directly use.
Do you have any updates on your work?
Also, do I have permission to use it in my project?

3

Sure thing - I’ll probably slap an LGPL license on it. It would be nice if you credited me as well! (Hamish Milne, hammil@theovermare.com)

Everything I’ve done is on the github. I’ve been working on other things since, but since there’s some interest I’ll be sure to finish everything off and clean it all up, including a function to generate the full output from a full input. I’m also open to adding other features on request!

4

That’s great. You’ll of course be credited :). I tried building your project but it look as though you were in the middle of some changes. It doesn’t build at the moment so I can’t test it yet.

All I really need to be able to do is read some shader code from a file, and be able to assign that shader to materials in my game. So obviously, the compilation step is missing, hence my interest.

I was previously trying to use CgBatch. Feel like I’m close, but then I found your work with 4.5+ so thought might be worth a look :). I’m more than happy to help if I’m able, perhaps with testing, but I certainly don’t have you insight into the internal workings!

I’m often in the unity IRC chat for direct discussions :).

5

Hi hammil. I’m thinking about my game’s design in relation to users being able to use custom shaders registered at runtime.

What is the expected output of the compilation. Can I end up with a Shader (the class). In otherwords, would I be able to make a Dictionary<string, Shader> populated with shaders compiled at runtime, and then use the registered shaders as required to assign to Materials etc? Or would the output be something like a string that should be used in the Material constructor? (I think I read something about this somewhere).

Thanks.

6

It would be the latter. The compile function would return a string that you could pass directly to the Material constructor. If you want to get the shader object from that (e.g. to assign to another material) you can use material.shader

7

Makes sense. Can easily work with that. :slight_smile: Thanks for the clarification.

1 Like
8

Wow, impressive reverse engineering!

You’ll have to keep in mind that the “protocol” between editor & unity shader compiler can and will change between Unity versions. E.g. in 5.0 it’s already different from 4.5/4.6.

9

That’s good to know, thanks @Aras .
Will I be able to take the 4.6 UnityShaderCompiler.exe for use in my game at runtime anyway?
In other words, will I run into trouble using 4.6 compiled shaders on the 5.0 runtime?

10

I wonder if the shader compilation process can be made available to users for runtime compilation purposes in unity 5.0…? Would really help for my modding goals! Should I keep dreaming?

11

Thanks! =) I’ll be sure to keep that in mind. Hopefully they’re similar enough that I can add the changes to the existing code and auto-detect the version.

At a rough guess, I’d say you probably could, since the output is really just the ASM/GLSL/etc. given directly to whatever driver/runtime you’re using. So unless they change the wrapper code or drastically change how shaders are dealt with at runtime (which would involve breaking backwards compatibility re. the Material constructor) I’ll speculate that it’ll be fine. Though I’d wait to confirm that before investing fully of course!

12

How this is even LEGAL I have no idea! impressive work! :slight_smile:

13

Legal status of reverse engineering depends on where you live, and for what reason you are doingit… it’s complicated :slight_smile:

14

Yea… :smile: Don’t think there’s a need to go into that topic here! :wink: (Would probably end with a mile-long post on how many times Microsoft locked their software in… :p)

15

I’ll take that as a compliment, then :stuck_out_tongue:

16

Unfortunately the Material(string) ctor is no longer there. I am checking UnityDecompiled, and now adding these will require a LOT of reverse engineering. I will most likely rewrtite my GPU Noise library in Native DLL / DX/GL contest with generated shaders. I am “checking” UnityGraphics.h if I can manually inject custom shaders in Unity pipeline.