Unity Incremental C# Compiler - deprecated

Deprecation Note:

_____________________________________________________________

Unity Incremental Compiler

One solution we are working on to get us closer to our 500ms goal is a new compilation pipeline, which includes a new C# incremental compiler. Built using Microsoft’s Roslyn open source compiler project, our new C# compiler is designed to dramatically accelerate iterating on your C# code as well as giving you access to the latest and greatest C# 7.2 features.

Building our own compiler has several advantages. First, our compiler can cache all kinds of data from one compilation to the next, which means that code and references that didn’t change won’t get reparsed or reloaded (hence the name “incremental compiler”). Second, even though this compiler could be used to compile any dotNET code, it was written with a-priori knowledge about compiling Unity projects, which means we can do things (or skip things) other compilers cannot.

Some Preliminary Numbers

In the image above, you can see the effect the compiler cache has over the compilation time. Notice that the time to compile the first assembly is the same between the built-in Unity compiler and the incremental compiler (about 1.8 seconds). This is because being the first assembly we compile, no caching can occur. Subsequent compilations however benefit tremendously from caching.

As you can see, the most interesting performance gains are achieved on subsequent “incremental” compilations, when iterating back and forth between code and Unity (when it typically hurts the most). The image above shows the effect of changing one file in an assembly which is referenced by many others. The assembly in question (Unity.Mathematics) is rebuilt, then all assemblies dependant on it are quickly re-emitted.

That graph shows a couple of important things. First, it shows that the compiler is optimized for iterative changes where just a few files are changed at a time. Second, it shows that to squeeze the most performance out of the compiler, it is crucial to modularize your code using asmdef files. The compiler heavily relies on analyzing dependencies between assembly to recompile only the bare minimum. Chugging all your code in one big Assembly-CSharp.dll will only result in modest performance gains.

Some disclaimer on the numbers

The numbers you see are numbers we were able to measure in our lab and are subject to change quite a bit, as we add features and make additional changes. Also, the numbers you see are in milliseconds, but adding those numbers together does not equal the total time for recompilation and domain reloading (the time “felt” by the user). This is because even with the built-in compiler, some of these compilations occur in parallel. Furthermore, the new compiler does not impact the time it takes to do a “domain reload” (or enter-playmode).

Other Cool Stuff

Other cool stuff we get from writing our own compiler is support for C# 7.2, which gives you nifty ref returns, readonly and ref structs. We are also giving you the ability to write your own language rules and custom error messages. This is useful if you want to enforce coding conventions or generate compilation errors when, say, GetComponent<> is used in an Update method, or inside a for-loop, etc.

Can I Help?

If you’d like to give this new compiler a try, we are making the early alpha available today. See below for installation instructions. It is still early days for this technology and you might find problems - we’d love to get your feedback, good and bad :slight_smile:

Installation Instructions

The incremental compiler comes as a package you can include in your project via the Package Manager manifest.

1- Download and install Unity 2018.1 BETA 12

The compiler might work with earlier 2018 Beta versions of Unity, but you will not be able to debug your code without B12.

2- Add this to your manifest file

{
    "dependencies": {
        "com.unity.incrementalcompiler": "0.0.30"
    },
    "registry": "https://staging-packages.unity.com"
}

3- Set your scripting runtime to version 4.x in project’s player settings
The compiler client loaded by Unity needs this.

4- Delete your project’s Library/ScriptAssemblies folder

This will remove the last assemblies compiled built by the built-in compiler and will ensure the new compiler starts with a fresh, full recompilation.

5- Configure the Compiler from the Preference window

A new “Compiler” section is added to the Unity Preference window. You can control some compiler settings from there.

Feedback

And we’d love to hear what you think!
Please feel free to post your questions and comments here.

51 Likes

I’m sure many have been waiting for some Roslyn upgrades and especially the newest C# version. It’s great to finally have access to it. I installed this in a fresh project just now, and nothing exploded. No wonder, as I haven’t written any explosion scripts or made particle effects yet :wink:

I guess it’ll take at least a few hours of use before we really appreciate the changes. It also looks like there are some 2D experiments waiting in that new staging location - at least I didn’t notice them in the package manager before.

Edit: Had a crash on updating some packages. Could be compiler-related, could be the beta-ness of the editor. Sent a report which is probably still uploading.

1 Like

Did everything on the guide… however step 4… I don’t see a compiler option tab in the pref window…

package manager shows the incremental compiler is installed for the project… it does seem faster in compiling though… maybe :slight_smile:

annoyance… it has the “Hold On”… window showing when it recompiling, and that blocks you from hitting the editor play button… which I do all the time after making a change so that it automatically plays the build once it has recompiled… for now that time saving with Roslyn seems a little irrelevant when I have to wait for it finish just to hit the play button now, :slight_smile: because even after the Hold On window goes away… it still has some editor delay where it doesn’t have the icon compiling at the bottom status bar… yet you can’t click on anything in the editor for a brief period of time like a couple of seconds, like the play button… so that ain’t great right now.

Compiling does seem a little faster though, but hard to measure, I tried to revert back by removing the package, but don’t forget to close visual studio if you do revert and remove the ScriptAssemblies folder (I dunno why not just have “ScriptAssemblies-roslyn” to make it a little easier to switch compiler instead of it using the same folder as the previous compiler…as I had some roslyn process with a handle on the ScriptAssemblies folder when deleting it, that made Unity think the disk was out of space after restarting Unity, as I couldn’t create any files in that folder with VS from previous session still running with some roslyn process.

If you don’t see a new section after quitting Unity, deleting the ScriptAssemblies folder and restarting, then the incremental compiler is not active.

What OS are you using?

I see what is wrong… you forgot to mention that the player settings for the project need to have the scripting runtime set to .net 4.5 …not Legacy

The compiler tab shows up now…my other issues still apply though :slight_smile:

2 Likes

You need to change scripting runtime to version 4.x in player settings

my bad, I’ll add this to the original post. thanks!

Installed and got the Compiler pref menu.
But compilation fails. Getting this:

UNetWeaver error: Exception :System.IO.IOException: Sharing violation on path Temp\Vendor.pdb
  at System.IO.File.Delete (System.String path) [0x00073] in <e1a80661d61443feb3dbdaac88eeb776>:0
  at Unity.UNetWeaver.Weaver.Weave (System.String assName, System.Collections.Generic.IEnumerable`1[T] dependencies, Mono.Cecil.IAssemblyResolver assemblyResolver, System.String unityEngineDLLPath, System.String unityUNetDLLPath, System.String outputDir) [0x002cf] in C:\buildslave\unity\build\Extensions\Networking\Weaver\UNetWeaver.cs:1862
  at Unity.UNetWeaver.Weaver.WeaveAssemblies (System.Collections.Generic.IEnumerable`1[T] assemblies, System.Collections.Generic.IEnumerable`1[T] dependencies, Mono.Cecil.IAssemblyResolver assemblyResolver, System.String outputDir, System.String unityEngineDLLPath, System.String unityUNetDLLPath) [0x0004c] in C:\buildslave\unity\build\Extensions\Networking\Weaver\UNetWeaver.cs:1888
UnityEngine.Debug:LogError(Object)
Unity.UNetWeaver.Log:Error(String) (at C:/buildslave/unity/build/Extensions/Networking/Weaver/Program.cs:20)
Unity.UNetWeaver.Weaver:WeaveAssemblies(IEnumerable`1, IEnumerable`1, IAssemblyResolver, String, String, String) (at C:/buildslave/unity/build/Extensions/Networking/Weaver/UNetWeaver.cs:1896)
Unity.UNetWeaver.Program:Process(String, String, String, String[], String[], IAssemblyResolver, Action`1, Action`1) (at C:/buildslave/unity/build/Extensions/Networking/Weaver/Program.cs:34)
UnityEditor.Scripting.ScriptCompilation.EditorCompilationInterface:PollCompilation()

Already deleted everything and did a reimport. Always fails the same way.
I can see that Unity does indeed have a handle on the pdb.

It seems that it will hold onto the first pdb it generates. I changed things a bit and it was another one that was held.

Any ideas?

Edit: Using b12 on Windows 10 x64 (forgot to include info).
I removed all assemblies from my project and left only one, to test.
Fails the same way.

Edit2: After reimporting the asmdef file a few times, it failed differently:

UNetWeaver error: Exception :System.InvalidOperationException: Operation is not valid due to the current state of the object.
  at Mono.Cecil.ModuleDefinition.ReadSymbols (Mono.Cecil.Cil.ISymbolReader reader) [0x0002f] in <28cdca1704d2491781795499c297b78b>:0
  at Mono.Cecil.ModuleReader.ReadSymbols (Mono.Cecil.ModuleDefinition module, Mono.Cecil.ReaderParameters parameters) [0x0004a] in <28cdca1704d2491781795499c297b78b>:0
  at Mono.Cecil.ModuleReader.CreateModule (Mono.Cecil.PE.Image image, Mono.Cecil.ReaderParameters parameters) [0x00081] in <28cdca1704d2491781795499c297b78b>:0
  at Mono.Cecil.ModuleDefinition.ReadModule (Mono.Disposable`1[T] stream, System.String fileName, Mono.Cecil.ReaderParameters parameters) [0x0000d] in <28cdca1704d2491781795499c297b78b>:0
  at Mono.Cecil.ModuleDefinition.ReadModule (System.String fileName, Mono.Cecil.ReaderParameters parameters) [0x0006c] in <28cdca1704d2491781795499c297b78b>:0
  at Mono.Cecil.AssemblyDefinition.ReadAssembly (System.String fileName, Mono.Cecil.ReaderParameters parameters) [0x00000] in <28cdca1704d2491781795499c297b78b>:0
  at Unity.UNetWeaver.Weaver.Weave (System.String assName, System.Collections.Generic.IEnumerable`1[T] dependencies, Mono.Cecil.IAssemblyResolver assemblyResolver, System.String unityEngineDLLPath, System.String unityUNetDLLPath, System.String outputDir) [0x0000f] in C:\buildslave\unity\build\Extensions\Networking\Weaver\UNetWeaver.cs:1763
  at Unity.UNetWeaver.Weaver.WeaveAssemblies (System.Collections.Generic.IEnumerable`1[T] assemblies, System.Collections.Generic.IEnumerable`1[T] dependencies, Mono.Cecil.IAssemblyResolver assemblyResolver, System.String outputDir, System.String unityEngineDLLPath, System.String unityUNetDLLPath) [0x0004c] in C:\buildslave\unity\build\Extensions\Networking\Weaver\UNetWeaver.cs:1888
UnityEngine.Debug:LogError(Object)
Unity.UNetWeaver.Log:Error(String) (at C:/buildslave/unity/build/Extensions/Networking/Weaver/Program.cs:20)
Unity.UNetWeaver.Weaver:WeaveAssemblies(IEnumerable`1, IEnumerable`1, IAssemblyResolver, String, String, String) (at C:/buildslave/unity/build/Extensions/Networking/Weaver/UNetWeaver.cs:1896)
Unity.UNetWeaver.Program:Process(String, String, String, String[], String[], IAssemblyResolver, Action`1, Action`1) (at C:/buildslave/unity/build/Extensions/Networking/Weaver/Program.cs:34)
UnityEditor.Scripting.ScriptCompilation.EditorCompilationInterface:TickCompilationPipeline(EditorScriptCompilationOptions, BuildTargetGroup, BuildTarget)

But on the compilation console, it’s the same

Internal compiler error: System.IO.IOException: Sharing violation on path C:\Users\Fausto\AppData\Local\Temp\qm8f2cig.h4e\Vendor.pdb" or "V:/LOSTALLOY/Aftertile/Unity3D/Temp/Vendor.pdb
  at System.IO.File.Copy (System.String sourceFileName, System.String destFileName, System.Boolean overwrite) [0x00192] in <e1a80661d61443feb3dbdaac88eeb776>:0
  at UnityEditor.Compilation.IncrementalCompiler.Compilation+<Run>d__22.MoveNext () [0x00399] in <b677897e5ef74c96b18b149e35f2365a>:0
Vendor.dll 7871ms  re-parsed: source(2271) metadata(0) concurrent compilations(1)

Wow, this is great. Happy birthday to all of us I guess.

3 Likes

ok yeah this is indeed a side-effect of using the incremental compiler, since it generates portable pdbs and causes Unity to try to load those.

I’ll investigate and reply back.
In the mean time I think the only solution is to disable the UNet module.

I considered the .NET 4.x requirement obvious (if there’s talk about C# 7.x, you can bet it there’s the latest .NET involved). I know there have been issues with .NET 4.x on the Windows side of Unity, but on macOS I have been using it without any issues in Unity 2017.x. Betas with experimental packages on top is a whole different matter though!

Is it possible to disable it? com.unity.unet?
Edit: com.unit.unet doesn’t exist. Not sure how to disable it to test.

Works good in my ECS demo, big thanks!
Does the current Entities 0.0.11 version already utilize ref returns? I don’t think I saw any in code. How easy would it be to add for me? Just with “return ref data;” or is there more to it?

With ref returns we can return addreses to structs and modify them directly in systems/jobs without writing back a new struct or make an update call. Is this correct?

These are great results, innovative idea!

If you write your own compiler, how likely is it, that you put yourself in a position where you’re stuck with this compiler and can’t benefit from upgrades of the current compiler technology? It would be a bummer, if this new tech causes a freeze where you can’t upgrade the language/compiler to the latest version, like we experienced in the past with your Mono fork.

Is this a long time feature, that is going to be supported by a dedicated team “forever”? New tech in Unity is often great on the surface, but if you dig deeper, you find issues and it occurred sometimes in the past, that UT don’t seem too interested to “finish the work” to iron out all the problems, which would be a tremendous issue regarding compiler tech.

4 Likes

This does help greatly with the compilation of scripts, I bet. However, it does not help with the reserialization of assets after the compilation, which in my tests can take up to 70% of the total “script compilation time”…

This of course varies from project to project, but at least for us the actual C# compilation time is not the problem.

This is just a modified Roslyn.
If all fails, you can just drop alexzzzz’s compiler replacement and and at worst get poorer compilation times.
Roslyn is a breeze to integrate and it’s just a few lines of code to get a fully working compiler wrapper.
As long as Unity isn’t modifying the generated code (or at least not depending on modifications therein), we should be safe.

1 Like

I think the same. The biggest criminal is the actual assembly reload time, and a large chunk of that time will be spent on serialization/deserialization on projects with lots of serialized stuff.

But this compiler really is a great first step and will favor splitting the code into multiple small assemblies that have no dependencies (so you get parallel compilation benefits).

Does anyone else have the issue in Visual Studio 2017 and the new compiler that reference dlls and namespaces like Unity.Collections are not loaded in the project?

Seems like 2018 will be the year where unity finally gets it’s act together. Looking forward to more

I think I encountered the same with VS 2017. Because VS is acting weird with 2018 beta in general, I didn’t give it too much thought.

Switching to vscode (which is using omnisharp) fixed the loading for me.