2019.3 slow build times

Unity Engine
,
1

In editor script recompiling is not terrible for me but compiling out a binary is getting horrific.

I make one line of code change and want to test on Windows standalone and/or Android and/or Linux? We are talking about 5+ minute compile times and the unity editor in complete frozen with no helpful status/progress indicators. Even if I enable the script only debug build, it shows minimal if any improvement.

In my project, not everything can be tested in editor. You need to run multiple standalone and/or on devices. Besides performance problems in editor, you need multiple instances for proper development.

What can I do to profile this issue better and address them? If I search around for improving build speeds, I see stuff like the assembly definitions but I’m not convinced the script compiling is the problem.

2

Depending on the size of your project, it’s easy for build times to be several minutes. Fast iteration on builds really isn’t viable except for the most trivial projects.

However, you should look at the Editor log after performing a build. There should be some timing metrics in it. Typically, one huge expense in creating a build is compiling shader variants. It’s possible that Unity is including more shader variants than you strictly need, and perhaps you could cut down on the number of variants to speed up your build.

Another idea is, if you’re iterating on testing something, you could create a build that just builds one scene, or a small number of necessary scenes, so you can quickly build just what you need to run the thing you’re trying to test. I have a few build scripts for building smaller portions of my project, which usually just means passing a different list of scenes into the build method.

Also note how much stuff you’ve placed into the Resources folder(s). Anything there will always be included in a build, even if it’s not used by the scenes you’re building. If you have a lot of stuff in Resources, that will increase the size of your build, potentially for assets that aren’t even used by that build.

Using Assembly Definitions probably won’t make much of a difference for builds. That’s mainly for quickly iterating within the editor when making code changes.

3

Is this with or without using the “Scripts Only Build” option?

Switch from IL2CPP to Mono on platforms where it’s supported. The turnaround times with Mono are significantly faster.

From the documentation:
Enable this setting to only rebuild the scripts in your application, and reuse data files from the build you previously executed. Before you can use this setting, you must build your whole Project once. The Scripts Only Build option significantly improves iteration times if you only changed the code in your application. This setting is only available if you enable Development Build.

6079602--659676--upload_2020-7-11_19-18-12.png

https://docs.unity3d.com/Manual/BuildSettings.html

1 Like
4

I’m not sure how often I make only code changes between builds (as opposed to changing something within a scene), but I never knew about this feature. Definitely a nice trick I’ll keep in mind.

1 Like
5

I use Script Only Build option and Mono. The former makes no difference in speed unfortunately. I’ve timed it and at best it saves 2 seconds, maybe just placebo. Not sure why.

Alot of time is spent on doing stuff like cubemaps for some reason. It’s impossible to tell since the editor is basically broken while compiling, can’t be touched and does nothing but flicker text boxes.

I wish we could profile this stuff better. Using the Editor Profiler, it just stops until you finish or miraculously hit the flickering “cancel” button on something that actually stops the build.
At the end of a build, for whatever absurd reason, it’s doing stuff like “AssemblyHelper.ExtractAssemblyTypeInfo()” which allocates over 4GB in a frame. Pumps the frame to over 400,000ms locking the editor.

6

Does the log file tell you anything about the timing? It should at least contain statements about how long it took to compile various shaders. I can’t recall whether the log file contains any other timing information that might shed some light.

7

Direct comparion
Options: StripDebugSymbols, CompressTextures, ForceOptimizeScriptCompilation, Il2CPP, Development, AllowDebugging, CompressWithLz4

The first 4 options seem to be required by unity at all times.

WITHOUT BuildScriptsOnly:
– 1/13 Build player 39s –
– 2/2 Compile scripts 54s –
– 3/13 Prepare assets for target platform 21s –
– 4/13 Building scenes 17s –
– 5/13 Building scene Assets/Scenes/bootstrapper.unity 17s –
– 6/13 burst 0s –
– 7/13 Build GlobalGameManagers file 0s –
– 8/13 Writing asset files 17s –
– 9/13 Packaging assets - globalgamemanagers.assets 0s –
– 10/13 Packaging assets - resources.assets 2s –
– 11/13 Packaging assets - sharedassets0.assets 14s –
– 12/13 Creating compressed player package 1s –
– 13/13 Postprocess built player 29s –

WITH BuildScriptsOnly:

– 1/5 Build player 53s –
– 2/5 Compile scripts 48s –
– 3/5 Prepare assets for target platform 21s –
– 4/5 burst 0s –
– 5/5 Postprocess built player 25s –

Build player, assets, and post processor take forever.
And yes, this is mono.

Compile scripts at 48s isn’t too terrible. But these two are offenders.

  • Finished compile Library/PlayerScriptAssemblies/Unity.Serialization.dll in 8.663926 seconds
  • Finished ILPostProcessor ‘Unity.Entities.CodeGen.EntitiesILPostProcessors’ on Assembly-CSharp.dll in 30.005564 seconds
  • Starting ILPostProcessor ‘Unity.Properties.CodeGen.PostProcessor’ on Assembly-CSharp.dll

If I disable CompressWithLz4, it’s an even longer build time so it’s not that either.

8

I shaved off 40 seconds by going into Wwise and finding their “AkBuildPreprocessor” script. In the OnPreprocessBuildInternal and OnPostprocessBuildInternal, if I skip the stuff they do entirely by returning early my results are:

1/5 Build player 19s –
2/5 Compile scripts 48s –
3/5 Prepare assets for target platform 17s –
4/5 burst 0s –
5/5 Postprocess built player 8s –

Of course, this stinks for audio programmers and designers, but this is something we can tinker with hopefully.

Still a bummer about those compile times. I dont know what to do about “EntitiesILPostProcessors”
There’s also at least a minute of time not accounted for in those steps 1 to 5. Just not sure where.

9

Those compile times seem really high to me. I just tried doing a build, just for comparison. Hard to say how much code I have, but the whole project is in the 10-15 GB range, with plenty of custom code. I was getting 10 seconds or so for the script compilation, however, I was getting it twice. I don’t know why that is. But still, that’s a lot less than what you’re seeing.

Some simple things to make sure you’re doing:

  • Build to an SSD if possible? That is, the directory where the build ends up should ideally be on an SSD.
  • Make sure your anti-virus isn’t scanning that folder. I don’t know how much of a difference that makes.
1 Like
10

Thanks for sharing your times.
SSD and antivirus has the whole drive excluded. It’s also a beefy machine, 64GB with 12 threads.

I’m not liking this codegen thing for Entities. Wherever it shows up is a stall during the compile.

11

I’ve attached what my build output has in case anyone has any ideas by the script compile part takes so long. I know that the idea is you can use asmdefs to break down Assembly-CSharp, but you can see here there are other things like Unity.Entities and Unity.Mathematics.Extensions.Hybrid taking longer…

EDIT:
Found this cool tool that kind of organizes it a bit rather than looking at that dump of text.

So the top 3 libs - math, coffee, asmplugins don’t need to be recompiled everytime. Any way to cache them?

Unity.Mathematics.dll
6,904.000 ms

Coffee.UIParticle.dll
6,600.000 ms

ASMPlugins.dll
6,554.000 ms

Assembly-CSharp.dll
6,066.000 ms

Unity.TextMeshPro.dll
4,047.000 ms

Unity.Timeline.dll
4,001.000 ms

Unity.Entities.dll
3,966.000 ms

Unity.InputSystem.dll
3,905.000 ms

Unity.ResourceManager.dll
3,804.000 ms

Unity.Burst.dll
3,703.000 ms

6083562–660270–buildscripts.txt (34.8 KB)

1 Like
12

There must be a way to avoid recompiling stuff that never changes. Like Unity.Mathematics or this Cofee.UIParticle.dll I have. Code in them never changes. I understand having to recompile if I do a rebuild or switch targets, but doing a scripts only build ontop of a full build should be smart. There must be something I can do, I just cant find the right search terms I guess.

ASMPlugins is in Assets/Plugins/Wwise folder

Unity.Mathematics ends up in the Library\PackageCache folder

13

Well, what I wasn’t sure about your log was whether each of those compile time values truly the elapsed time compiling just that one DLL, or basically the “time” at which it finished compiling that DLL, even if it started just a few ms before it finished. It seemed like a bunch of the compile times were very similar, and roughly increased in value, making me think it was more like just the time it completed the compilation. But I don’t know.

1 Like
14

Yeah it’s hard to tell. I also tried moving things like Coffee into the plugins folder which is not supposed to recompile Everytime but definitely does.

15

Bump

16

Bump

17

Bump

18

I’m trying to figure out how to improve this to. I want to be able to do quick iteration of script only builds. Right now it takes me 20 seconds, with most of that recompiling assembly definitions that didn’t change.

19

I unfortunately gave up and left Unity. Their libraries like Unity Mathematics and other packages were killing all of my productivity. I am now much happier in C++ land.

2 Likes
20

I believe Unity fixed this in 2021.1 with Incremental Script Compilation:
https://discussions.unity.com/t/820499

If this isn’t going to speed up your compile times, maybe reply in that thread.