IL2CPP Build Time Improvements - Seeking Feedback

Hello Alpha Users!

Multiple teams in Unity have been working on improving iteration time of player builds the past few releases. You should have noticed faster IL2CPP builds during the 2020 release cycle and even more improvements are coming in 2021.

To this end, we have been working on a new option for IL2CPP that generates much less code (up to 50% less). This allows for both faster iteration times and smaller executable files. There may be a small runtime performance impact due to the different code generation.

This graph shows improvements to compile times with this option enabled:

We are asking that you try out this new option with your IL2CPP builds. We want to know if you see real world performance issues when running with this option. Also, let us know if the builds are faster and smaller.

Note: This is available in alpha builds only, from 2021.2.0a17 onward.

How to Enable: Change the "IL2CPP Code Generation" setting drop down in the Build Player window to "Faster (smaller) builds". The default and previous behavior is the "Faster runtime" setting.
6873116--857248--upload_2021-5-20_12-37-26.png

Please respond with:

  • Any difference in FPS or startup your applications see when running with this option enabled.
  • The profile.json file in the generated IL2CPP C++ output folder (next to build output folder) for builds with and without this option. You can also send for cold and warm (incremental) builds. This allows us to see where build time is being spent and saved.
  • You can also send stats on generated C++ output size and final binary size.

Respond to this thread with any questions or findings. Additionally, you can DM me if you have information you don’t want to share in public.

Thanks!

17 Likes

Sounds interesting!
I'm wondering a bit why you share detailed graphs about the build times but not about the possible performance regression. We're using a lot of generics so it would certainly be interesting which microbenchmarks show what slowdowns, and also how the above examples do in terms of performance. Mind sharing that?

4 Likes

Hello,

I shared the build times as the benefits of this option are clear. It is unclear what the negative impacts are for real projects. When running test projects we didn't see negative performance impact beyond ~10% of script time in a frame. That's why we are asking for more feedback, to have a larger sample size for finding regressions.

Here is an example microbenchmark:

    public class EqualityComparerEnum
    {
        private MyEnum _enum;
        private EqualityComparer<MyEnum> _comparer;

        public EqualityComparerEnum()
        {
            var rand = new Random(12345);
            _enum = (MyEnum)rand.Next(0, 3);

            _comparer = EqualityComparer<MyEnum>.Default;
        }

        [Benchmark]
        public void CallToEnumEqualityComparerGetHashCode()
        {
            _comparer.GetHashCode(_enum);
        }

        enum MyEnum
        {
            A,
            B,
            C,
            D
        }
    }

This (the call to GetHashCode) runs an order of magnitude slower (~12x) with full generic sharing enabled.

Unfortunately, I don't have any projects to really test the changes on but I just want to add my two cents.
Personally, I'd rather have longer build times for more performance. The Release setting could be close to final and faster with these new changes while Master could take longer but maximizes performance.

3 Likes

[quote=“Hertzole”, post:4, topic: 830253]
Unfortunately, I don’t have any projects to really test the changes on but I just want to add my two cents.
Personally, I’d rather have longer build times for more performance. The Release setting could be close to final and faster with these new changes while Master could take longer but maximizes performance.
[/quote]
Presumably this will be a configurable setting. Especially because there is bound to be corner-case code where sharing generics proves to be very slow (like the micro-benchmark mentioned above).

Glad to see this! I would try this out if we were on the Alpha. Looking forward to seeing what other people observe.

[quote=“Kleptine”, post:5, topic: 830253]
Presumably this will be a configurable setting.
[/quote]

This is partly why we are trying to gather feedback. In order to know if and how it should be configurable.

5 Likes

I observed a shorter build time when trying this out, but I immediately get nullreferenceexception when I run the build (happening when loading an addresable scene). Building with mono works though.

So I have to say that I'd much rather it works than it being fast hehe :)

Unity 2019.4.21f1
Clean - 
Build completed with a result of 'Succeeded' in 1431 seconds (1431279 ms)
Build completed with a result of 'Succeeded' in 468 seconds (467873 ms)
Build completed with a result of 'Succeeded' in 462 seconds (462387 ms)

Unity 2020.2.6f1
Clean - Build completed with a result of 'Succeeded' in 1052 seconds (1051999 ms)
Build completed with a result of 'Succeeded' in 136 seconds (135982 ms)
Build completed with a result of 'Succeeded' in 135 seconds (134927 ms)

Unity 2021.2.0a6.1011
Clean - Build completed with a result of 'Succeeded' in 681 seconds (681159 ms)
Build completed with a result of 'Succeeded' in 102 seconds (102462 ms)
Build completed with a result of 'Succeeded' in 105 seconds (105499 ms)

6880268--803657--il2cpp.zip (1.57 MB)

1 Like

I gotta say I am impressed by the speed upgrade when going from 2019.4 LTS to 2020.2. Cant wait for 2020 LTS version, its going to be very good!

1 Like

[quote=“Skjalg”, post:7, topic: 830253]
I observed a shorter build time when trying this out, but I immediately get nullreferenceexception when I run the build (happening when loading an addresable scene).
[/quote]

Do you get a NullReferenceException even without the option enabled? Or only when the option for full generic sharing is enabled?

I've just ran some tests and got surprising results, in that building is SLOWER than before!

(Note: All builds are development builds, with ILC2PP configuration "master")

Build Time
[quote]
A. Unity 2020.2.6f1
C:\Program Files\Unity\Hub\Editor\2020.2.6f1\Editor\Data\il2cpp/build/BeeSettings/offline/bee.exe exited after 280347 ms.
Build completed with a result of 'Succeeded' in 327 seconds (327026 ms)

B. Unity 2021.1.0a8
C:\unity-alpha\2021.2.0a8\Editor\Data\il2cpp/build/deploy/netcoreapp3.1/il2cpp.exe exited after 526753 ms.
Build completed with a result of 'Succeeded' in 565 seconds (564805 ms)

C. Unity 2021.1.0a8 with ENV
C:\unity-alpha\2021.2.0a8\Editor\Data\il2cpp/build/deploy/netcoreapp3.1/il2cpp.exe exited after 337287 ms.
Build completed with a result of 'Succeeded' in 374 seconds (374124 ms)
[/quote]

Build Size
[quote]
A. Unity 2020.2.6f1
ALL:
* 1.25 GB (1.362.374.993 bytes)
* 472 files, 18 folders

EXCL: IL2CPP output:
* 955 MB (1.001.884.001 bytes)
* 48 files, 14 folders

B. Unity 2021.1.0a8
ALL:
* 1.26 GB (1.364.152.963 bytes)
* 430 files, 17 folders

EXCL: IL2CPP output:
* 989 MB (1.037.721.313 bytes)
* 50 files, 14 folders

C. Unity 2021.1.0a8 with ENV
ALL:
* 1.13 GB (1.217.775.321 bytes)
* 386 files, 17 folders

EXCL: IL2CPP output:
* 933 MB (978.953.425 bytes)
* 50 files, 14 folders
[/quote]

I was quite careful in testing, making sure to close unity between each build (& after upgrading before making B.) so this is quite weird. I've also made some profiler snapshots for each build, I'll send you a DM with those & some extra details (like the profile.json) later today.

2 Likes

[quote=“TimHeijden”, post:10, topic: 830253]
I’ve just ran some tests and got surprising results, in that building is SLOWER than before!
[/quote]

Thanks for your testing of this @TimHeijden ! I took a look at your build profiler output and have a few thoughts.

  1. I see you are building in master/“ReleasePlus” build config. This enables LTO (link time optimization). While this may help performance, it’s not something you want to iterate with as it is extremely expensive. It looks like 95% of your entire build time is the link event. I’d suggest Release for your daily work and it’s something we should note more clearly to improve iteration.
  2. I see you are using the VS 2017 toolchain. We have found the VS 2019 linker much faster. It may be something else to keep in mind when you can upgrade.
  3. It seems like the “C” case didn’t pickup the environment variable. You will want to make sure it is set, and kill/restart Unity and Unity Hub.

If you have time and are interested in providing more data, I would suggest:

  1. Just doing runs for “B” and “C” cases. This is the most interesting data points for this new option.
  2. Use Release rather than ReleasePlus/master config.
  3. Again, for “C” case make sure environment variable is set and all instances of Unity/UnityHub are killed to pick up new value
  4. Do both clean (which is what your data seems to come from) and incremental builds. Tweak one line in a script and build again, as all things should be faster but we are mainly focused on this incremental use case.

Thanks again for providing feedback!

1 Like

[quote=“joncham”, post:11, topic: 830253]
Thanks for your testing of this @TimHeijden ! I took a look at your build profiler output and have a few thoughts.

  1. I see you are building in master/“ReleasePlus” build config. This enables LTO (link time optimization). While this may help performance, it’s not something you want to iterate with as it is extremely expensive. It looks like 95% of your entire build time is the link event. I’d suggest Release for your daily work and it’s something we should note more clearly to improve iteration.
  2. I see you are using the VS 2017 toolchain. We have found the VS 2019 linker much faster. It may be something else to keep in mind when you can upgrade.
  3. It seems like the “C” case didn’t pickup the environment variable. You will want to make sure it is set, and kill/restart Unity and Unity Hub.

If you have time and are interested in providing more data, I would suggest:

  1. Just doing runs for “B” and “C” cases. This is the most interesting data points for this new option.
  2. Use Release rather than ReleasePlus/master config.
  3. Again, for “C” case make sure environment variable is set and all instances of Unity/UnityHub are killed to pick up new value
  4. Do both clean (which is what your data seems to come from) and incremental builds. Tweak one line in a script and build again, as all things should be faster but we are mainly focused on this incremental use case.

Thanks again for providing feedback!
[/quote]
Hey Jon, thanks for clarifying the main use case you’re looking for. I think I’ll do another test later today and otherwise next week. I had killed unity & the hub, so its strange that “C” didn’t pickup the environment variable… will check more closely. (I had assumed it worked because the build time was so much shorter again)

As for building “ReleasePlus”, I was already fully aware that its not really intended for iteration, but I was also very curious about the possible performance drop and thus thought I’d use the “optimum” setting for that. Note that I’m also very much of the opinion that unity devs should be able to choose for the maximum runtime performance, especially when developing for slower platforms than PC and trying to hit a stable FPS, 10% extra performance cost for scripting is quite a lot!

In addition, I am indeed using VS2017. This is because some platforms require use of 2017 rather than 2019. Good to know that 2019 is faster though, will keep that in mind!

Couldn't help myself, did it right away. This time I used the IL2CPP configuration "release" instead, and also did cold/warm tests. All builds are using Unity 2021.1.0a8

I hope this time the environment variable stuck properly. Stats may be slightly skewed because I restarted my PC between no-env & env builds.

Build Time
[quote]
NO-ENV - Cold
C:\unity-alpha\2021.2.0a8\Editor\Data\il2cpp/build/deploy/netcoreapp3.1/il2cpp.exe exited after 101323 ms.
Build completed with a result of 'Succeeded' in 149 seconds (148616 ms)

NO-ENV - Warm
C:\unity-alpha\2021.2.0a8\Editor\Data\il2cpp/build/deploy/netcoreapp3.1/il2cpp.exe exited after 19697 ms.
Build completed with a result of 'Succeeded' in 66 seconds (66143 ms)

ENV - Cold
C:\unity-alpha\2021.2.0a8\Editor\Data\il2cpp/build/deploy/netcoreapp3.1/il2cpp.exe exited after 81064 ms.
Build completed with a result of 'Succeeded' in 136 seconds (135879 ms)

ENV - Warm
C:\unity-alpha\2021.2.0a8\Editor\Data\il2cpp/build/deploy/netcoreapp3.1/il2cpp.exe exited after 14553 ms.
Build completed with a result of 'Succeeded' in 53 seconds (53383 ms)
[/quote]

I again sent a DM with the other details inside.

2 Likes

When trying to Il2CPP build an empty project with 2021.2.0.0a9.1376 i get this error during build time:

Exception: Unity.IL2CPP.Building.BuilderFailedException: Build failed with 190 successful nodes and 1 failed ones
Annotation: C_Win64_VS2019Pch y2vq/pch-cpp-3692469889848082439.pch (+obj)
...
Files (x86)\Windows Kits\10\Include\10.0.18362.0\ucrt" /I"C:\Program Files (x86)\Windows Kits\10\Include\10.0.18362.0\cppwinrt" /Fo"y2vq\pch-cpp-3692469889848082439.obj" /D_HAS_AUTO_PTR_ETC=1 /D_SILENCE_ALL_CXX17_DEPRECATION_WARNINGS /D_SILENCE_ALL_CXX20_DEPRECATION_WARNINGS /Zc:__cplusplus /std:c++17
ExitCode: 1
Stdout:
pch-cpp.hpp
c1xx: fatal error C1083: Cannot open source file: 'Files\Unity\Hub\Editor\2021.2.0a9\Editor\Data\il2cpp\libil2cpp\pch\pch-cpp.hpp': No such file or directory
pch-cpp.hpp
c1xx: fatal error C1083: Cannot open source file: 'Files\Unity\Hub\Editor\2021.2.0a9\Editor\Data\il2cpp\libil2cpp\pch\pch-cpp.hpp': No such file or directory
pch-cpp-3692469889848082439.cpp
C:\Users\User\Desktop\tessddd\New Unity Project\Library\il2cpp_cache\buildstate\y2vq\pch-cpp-3692469889848082439.cpp: fatal error C1083: Cannot open include file: 'C:\Program': No such file or directory

It seems that pch-cpp.hpp is missing although it is on my computer.

Any idea ? I really wish i could try the 2021.2 new features.

[quote=“pohype”, post:14, topic: 830253]
When trying to Il2CPP build an empty project with 2021.2.0.0a9.1376 i get this error during build time:
[/quote]

There was a regression related to the product being installed into a path with a space in it. This is being fixed. You can track this issue here: https://issuetracker.unity3d.com/issues/building-projects-with-il2cpp-scripting-backend-on-windows-fail-due-to-fatal-error-c1083

@joncham yup it seems due to the space of "Program Files", thank you !

@joncham , would these improvement also work when building for WebGL or is it limited to mobile/desktop toolchains?

Thanks

[quote=“KamilCSPS”, post:17, topic: 830253]
@joncham , would these improvement also work when building for WebGL or is it limited to mobile/desktop toolchains?

Thanks
[/quote]
It also applies to WebGL. The build time improvement on WebGL may vary due to the build process for that platform being more complex than most.

2 Likes

I've noticed faster builds lately (I generally update to the latest alpha when I notice it, or soon thereafter). Actually there have even been times I was like did it really finish already? For WebGL that's a pretty big deal :smile: So yeah.. whatever you're doing, thanks.

2 Likes

@joncham A very long moment is spent waiting for the Microsoft Incremental Linker. I noticed that this process is single thread. On console, we are wasting a considerable amount of time because of this. Would it be possible to analyze if you could use all available CPU cores? Thank you!
7047643--835876--upload_2021-4-16_13-8-25.png

2 Likes