Hello,
Earlier this year, many users have reported increased waiting times (from 2s previously to 4s/6s) when changing one C# script + waiting for a domain reload on a simple Core 3D template or when entering Play Mode, in some 2020.x versions and newer 2021.x versions. We know how much iteration time during development is critical to your success and so, we are creating this post to centralize our communication about, and keep you better informed on the work involved on improving iteration time. Here are some links to previously reported issues:
-
Public Issue: Unity Issue Tracker - Increased Script Assembly reload time
-
Hardware Benchmarks 5800x 8400 10900f: Hardware Benchmarks 5800x 8400 10900f
-
Script change in b8 takes 50% longer to compile+reload than in 2020.1.17: script change in b8 takes 50% longer to compile+reload than in 2020.1.17
-
Any update regarding “Increased script assembly reload time”? Any update regarding "Increased script assembly reload time"?
We have marked an issue with increased reload time as “closed” in our Issue Tracker. That doesn’t mean that we’re done investigating this issue, but the work involved to fix this regression spans across multiple versions of Unity, with multiple fixes from various different teams. And our issue tracking and release process is not suited for such a wide-ranging and long piece of work (usually one basic issue = one fix from one team). We want to keep you regularly informed about our progress, so we’re going to use this thread as a hub to:
- Post regular updates
- Collect additional feedback, especially whenever we have intermediate fixes that we would like you to help us validate
First, thank you to all our users who have been reporting these issues and trying to help us find out what has been happening.
These reports revealed that these regressions happened during some stages in the 2020.x release cycle, and that the situation hasn’t improved for 2021.1 or 2021.2. Now, why haven’t we been able to detect such regressions? While we do have performance tests, they were not fine-grained enough. As such, they were not tracking iteration time on templates like the Core 3D template nor were they tracking regressions that could be caused by packages. Also, at the time, we lacked fine-grained profiling markers in the Editor to detect the cause of such regressions and to correlate them between different editor versions.
From our investigations in April earlier this year, we discovered multiple causes of these regressions:
- Several locations have individually added a small increase in time (e.g. 20ms) but the number of locations made this worse (e.g. x50) generating at least around a second of domain reload impact, and vastly dependent on many variable factors (e.g. number of packages installed, number of windows opened or the layout).
- A change in the compilation pipeline (deterministic feature in 2020.x+ and a new compilation pipeline in 2021+) helped to improve the situation for large solutions but made it worse for smaller projects or smaller changes (e.g. Default Core 3D template), impacting the domain reload by a second.
- We had no good way to get in-depth performance metrics about domain reloads from users.
- Around the same period of time, we also added some packages to the default Core 3D template, without taking the time to look at their impact on iteration time.
The key problem that we found is that there was no “single” source of performance regression, but that:
- Many teams at Unity had added functionalities that had a negative impact on domain reload performance.
- It had totally gone under our radar.
Therefore, we decided to work on different aspects of the issue in parallel:
Domain Reload in-depth performance metrics
Earlier this year, we added a lot of performance markers into the domain reload process, as well as a user-controlled option to add the measurements to the editor logs. This helps you see which component is taking time at which stage of the domain reload process (serialization / deserialization, InitializeOnLoad, Application events, Awake instances, etc.) in the editor log. We internally use this quite extensively now to track our progress, and we also made it available as a diagnostic switch in 2021.1 and later (see this message to learn how to enable it), so that you can give us detailed information about what happens on your machine during a longer-than-expected domain reload. You can also use this to quickly fix for Editor performance issues, when you see that for example a UI widget or an unused package have a significant impact.
A task force dedicated to finding and fixing regressions across the whole code base
We’ve also set up a task force led by our Scripting and Performance teams to identify any opportunity to optimize everything happening during both Script compilation and Domain reload.
Here are the areas where improvements have been introduced positively impacting domain reload time for the upcoming 2022.1:
- Type cache generation
- Profiling tools
- Menu population
- Static batching
- Particle prewarming
- Visual Studio package
- Terrain package
- Polybrush package
In the coming weeks, we will clarify which of these fixes can be backported safely to older editor versions (2021 LTS and potentially 2020 LTS).
And here are the areas being currently worked on:
- Script compilation (especially for small projects)
- Optimizing IO and memory usage for projects with a large amount of C# scripts
- MonoScript type resolution
- Scene backup and updates
- Visual Scripting package
- Legacy plugin system
These optimizations are sometimes done by the task force member themselves, or via direct collaboration with the team owning the relevant component. The Visual Studio Package initialization optimization, for example, has been implemented partially by the Scripting team, and partially by our partners at Microsoft.
In addition to this work, we are going to add broader regression tests to track that changes happening in the Unity Editor/Engine codebase or in a satellite package aren’t slowing down iteration time.
Some of those fixes are easily backportable to previous versions of Unity, and we will make sure that they land in patches for those versions as well. Some other fixes are in packages code and will require updating the packages you use in your projects.
We will come back to this thread, as frequently as we can, to provide any important updates about our progress.
We hope that this post helps to clarify the situation and gives enough insights about our determination to improve iteration time dramatically. We have learned a lot from this issue and we are confident that the tooling we are putting in place will really speed things up when it comes to improving iteration time for both large and small projects.