Improving iteration time on C# script changes

Unity Engine
, ,
1

Hello,

Earlier this year, many users have reported increased waiting times (from 2s previously to 4s/6s) when changing one C# script + waiting for a domain reload on a simple Core 3D template or when entering Play Mode, in some 2020.x versions and newer 2021.x versions. We know how much iteration time during development is critical to your success and so, we are creating this post to centralize our communication about, and keep you better informed on the work involved on improving iteration time. Here are some links to previously reported issues:

We have marked an issue with increased reload time as “closed” in our Issue Tracker. That doesn’t mean that we’re done investigating this issue, but the work involved to fix this regression spans across multiple versions of Unity, with multiple fixes from various different teams. And our issue tracking and release process is not suited for such a wide-ranging and long piece of work (usually one basic issue = one fix from one team). We want to keep you regularly informed about our progress, so we’re going to use this thread as a hub to:

  • Post regular updates
  • Collect additional feedback, especially whenever we have intermediate fixes that we would like you to help us validate

First, thank you to all our users who have been reporting these issues and trying to help us find out what has been happening.

These reports revealed that these regressions happened during some stages in the 2020.x release cycle, and that the situation hasn’t improved for 2021.1 or 2021.2. Now, why haven’t we been able to detect such regressions? While we do have performance tests, they were not fine-grained enough. As such, they were not tracking iteration time on templates like the Core 3D template nor were they tracking regressions that could be caused by packages. Also, at the time, we lacked fine-grained profiling markers in the Editor to detect the cause of such regressions and to correlate them between different editor versions.

From our investigations in April earlier this year, we discovered multiple causes of these regressions:

  • Several locations have individually added a small increase in time (e.g. 20ms) but the number of locations made this worse (e.g. x50) generating at least around a second of domain reload impact, and vastly dependent on many variable factors (e.g. number of packages installed, number of windows opened or the layout).
  • A change in the compilation pipeline (deterministic feature in 2020.x+ and a new compilation pipeline in 2021+) helped to improve the situation for large solutions but made it worse for smaller projects or smaller changes (e.g. Default Core 3D template), impacting the domain reload by a second.
  • We had no good way to get in-depth performance metrics about domain reloads from users.
  • Around the same period of time, we also added some packages to the default Core 3D template, without taking the time to look at their impact on iteration time.

The key problem that we found is that there was no “single” source of performance regression, but that:

  • Many teams at Unity had added functionalities that had a negative impact on domain reload performance.
  • It had totally gone under our radar.

Therefore, we decided to work on different aspects of the issue in parallel:

Domain Reload in-depth performance metrics
Earlier this year, we added a lot of performance markers into the domain reload process, as well as a user-controlled option to add the measurements to the editor logs. This helps you see which component is taking time at which stage of the domain reload process (serialization / deserialization, InitializeOnLoad, Application events, Awake instances, etc.) in the editor log. We internally use this quite extensively now to track our progress, and we also made it available as a diagnostic switch in 2021.1 and later (see this message to learn how to enable it), so that you can give us detailed information about what happens on your machine during a longer-than-expected domain reload. You can also use this to quickly fix for Editor performance issues, when you see that for example a UI widget or an unused package have a significant impact.

A task force dedicated to finding and fixing regressions across the whole code base
We’ve also set up a task force led by our Scripting and Performance teams to identify any opportunity to optimize everything happening during both Script compilation and Domain reload.

Here are the areas where improvements have been introduced positively impacting domain reload time for the upcoming 2022.1:

  • Type cache generation
  • Profiling tools
  • Menu population
  • Static batching
  • Particle prewarming
  • Visual Studio package
  • Terrain package
  • Polybrush package

In the coming weeks, we will clarify which of these fixes can be backported safely to older editor versions (2021 LTS and potentially 2020 LTS).

And here are the areas being currently worked on:

  • Script compilation (especially for small projects)
  • Optimizing IO and memory usage for projects with a large amount of C# scripts
  • MonoScript type resolution
  • Scene backup and updates
  • Visual Scripting package
  • Legacy plugin system

These optimizations are sometimes done by the task force member themselves, or via direct collaboration with the team owning the relevant component. The Visual Studio Package initialization optimization, for example, has been implemented partially by the Scripting team, and partially by our partners at Microsoft.

In addition to this work, we are going to add broader regression tests to track that changes happening in the Unity Editor/Engine codebase or in a satellite package aren’t slowing down iteration time.

Some of those fixes are easily backportable to previous versions of Unity, and we will make sure that they land in patches for those versions as well. Some other fixes are in packages code and will require updating the packages you use in your projects.

We will come back to this thread, as frequently as we can, to provide any important updates about our progress.

We hope that this post helps to clarify the situation and gives enough insights about our determination to improve iteration time dramatically. We have learned a lot from this issue and we are confident that the tooling we are putting in place will really speed things up when it comes to improving iteration time for both large and small projects.

113 Likes
2

Is this fix about issue #1143037, where static batching n scenes on entering play mode in the editor actually batches 1+2+…+n scenes? That cost us ~60 seconds on enter play mode back when I reported it in 2019, but the bug report got “Our developers are currently working on other projects that are of a bit higher priority than this case.”

2 Likes
3

This static batching optimisation isn’t specifically related to that issue you listed. Thanks for highlighting it. I’ll discuss it with the team that made the new optimization to see if we can make gains there too.

The optimization listed above was to move the static batching work to happen in parallel across available cores, greatly speeding up the time taken on multi-core machines.
There is also a secondary optimization which speeds up the sorting phase by removing some unnecessary logging.

1 Like
4

I just want to chime in to say that this type of communication is very valuable and appreciated. Thank you!

31 Likes
5

I agree with joNax. It’s good to know Unity is aware of this issue, is taking it seriously, and is actively working on getting this fixed. Good job!

6 Likes
6

Thank you, xoofx.

Including the fixes described above – is parity with 2019.x achieved? I don’t think the task force should rest until iteration returns to sub second.

5 Likes
7

Thank you for communicating about this important issue, xoovf.

Now I really hope management doesn’t pull you guys away from that and put you on other “priorities”. Fingers crossed.

1 Like
8

It is unlikely that we will get to strict parities because in the meantime, more features have been added. But we hope to get back to an iteration time that is between 1s to 2s for a simple project changing a single C# script of the main assembly.

There is a hard limit also in how much code we can run on the main thread, and moving this code in separate threads will require a lot more substantial work than the optimization done for mitigating the recent regressions.

5 Likes
9

I really appreciate the update, thank you xoovf and team.
This is such an important issue to me, it makes me feel a lot better having this acknowledgement.

Please let us know how the community can help. Whether it be profiling data or even uploading whole projects, if that helps. I know I’d be happy to provide, if it proves useful.

2 Likes
10

I appreciate this update!

I wanted to focus on briefly mentioned detail in particular: as much as possible, it would indeed be great if all fixes are ported to 2020 LTS. I understand that this is caused by a multitude of factors, some more recent then others. But for parts that originate from changes in Unity 2020, it would be good to rectify them for 2020 LTS - not every project has ability to jump to 2021 and getting stuck with permanently slowed down iteration time wouldn’t be ideal.

4 Likes
11

Agree with @stonstad and got surprised by that answer… sorta asumed the final objective was to return to previous iteration times, independently of how many features are added to the engine. If we are going to be losing seconds to every thingie that Unity introduces, the whole thing would be unsustainable in a matter of months :hushed:

1 Like
12

Thanks for the updates

We are currently at 2019LTS, and we want to migrate our projects to 2021 LTS (as soon as it is available)
Would it be possible to have these optimizations landing on the 2021 LTS tech cycle?

1 Like
13

Is it possible to turn off the auto compiling, and manually trigger it when I want to? after I hit save in VS unity turns unresponsive for up to 10 secs, and only then does the green percentage bar appear, for another 5-6.

5 Likes
14

If I understand you correctly, then you can do that from Edit → Preferences → General and turn off Auto Refresh and Directory Monitoring. When you need Unity to compile/refresh, hit CTRL+R. Just know that if you forget to hit CTRL+R then you won’t have the newest changes from your script in Unity. I ended up creating a new button in the editor that I use instead of play mode that refreshes first so I reduce the chance of going into playmode with old scripts.

5 Likes
15

Try to compare it to making a script change in Notepad. This could be VS acting up - it often does.

1 Like
16

That’s a brilliant work-around! Would you mind posting your button source on a Gist?

1 Like
17

Thanks :), it’s nothing fancy, and required me to remap some muscle memory for a bit, but otherwise, it works well enough for me at least.
Sure, you can find it here: https://github.com/Per-Morten/UnityUtilities/blob/main/RefreshAndPlayButton.cs

9 Likes
18

Reading this thread has alone regrown some hairs on my head. Thank you @xoofx I know you personally push for a lot of boring/subtle but ultra important improvements behind the scenes!

What would be the ideal way to present additional problem areas for this? For example, our project has become extremely laggy with the in-editor experience over the past couple of years. Here is a quick example of me interacting with a couple of drag-n-drop elements:

We’ve done our best to debug and isolate the root cause but its been too hard without deeper metrics into Unity Editor.

19

If you don’t want to learn a new button and an extra window, just put it in a file in an Editor folder:

using UnityEditor;
using UnityEngine;

public static class EnterPlayMode
{
    [RuntimeInitializeOnLoadMethod]
    private static void Init()
    {
        EditorApplication.playModeStateChanged += OnEnterPlayMode;
    }

    private static void OnEnterPlayMode(PlayModeStateChange obj)
    {
        if (obj == PlayModeStateChange.ExitingEditMode) AssetDatabase.Refresh(ImportAssetOptions.Default);
    }
}

Note for people who read this out of context: read the thread. This editor script should only be used when you turn off the directory monitoring and auto refresh. If you don’t have problems with your code editor and Unity editor working together by default, it isn’t for you, probably.

11 Likes
20

Ok, that’s way better, thanks :slight_smile:

1 Like