Asset Importing speedups in 2021.2 - call for feedback

Hi everyone, continuing our focus on performance and fast iteration, we’ve added some asset import improvements to 2021.2 alpha lately. These include texture and model import optimizations, a new option to import textures and meshes in parallel, and various other scalability optimizations.

Here’s a test importing all the textures, models, prefabs and scenes from Book of the Dead: Environment project.

This contains 2.25GB of source asset data (346 textures, 133 models, 214 prefabs, 6 scenes). Timings are on Windows, AMD ThreadRipper 1950X (16 threads), SSD storage machine. We’re comparing Unity 2020.3.10, 2021.1.9 and the current 2021.2 alpha (a19).

Model (fbx, obj, sketchup, …) importing in particular got optimized quite a bit in 2021.2. On average it’s maybe “twice as fast”, but we’ve also seen some pathological cases where a model that used to take hours to import, now imports in several minutes.

Note that parallel asset importing is off by default for now. When turned on, at the moment it only works for texture & model imports, all other asset types are imported in a single process as before. The setting is in Project Settings → Editor → Refresh section. In this example, we see enabled parallel asset import with 4 asset import worker processes.

If, for iteration time purposes, we also use the “Fast” texture compressor option in Build Settings (this does not change any texture sizes or formats, just spends less effort trying to come up with the best possible compressed texture pixels; see this thread for details), all the assets import in 38 seconds. This is a 3.8x speedup compared to the 2021.1 version!
7224691--869056--Clipboard05.png

Switching the same project to Android (ASTC texture format) platform for the first time takes 87 seconds, using both parallel asset import with 4 asset import work processes and the Force Fast Compressor option. This speeds up ASTC texture compression dramatically (6.6x speedup!). However the textures will contain slightly more compression artifacts, so you don’t want to have this setting on for your final build.

Scaling to Large Projects
In our test projects with a large number of assets (like “a million”), we’ve managed to cut the editor startup time by around 50%, compared to previous versions. This is due to various optimizations to asset database processing and data structures. For example, opening a project with 900 thousand asset files, when everything is up to date, took over 3 minutes before, and now takes just over a minute.

Opening of scenes with very large & flat hierarchies (lots of objects at the root level) is also much faster now.

Feedback needed!

It would be useful to have more data on how these improvements work for your 2021.2 alpha projects so far, or if you’re running into any issues. We are therefore asking you to share your Editor performance benchmarks! Please share your numbers in the thread below or send me a DM.

How do you get to those numbers? You can take a look at the editor log file to see import times of individual assets or the whole import (it looks like “Asset Pipeline Refresh: Total: 139.7 seconds”). Among the several command line arguments you can supply upon opening Unity, the -profiler-enable command line argument allows you to profile the Editor during launch. Using this argument can help you see what happens during startup and what takes time.

We’ve also created this Editor Log Parser that will output a CSV file with all the numbers, so that we can put some of those easy to read graphs together. This is especially useful when there is no library and the full project is reimported.

Here are the numbers that we’re most interested in, in the simplest possible format:
Startup
Unity 2019.4: xx seconds
Unity 2021.2: xx seconds

Full project import:
Unity 2019.4: xx seconds
Unity 2021.2: xx seconds

31 Likes

Full project import would be delete library → open? Or right click → reimport?

These are such important improvements. You cannot imagine how happy that makes me. Thanks a lot Aras and team!

2 Likes

Will startup time optimize backport to 2020?
Start a project in HHD also quite slow…

Either, as long as comparison is the same between versions.

As someone with a threadripper who has been struggling with how slow the asset importer is on 2020, this is a great improvement. WP Aras & team. Will definitely be giving it a crack soon, and will put numbers here.

This is very exciting. I’m especially happy to see the editor becoming more multicore friendly. Good job!

How can I get the startup time? I’m trying to see if it’s on the Editor.log, but there are way too many timings there, and I’m not sure which one is the one you want. Is this it?
[Project] Loading completed in 14.133 seconds

Tried the SimpleEditorLogParser, but it’s giving me “Log is not in a supported format. Please implement new parsing for new format present in this log file.” for Unity 2020.3.0 (current project version).

But from just reading the log, startup times are

2020.3.0f1:
Approximately 15.5 seconds

2021.2.0a20:
Approximately 14 seconds

BUT; I had to comment out some some code that happened at startup in 2020.3, since it caused errors on startup in 20201.2 (reflection code that interacted with things that have been removed). Specifically, I had two things that hooked up to DragAndDropService in order to handle some dragging and dropping. I don’t know why it broke, but I don’t have the time to update it now. I also had a custom UnityEditor.OverlayWindow that I booted at startup (apparently, as it was throwing exceptions on boot!). Those have been replaced now, so I had to comment that out as well.

I don’t know how much cost running that code had, but it should probably take something like 1.5 milliseconds, not 1.5 seconds, so there’s a good chance there’s an improvement.

Reimport all time is next, I guess.

1 Like

You can launch the editor with the command line argument -profiler-enable
(i.e Unity.exe -projectpath path_to_project -profiler-enable)
That will capture the timing for the first frame and put it in the profiler that you can bring up :slight_smile:

1 Like

It seems like Assets->Reimport all is down from 2052 seconds to 994 seconds. Going to check deleted library as well

6 Likes

OMG this is amazing. Here my start-up numbers:

2021.1.11: 3m04s
2021.2.20: 0m36s

Assets folder: 102.448 files, 4544 folders, 43.3Gb

10 Likes

I couldn’t use the profiler, because it would just try to import the profiler data for hours, so I ended up using the Editor Log for the timings. I also had to redo the imports a few times, because it would give me out of memory exception on 2020.3. The time of 2021.2 was in the middle of the editor log, so I don’t know how accurate it is. I opened the project a few times until the startup time settled on a number (The first one right after an import was a little bit higher).

The project is mostly PNG files that would be loaded into sprites (~65K png files, ~ 9.5 GB). 2021.2 is set to 6 worker threads:

Startup:
2020.3.11f1: 13.135
2021.2.0a20: 10.869 20% faster

Import:
2020.3.11f1: 5,964 seconds (1:39 hours)
2021.2.0a20: 2618 seconds (43 minutes) 2.27x faster

This is on a 8700K 6 Cores, 32 GB of RAM and SSD. During the import the PC was completely on its own only doing this. I sometimes looked at the CPU/Disk usage. It was usually around 25% CPU usage and 140 MB/s disk usage.

1 Like

Ok, I made a mistake. The numbers for the 2021.2 are for the default import with no worker count set, which is amazing. I’m gonna try again with 6 workers to see how it changes it.

FYI in our measurements it typically makes sense to use about 25% of workers as your CPU has threads – the reason being that e.g. importing one texture is mostly multithreaded already, and importing meshes (since 2021.2) is partially multithreaded too. So if you create N=CPU-threads workers which all have N threads doing work, it’s a bit of CPU over-subscription.

With 6 workers it’s now taking 100% CPU usage and >300MB/s disk usage. I’m still not super sure about the accuracy of these numbers as I still get out of memory errors at the end of the imports. I’ll try to see if I can fix those.

1569 seconds (26 minutes) : 3.8x faster than 2020.3 1.6x faster than 2020.2 without worker threads

I guessed that it would be the case. I just wanted to go wild and see if I can get some more juice out of it. 3.8x still seems logical as the original was going around 25%.

The importing of packages is also multi-threaded now? I saw it goes over 70% when they’re being processed.

Also, what’s the difference between In-Process and Out of Process workers? Does it matter for performance or is it just as a test option?

Packages are no different from any regular assets, as far as importing is concerned.

Hmm you might be hitting https://issuetracker.unity3d.com/issues/editor-crashes-with-system-out-of-memory-error-when-importing-size-heavy-textures which I was just fixing this week (workaround for now would be to switch editor non-DX11 graphics API).

That’s an amazing news, because we had this issue for a long time. Our workaround always was to just open the project again until it reimports everything. The same thing happens when packing the sprite atlases.

I can’t give exact figures, but my (rather large) project import went from “about a week” in 2020 to “about two days” in 2021.1 and “7 hours” in 2021.2a21; but in 2021.2b1 the import takes a little bit longer and never finishes due to an out of memory error. I don’t use dx11, though.

2 Likes