[IN-36148] Huge performance drop in BuildLayoutGenerationTask (1.20.3 -> 1.21.9)

[Alan-Liu]

After I upgraded our project to use Addressables 1.21.9 from 1.20.3, BuildLayoutGenerationTask became very slow. Before, it took only less than 6 seconds to finish, but after upgrading, the Editor stuck in BuildLayoutGenerationTask for over 10 minutes and I had to kill the Editor process.

After some investigation, I found it’s due to the following code:

public class BuildLayoutGenerationTask : IBuildTask
{
    // ...

    private LayoutLookupTables GenerateLookupTables(AddressableAssetsBuildContext aaContext)
    {
        // ...

        foreach (BuildLayout.File file in lookup.Files.Values)
        {
            // ...
 
            // Cache all object types from results to find type for when implicit asset
            Dictionary<ObjectIdentifier, Type[]> objectTypes = new Dictionary<ObjectIdentifier, Type[]>(1024);
            foreach (KeyValuePair<GUID, AssetResultData> assetResult in m_Results.AssetResults)
            {
                foreach (var resultEntry in assetResult.Value.ObjectTypes)
                {
                    if(!objectTypes.ContainsKey(resultEntry.Key))
                        objectTypes.Add(resultEntry.Key, resultEntry.Value);
                }
            }
   
            // ...
        }
 
        // ...
    }
}

In our project, lookup.Files.Values has 26336 items, m_Results.AssetResults has 36242 items and objectTypes has 695450 items.

It seems the code of constructing objectTypes can be moved outside the loop:

public class BuildLayoutGenerationTask : IBuildTask
{
    // ...

    private LayoutLookupTables GenerateLookupTables(AddressableAssetsBuildContext aaContext)
    {
        // ...
 
        // Cache all object types from results to find type for when implicit asset
        Dictionary<ObjectIdentifier, Type[]> objectTypes = new Dictionary<ObjectIdentifier, Type[]>(1024);
        foreach (KeyValuePair<GUID, AssetResultData> assetResult in m_Results.AssetResults)
        {
            foreach (var resultEntry in assetResult.Value.ObjectTypes)
            {
                if(!objectTypes.ContainsKey(resultEntry.Key))
                    objectTypes.Add(resultEntry.Key, resultEntry.Value);
            }
        }

        foreach (BuildLayout.File file in lookup.Files.Values)
        {
            // ...
        }
 
        // ...
    }
}

With this modification, BuildLayoutGenerationTask took about 80 seconds to finish, which is still worse than the old version, but is usable in our project.

Hope Unity developers can take a look at this issue.

[Alan-Liu]

Here is the comparision between 1.20.3 and 1.21.9 (with my modification above):
1.20.3:

1.21.9:

[unity_shane]

@Alan-Liu Hey there - nice catch as always. I’ll keep an eye on this and see if we can get this in soon. I’ll also take another look at the build report code to see if there’s any other straightforward optimizations we missed. Thanks for pointing this out! BuildLayoutGenerationTask is pretty much always going to be slower on the newer versions given that the build reporting we do now is much more robust, but 10 minutes is definitely unacceptable.

[einWikinger]

@Alan-Liu , did you also notice that the BuildLayoutGenerationTask is requiring more RAM than before? In addition to it becoming slow, it also started to require RAM in the realm of nearly a hundred of gigabytes (for our project).

[Alan-Liu]

@einWikinger

@Alan-Liu , did you also notice that the BuildLayoutGenerationTask is requiring more RAM than before? In addition to it becoming slow, it also started to require RAM in the realm of nearly a hundred of gigabytes (for our project).

I didn’t notice that before, but I did some tests today, and confirmed there is an unexpected large memory usage in BuildLayoutGenerationTask of Addressables 1.21.9.

Here is the data from my test in Windows Editor (Unity 2020.3.30 + Addressables 1.21.9):

1,2,3 refer to the locations in the code:

public class BuildLayoutGenerationTask : IBuildTask
{
    // ...
 
    private BuildLayout CreateBuildLayout()
    {
        AddressableAssetsBuildContext aaContext = (AddressableAssetsBuildContext)m_AaBuildContext;
        LayoutLookupTables lookup = null;
        BuildLayout result = null;
        using (m_Log.ScopedStep(LogLevel.Info, "Generate Lookup tables"))
        {
            // ----------- 1 -----------

            lookup = GenerateLookupTables(aaContext);

            // ----------- 2 -----------
        }
        using (m_Log.ScopedStep(LogLevel.Info, "Generate Build Layout"))
            result = GenerateBuildLayout(aaContext, lookup);

        // ----------- 3 -----------
        return result;
    }
 
    // ...
}

Apparently, something in GenerateLookupTables caused the memory increased about 8GB.

Looking through the code, I think it’s due to the following code:

private static Dictionary<long, string> GetObjectsIdForAsset(string assetPath)
{
    UnityEngine.Object[] assetSubObjects = AssetDatabase.LoadAllAssetsAtPath(assetPath);
    Dictionary<long, string> localIdentifierToObjectName = new Dictionary<long, string>(assetSubObjects.Length);
    if (assetSubObjects != null)
    {
        foreach (var o in assetSubObjects)
        {
            if (o != null && AssetDatabase.TryGetGUIDAndLocalFileIdentifier(o, out _, out long localId))
                localIdentifierToObjectName[localId] = o.name;
        }
    }

    return localIdentifierToObjectName;
}

As you can see, it calls AssetDatabase.LoadAllAssetsAtPath to load assets and during the execution of BuildLayoutGenerationTask, Unity dosen’t have any chance to unload them, so the memory keeps increasing until all assets that should be packed in asset bundles are loaded.

A workaround is that unloading unused assets after BuildLayoutGenerationTask.GenerateLookupTables processes some asset bundles. Here is an example (Diff for BuildLayoutGenerationTask.cs):

===================================================================
@@ -241,6 +241,8 @@
             Dictionary<string, List<BuildLayout.DataFromOtherAsset>> guidToPulledInBuckets =
                 new Dictionary<string, List<BuildLayout.DataFromOtherAsset>>();

+           long memory = UnityEngine.Profiling.Profiler.usedHeapSizeLong;
+
             foreach (BuildLayout.File file in lookup.Files.Values)
             {
                 Dictionary<string, AssetBucket> buckets = new Dictionary<string, AssetBucket>();
@@ -464,6 +466,9 @@
                         }
                     }
                 }
+
+                if (UnityEngine.Profiling.Profiler.usedHeapSizeLong - memory >= 1024 * 1024 * 1024)
+                    EditorUtility.UnloadUnusedAssetsImmediate();
             }

             foreach (BuildLayout.File file in lookup.Files.Values)

The following data is from the same test using the workaround above:

[Alan-Liu]

Hi, @unity_shane , can you take a look at the memory usage issue above?

Update:
I submitted a new bug report for this issue: IN-37708.

[unity_shane]

@Alan-Liu @einWikinger , Thanks for bringing this issue up - we’ll do what we can to take a look at the memory issue ticket soon. Thanks for making a ticket! Also, I was able to get the fix you described before into the next release, so it’ll be shipping with 1.21.10. There hasn’t been time to do a deeper dive into optimizing BuildLayoutGenerationTask yet, but it’s definitely on the list. Thanks again!

[einWikinger]

@Alan-Liu thanks for looking into it that deeply!
@unity_shane We’ve hotfixed it on our branch now as well, but waiting for the official 1.21.10 release now

[Alan-Liu]

Here is the public link for the memory usage issue: Unity Issue Tracker - Large memory usage when using BuildLayoutGenerationTask