Automate build w/ custom Addressables build script (ie Variation in example repo)

Unity Engine
1

Hi,

We currently use a custom “build maker” script to automatically make builds for us on different platforms, etc.

This is the code we use to make builds & to automate the creation of the Addressables bundles:

// Set Build Target
EditorUserBuildSettings.SwitchActiveBuildTarget(BuildTargetGroup.Standalone, target);

// Handle Addressables, which are actually Asset Bundles.
AddressableAssetSettings.CleanPlayerContent();
AddressableAssetSettings.BuildPlayerContent();

// Build it
BuildPipeline.BuildPlayer(Scenes, FullPathForSingleBuild(buildDate, branch, target), target, GetBuildOptions(optionsMask));

Few questions:

  • We have recently implemented “variations” by following the example shown in the sample repo, pretty much an exact copy of it (‘Advanced/Addressables Variants’ example), for shipping assets in different resolutions.
    It appears that the example shown requires you to build Addressables bundle via the Addressables window → Build button → New Build → “Packed Variations” button.
    Question: How do we automate this / what’s the equivalent way to achieve this via code?

  • We’ve been trying to speed up our builds (and load times), and I see that this package is supposed to already support a semi-intelligent level of “incremental/delta” package builds/updates.

  • Question:*
    A) Is there something specific needed to get incremental building to work, for faster builds?
    B) Does our code above (specifically the ‘CleanPlayerContent()’ call) defeat the incremental functionality – meaning, are we shooting ourselves in the foot?

  • If we know that none of our Addressables have changed, or if we know that our “video frames” did not change but our models did, is there a way [via code] for us to trigger a build for only those groups – or is it all or nothing? Basically we’d love to add to our ‘buildmaker UI’ a way to check/uncheck which groups are re-built & which aren’t; often we’ve just made a small code hotfix & we’d love to get that shipped rapidly, without waiting on a full Addressables ‘packaging’ process to occur [for each target platform heh].

  • Question:* Is there a way to build only certain Addressable groups – or is it an all-or-nothing operation?

Thank you!

2

Bump – this is holding us up and we’ve scoured the docs. @unity_bill perhaps?

3

I’ll flag this for the team to have a look!

4

Thank you very much.

5

Question: How do we automate this / what’s the equivalent way to achieve this via code?

you need to set the index of the builder to use, like so:

int index = -1;
        for (int i = 0; i < AddressableAssetSettingsDefaultObject.Settings.DataBuilders.Count; ++i)
        {
            BuildScriptBase builder = AddressableAssetSettingsDefaultObject.Settings.DataBuilders[i] as BuildScriptBase;
            if (builder == null)
                continue;
            if (builder.Name == "Packed Variations")
            {
                index = i;
                break;
            }
        }
        if (index == -1)
        {
            Debug.LogError("Could not find Packed Variations builder");
            return;
        }
        AddressableAssetSettingsDefaultObject.Settings.ActivePlayerDataBuilderIndex = index;
        AddressableAssetSettings.BuildPlayerContent();

Question:
A) Is there something specific needed to get incremental building to work, for faster builds?

No, when building the first time what can be cached will be cached to Library/buildCache. This includes cache of built content.

B) Does our code above (specifically the ‘CleanPlayerContent()’ call) defeat the incremental functionality – meaning, are we shooting ourselves in the foot?

CleanPlayerContent will wipe the cache, so yes, you are removing any form of faster builds. Not necessarily needed, but, good to do that for a release build for peace of mind. But no need to clean during development.

Question: Is there a way to build only certain Addressable groups – or is it an all-or-nothing operation?

You can disable a group from being included in the build from the Group’s bundled asset’s schema settings. This does removing it completely, including from the catalog.
A lot of the time when building (and you have cached data) is spent calculating dependencies and what has changed and needs to be rebuilt.
When building only a set of your content sometimes, and other times the full content. It could cause you issues. When building dependencies between bundles are setup, and anything that is referenced and not in another bundle being built. Is included implicitly.
It is always safer to build the full content, it keeps it reliable.
It is an advanced use-case, and not without it’s own issues, such as handling shaders. You could use a multi build approach making multiple catalogs, thats are kept the same, between builds. And you know that none of the builds interact with content outside of each other, or can be ok with duplication with implicit asset inclusion.
I would advise seeing if you can use the cache before exploring this. It is quite difficult to fully understand the behaviour and can cause issues if not done right.

1 Like
6

Thank you very much, extremely helpful.

One follow-up (which may admittedly be a dumb/newbie Q): Our “build maker” code runs N builds, one for each platform we’re building for, so usually 2-3 platforms/build targets/actual builds. Does BuildPlayerContent() need called for each build target, or is the built bundle cross-platform usable / do we only need to call it once?

Ideally some of this – the first few answers, especially – could make it into the docs (and/or into the sample repo). Especially the “build automation” aspect, which there isn’t really a great example of anywhere (at least that I’ve seen), and it’s super important & likely very commonly used stuff.

Knowing that the “CleanPlayerContent()” call removes any ‘incremental’ gains – that’s super great to know, we’ve been using the package for over a year now & didn’t really know that, would have been great to see as an example in the repo (a “how to automate builds” example would be lovely TBH).

7

Yes, you need to call build for each platform. Most content is platform dependant.

If this is automated code, then preferably set the platform you want to build with the command line, executing your build and don’t switch platforms during execution of your “build maker” code.
It will likely not have issues, with editor domains not being reloaded after the switch (player code will be correct). But if there are, they are always a nightmare to discover and track down. So it is better to be safe.

BTW, I noticed: “often we’ve just made a small code hotfix & we’d love to get that shipped rapidly, without waiting on a full Addressables ‘packaging’ process to occur [for each target platform heh].”

Note, that Addressables does not include code changes. Code is run from the Player build. Content in Addressables that uses code contains the “assembly name”, “namespace” and “classname” data. Which are then used to lookup and load the correct script built to the Player.
If you are only changing scripts. Then you need to do a Player build, not an Addressables build.

8

It is automated code but we don’t run them via CLI right now, it’s an EditorWindow with checkboxes and a “run” button. It is setup to be able to run from CLI as it just takes an integer/bitmask for “options” and then runs the appropriate builds, but that’s most useful for running a “remote build” on another machine so that it doesn’t tie up a dev’s working directory.

We totally understand that Addressables doesn’t include code, is just assets. What is less clear is how to correctly/safely “preserve” the built Addressable bundles between builds – especially when build targets are changing. The issue is that we (like everyone, I’d think) need an extremely repeatable and universal build process – it needs to ‘just work’ for everyone on the team, every time, without question/doubt/concern.

Here’s a more “complete” example of our build automation code:

What happens here is that we will be building for 2-3 platforms (ie macOS, Windows, occasionally Linux); our build code ‘enqueues’ the proper build targets and then runs the builds needed sequentially until all have been made.

Since Addressables bundles are platform-specific, how are we supposed to do this without calling CleanPlayerContent()? Is the answer to write code that looks at the filesystem (at the build output path), and use System.IO to manually mutate the outputted contents to remove extraneous (ie non-platform relevant) stuff from the resulting StreamingAssets folders in each of the builds? That seems kind of hacky/messy & error-prone.

Our ideal world is we hit “build” and the result that Unity spits out is guaranteed to be “golden” – at the end, our build code spits out some commands to upload the build to Steam. We hit build, walk away for 30 minutes, come back & copy the “deploy commands” that our build-maker outputted to the console, run them in Steamcmd, and we’re done.

It works well & is extremely repeatable [though admittedly I think switching build targets via code was error-prone on macOS editor last I tried it] – it’s just awfully slow, we don’t get any ‘incremental’ benefit. Sure, we could skip the addressables build calls (both ‘clean’ and ‘build’) – but that means we’re expected to have the bundles already built in StreamingAssets, and my understanding is that it means we’d need to manually remove the “mismatched” bundle for each platform (ie ensure the Windows build doesn’t contain the macOS bundle, and vice versa) – correct? Or am I missing something here?

9

We checkout the project for each platform to a separate folder. We never have to change build targets inside Unity between builds.

All folders contain the same project, we pull the latest changes from git before we build, but the “secret” to make the build fast is to keep the projects “Library” folder. It’s where the build system caches all its data and it seems, at least with Asset Pipeline v1, changing the build target invalidates the cache. Using a different project folder for each platform seems to by-pass the cache issue.

1 Like
10

Heh, I see. I assume you then run the builds via CLI so you can run 2 commands & have them both being built in parallel + have the full speed of the cache too, eh?

This should be documented somewhere, I swear. I knew switching build targets felt like it was doing quite a bit of work, and we’ve occasionally [relatively rare] needed to nuke WORKINGDIR/Library/* to fix some kind of “stateful” bug – but I never thought switching build targets was ‘nuking cache’ anywhere even remotely similar to that degree.

Even just anecdotally, I’d be curious to hear how quick your builds are (and I assume you’re always building Addressables eh?) – our project weighs in around 2-3GB (post-build) and we are seeing >= 30min total build times for two platforms (win/mac, both on Mono / not IL2CPP).

Edit to add: Those build times are on new & very high-end machines; latest-gen CPUs, copious amounts of RAM, extreme-performance disk/RAID/IO config, etc…

11

Yes it’s via CLI and yes they do run in parallel.

However, the Unity build process is quite resources hungry and building more than a single project at a time on the same machine doesn’t lead to faster builds. I don’t recommend to build in parallel on the same physical machine, I would recommend to distribute builds across different physical machines instead.

Yes, building Addressables always. It totally depends on:

  • How well is your asset bundle content structured, [see here]( https://discussions.unity.com/t/645929 page-5#post-4473202).
  • How much content changed since the last build, because Unity supports incremental builds, which happens when you don’t clear the Library.

Incremental builds take most of the time less than 10 minutes here, but if there are a lot of content changes, it takes longer.

12

I am not sure what questions there are there, so I will go through how the caching works and hope that answers your questions.

Platform switching:
adbV1: Clears all imported content when switching platforms and needs to reimport everything in the project. Because the import artefacts are stored in a way of 1 artifact for 1 asset in the project.
adbV2: The Library folder contains import artifacts for each platform (some could be shared as it is stored by a hash of the resulting binary). When switching platforms, it will only import what has changed. Since you last had imported that platform. So regularly platform switching is expected to be very fast. If you switch platform and an Asset has changed since you last imported it in that platform, then it’ll import the new result and remove the old result for that platform.

Addressables:
Stores build artifacts in Library/buildCache. Depending on the project, a lot of the time is spent simply figuring out what is being built and how the Assets depends on each other. Then it will build any that are not in the cache and save to the cache.
You can see where Addressables spends time building using the trace data outputted from Diagnostic tools | Addressables | 1.17.17

Incremental building as you may be aware of from the legacy build pipeline doesn’t exist anymore. Instead everything comes from the cached data, which has the what you could call incremental building.
Building Addressables for multiple platforms. (Default local build paths) AssetBundles are generated and stored in the Library/com.unity.addressables folder for each platform separately. Then when building the Player, it will copy the relevant AssetBundles from the Library to your StreamingAssets and remove it again after completing the build.
You do not need to worry about cleaning these build results between builds. For the local content, the AssetBundles are deleted every build. If your build results in the same AssetBundle, then it will have been cached in the buildCache, so it doesn’t need to rebuild it.
If you are building to a custom (or remote build path) location. Then it will not delete before building and saving the AssetBundles there. As they are expected to be managed for your own remote release strategy.

If you delete the Library folder, then you remove everything.
If you CleanPlayerContent, then you remove any cached build data.

13

Wowzers. Massive gains via the guidance provided above, and especially via changing our builds to be CLI-based with 1 repo per build target as @Peter77 suggested. Previously our build times for 2 platforms were >30 minutes total.

After changing our build automation around (CLI driven w/ 1 repo each), well, the results really say it all: we’re now seeing builds for 2 platforms consistently taking under 10 minutes. Back-to-back builds (ie no changes at all) are lightning speed, relative to before, and routinely are <=5 minutes total for two platforms (!!). Pretty remarkable difference.

This stuff should be documented – ideally a “automated project build w/ addressables” example. Thinking about just how much time we wasted over the year or so, just waiting on builds to run, it’s a little bit frustrating since the ‘solution’ (ie massive speed gains) were so close at hand. Unless these kinds of things are documented though, quirks and all, the only way to discover this would have been via trial-and-error (or community / word-of-mouth, as was the case here).

Massive thank you @Peter77 – if you’re ever in the Vegas area LMK, I’d love to buy you a few drinks. Ty!! :slight_smile:

1 Like
14

yes, It’ll be a good idea to get some documentation around this area and CI building in particular. I will make a ticket in our system to work on it.
We know that there are some pain points with CI building, and are working on improving this.

1 Like
15

Alright, so we’ve got this setup now – using exactly the code supplied above for the packing – but we’re getting an error in builds (on both macOS and Windows targets).

Exception encountered in operation Resource<ResourceManagerRuntimeData>(settings.json), status=Failed, result= : Invalid path in TextDataProvider : 'D:/SteamLibrary 2/steamapps/common/SimCasino/SimCasino_Data/StreamingAssets/aa/settings.json'.
(Filename: C:\buildslave\unity\build\Runtime/Export/Debug/Debug.bindings.h Line: 39)
RuntimeData is null.  Please ensure you have built the correct Player Content.
(Filename: C:\buildslave\unity\build\Runtime/Export/Debug/Debug.bindings.h Line: 39)

It works fine in editor play mode. Editor logs don’t show any issues during the builds. We tried upgrading to the absolute latest Addressables release (Preview release; then re-vendoring it + patching it) – same issue.

We’re running editor version 2019.4.23f1 and we tried both our original Addressables version and the latest preview version – results are the same, the content is missing in the builds.

LMK what we can provide to help debug this. It seems similar to the issue in this post – but so far nothing we’ve tried has resolved the issue. Any guidance much appreciated.

16

This fix would have no impact on code interacting with the settings file, if you think it is the same issue. Then it will be best to join that thread. Or make a new bug report for the issue if you think it’s different. Keeping the issue in one place.