What exactly does the addressables_content_state contain?

Hi there

In my application, all groups Can Change Post Release. I am trying to figure out a build pipeline for our content. I noticed that if I clean addressables, modify a prefab, do a fresh addressables build (not update) and upload the result to the server, the client only downloads the changed data. It ‘understands’ which content is new and which content is unchanged. I imagine this is because the catalogue contains some information about the contents of each group, and the client can use this information to figure out what data has actually changed.

So, then, what is the point of addressables_content_state? Is it simply to tell me when updates aren’t possible (Editor version changed, addressables upgraded?) But otherwise does nothing?

From what I understand, the asset bundles rely on the fact that everything is ‘deterministic’ when being built, so whether I use addressables_content_state and explicitly perform a content update vs performing a clean build, as long as nothing has changed the output is the same, right?

Ultimately: I want to know what addressables_content_state does, so I can replicate the appropriate checks for my own build system to prevent issues.

1 Like

Answer my own question after a quick dive into source code:

The data structure looks like this:

    /// <summary>
    /// Data stored with each build that is used to generated content updates.
    /// </summary>
    [Serializable]
    public class AddressablesContentState
    {
        /// <summary>
        /// The version that the player was built with.  This is usually set to AddressableAssetSettings.PlayerBuildVersion.
        /// </summary>
        [SerializeField]
        public string playerVersion;

        /// <summary>
        /// The version of the unity editor used to build the player.
        /// </summary>
        [SerializeField]
        public string editorVersion;

        /// <summary>
        /// Dependency information for all assets in the build that have been marked StaticContent.
        /// </summary>
        [SerializeField]
        public CachedAssetState[] cachedInfos;

        /// <summary>
        /// The path of a remote catalog.  This is the only place the player knows to look for an updated catalog.
        /// </summary>
        [SerializeField]
        public string remoteCatalogLoadPath;
    }

And here it spells out exactly what it does to make sure Update is possible:

        internal static bool IsCacheDataValid(AddressableAssetSettings settings, AddressablesContentState cacheData)
        {
            if (cacheData == null)
                return false;

            if (cacheData.editorVersion != Application.unityVersion)
                Addressables.LogWarningFormat("Building content update with Unity editor version `{0}`, data was created with version `{1}`.  This may result in incompatible data.", Application.unityVersion, cacheData.editorVersion);

            if (string.IsNullOrEmpty(cacheData.remoteCatalogLoadPath))
            {
                Addressables.LogError("Previous build had 'Build Remote Catalog' disabled.  You cannot update a player that has no remote catalog specified");
                return false;
            }
            if (!settings.BuildRemoteCatalog)
            {
                Addressables.LogError("Current settings have 'Build Remote Catalog' disabled.  You cannot update a player that has no remote catalog to look to.");
                return false;
            }

            if (cacheData.remoteCatalogLoadPath != settings.RemoteCatalogLoadPath.GetValue(settings))
            {
                Addressables.LogErrorFormat("Current 'Remote Catalog Load Path' does not match load path of original player.  Player will only know to look up catalog at original location. Original: {0}  Current: {1}", cacheData.remoteCatalogLoadPath, settings.RemoteCatalogLoadPath.GetValue(settings));
                return false;
            }

            return true;
        }

Here are the errors and warnings:

Addressables.LogWarningFormat("Building content update with Unity editor version `{0}`, data was created with version `{1}`.  This may result in incompatible data.", Application.unityVersion, cacheData.editorVersion);
Addressables.LogError("Previous build had 'Build Remote Catalog' disabled.  You cannot update a player that has no remote catalog specified");
Addressables.LogError("Current settings have 'Build Remote Catalog' disabled.  You cannot update a player that has no remote catalog to look to.");
Addressables.LogErrorFormat("Current 'Remote Catalog Load Path' does not match load path of original player.  Player will only know to look up catalog at original location. Original: {0}  Current: {1}", cacheData.remoteCatalogLoadPath, settings.RemoteCatalogLoadPath.GetValue(settings));
1 Like

So the addressables_content_state.bin file is only relevant when using “Update a Previous Build” to check at build time if that would work?

If the game ‘understands’ which content is new and which content is unchanged even when using “New Build”, why would I need that .bin file and use “Update a Previous Build” at all?

Do you have the .bin file under version control?

1 Like

Here is what I’ve been able to work out through experimentation, because as you know the dev’s are not very active here:

  • addressables_content_state.bin just stores information about the state of the editor at the time of building bundles to ensure the new bundles will be compatible with the existing build. Nothing more.

  • You can build bundles with “New Build” (Bundle A)… upload them and download them to your device… then you can rebuild the bundles again from scratch with the “New Build” option (Bundle B) and upload those too. The device will correctly identify the new assets from Bundle B as being the same as Bundle A, so it wont re-download them.

  • None of this functionality relies on addressables_content_state.bin. All of this works without it. We don’t need to use the “Update a Previous Build” option for this to work.

  • This functionality depends entirely on the build cache in your library folder. If you delete the build cache and run through this process again your device will be forced to download Bundle B entirely, because it thinks everything inside it is different. The new Bundle B will be ever so slightly different (few bytes/kb’s in size), even though the content is supposed to be exactly the same as Bundle A…

  • It seems like when you build the bundles, unity checks the Build Cache to see if it has built that asset before in the past, and if it has then it re-uses the same result. Because (in my tests) the asset bundle build pipeline is not deterministic. It is always slightly different results. The only thing that makes it deterministic is this build cache. No build cache = new content must be re-downloaded in its entirety because all the bundles in the build will be slightly different.

  • This means you need to make sure you are always using the same machine to build asset bundles.

We don’t use addressables_content_state.bin at all. To perform content updates, we just do an entirely new build. So long as we maintain the build cache Unity correctly recognises which asset bundles have changed and only downloads those new assets - we use the “New Build” option every single time.

8 Likes

Thank you for the reply, very helpful!

This behavior seems not mentioned in the document. Can @unity_bill give us an explanation?

1 Like

TLDR; How do you handle that Addressables is able to check for catalog updates when using “New Build > Default Build Script”? Do you patch the catalog filename as shown at the end in the video?

I did experiment with Addressables content updates during the past days too and it’s been a nightmare so far. The required functionality is so trivial, yet it’s so utterly complicated with Addressables.

I’m using Unity 2019.4.20f1 and Addressables 1.18.4.

What I found is when using “Update a previous build” with the addressables_content_state.bin file, Addressables is able to detect if a new catalog is available on a remote location.

However, if I use “New Build > Default Build Script”, then Addressables generates a new catalog and stores it under an unique new filename, causing the existing Player to no longer detect any catalog changes, because the player still points to the old catalog under the old filename.

I found the Player (Build) stores the catalog URL in the file StreamingAssets\aa\settings.json, here is the relevant snippet from that file:

{ ... "m_InternalId":"http://YourServerAddress/catalog_2021.06.07.13.01.40.hash", ...

Using “Update a previous build” makes sure the new catalog is saved using the same name as before. The catalog name is also stored in the addressables_content_state.bin file. Using “New Build > Default Build Script” on the other hand saves the catalog under a new name.

I found addressables_content_state.bin being really fragile, because once you trigger a “New Build > Default Build Script” by accident (which I did during tests), you break the entire update forever and cannot go back, because the new catalog name is stored in addressables_content_state.bin. Adding addressables_content_state.bin to version control would be very important to be able to go back in this case.

Anyway, I really don’t like the addressables_content_state.bin workflow, because it adds so much overhead. We create builds on a build server, which would then need to backup/restore or version control the addressables_content_state.bin file, while nobody on the team wants to deal with it locally. Well, nobody else must ever deal with this file, because it would break future content updates.

I would really want to simply use “New Build > Default Build Script” for every build. As you wrote earlier, bundles with no content changes are not rebuilt, so that’s expected what’s needed.

The issue that persists is the catalog URL. How do you handle that Addressables is able to check for catalog updates when using “New Build > Default Build Script”? Do you patch the catalog filename as shown at the end in the video?

https://www.youtube.com/watch?v=XMtyihUZY0k

PS: Sorry for bad English and accent in the video.

There is an option to specify Catalog file name manually, I use the same Catalog file name always. So doesn’t matter how i build my bundles, results in same Catalog name.

Client is shipped with Blank addressable build (contains only empty Catalog file)

The asset bundles are built. The built bundles are then uploaded to the server. Identified by a unique id.

[Server url]/[unique id]/catalog file

The client connects to server on startup, asks for latest asset bundles URL. Server returns ID of the most recent asset bundles.

Client sets LoadPath to [server url]/[id from server]

That’s the jist of it. After LoadPath is set, if you do:

var resourceLocator = await Addressables.InitializeAsync();
var downloadSize = await Addressables.GetDownloadSizeAsync(resourceLocator.Keys);

It’ll work correctly.

2 Likes

I’m not on my PC at the moment but you might need to also toggle on automatic Catalog update option. Sorry there is so many settings and options, I could be missing a lot of steps potentially… Took me a long time to figure out a workflow that works.

Also I have not had a chance to watch through your video but I hope what I posted is helpful, I can have a look tomorrow some time, maybe I can out together a video of my pipeline if I have some time.

Thank you for the quick reply. I have the “Auto Catalog update” disabled, took me quite a while before I figured that one out too.

There is an option for a custom catalog build path, but if I specify a file name there, Addressables uses it as directory instead and stores the catalog using an unique name under this folder instead.

If you could let me know where to specify the catalog file name, that would be very helpful.

It’s the ‘Player Version Override’ field in the Catalog foldout on the Addressable Assets Settings

3 Likes

Thank you! It was sitting in plain sight the entire time, OMG. I always thought of ‘Player Version Override’ it would override the Player Version in the Player Settings. :sweat_smile:

Haha… yes… it is a very bad name for the variable…

1 Like

How did you go Peter?

1 Like

I’ve the basic thing working, still using a local hosting service. Thank you for the help!

I’m not really certain what the condition to trigger a catalog update is.

My initial assumption was the catalog updates if content is added or removed or the size of an asset changes (in case the catalog contains the file-size). But it seems the catalog even updates if I change the camera clear color in an existing scene.

It seems the catalog update occurs always, regardless of using “Update a previous build” or just “New build”.

You mentioned you use GetDownloadSizeAsync to check for an update. I’m also checking for a catalog update. I’m not sure if that’s really necessary though, because I don’t have the full picture of what all of this is actually doing.

I hate all this black-box-magic. In the past I’ve implemented all of this from the ground up for non-Unity projects in a couple days. It’s really no rocket-science. But Addressables and its documentation makes it so difficult, I’m loosing such much time over this “obfuscated” API and cheap documentation. Not to mention there are still so many bugs with it. It’s very frustrating experience working with Addressables.

3 Likes

Addressables checks the catalog.hash before downloading the .json. It will trigger a catalog update if the hash is different to whats in the local cache.
I think in your case changing the camera color is changing the bundle contents/hash, which in turn changes the catalog hash.

3 Likes

Also bear in mind that your current flow of just using “New Build” for doing catalog updates works fine if you have all your addressables groups as “Can Change Post Release” and no local bundles at all.
The moment you want/need to ship addressables with the player and you wan’t those to potentially be updated without having to release a new player build, you will need to use the “Update Content Flow” and start using and tracking the addressables_content_state.bin file.

1 Like

I was always wondering a bit about the info in this thread about the file contents because our bin files apparently included all the info that is in the catalog, maybe also dependencies and whatnot - files were easily more than 1 mb and we also had issues that too many added or removed files would make it necessary to recreate bin files and rebuild the app with it because we got missing dependency exceptions otherwise. Basically the opposite of what addressables was trying to achieve. The app was trying to load files with IDs that were not mentioned in the new catalog or the recreated bundles because it was apparently still getting the ids from the bin file. We also never deleted any old bundles when updating recreated ones to make old app versions still be able to load “their” dependent bundles, less than ideal also because old app versions would not always download the newest bundle versions. This seems to have changed with the recent update to 1.19.17 (although I don’t see any mention of the fact in the changelog) - the files are now a mere 1-2 kb. We haven’t done any further tests about changed dependency loading behaviour though.