Addressables seems to use Local catalog instead of Remote catalog after new Player Build

[jordivanduijn]

Hi there!

I’m having trouble with my addressables setup. It’s likely something I don’t understand but it might also be a bug.

This is my setup:

• I’m using Addressables 2.2
• I’m using Unity 6000.0.31f1
• I have Local Addressables
• I have Remote Addressables uploaded to Unity CCD using the CCD Manager
• In the Addressable Asset Settings I have checked Build Remote Catalog, set Build & Load Paths to Remote and checked Only update catalogs manually

This is my problem:

• When I make a fresh Player build (Android) right after I have made a Build to CCD everything works as expected and my app loads all remote addressables correctly
• If I now update some remote addressables and choose Update Previous Build from Build to CCD, everything still works correctly; my app loads all remote addressables and they have been updated
• But if I don’t touch the remote addressables, make 1 small change in a shadergraph (the shadergraph is a local addressable), rebuild the local addressables (after choosing Clear Build Cache in the Addressables Groups window) and then make a new Player build, this new app will give 404 errors because it is trying to load remote addressables from a seemingly local catalog.

This is the specific error message:
RemoteProviderException : Unable to load asset bundle from : https://[...].client-api.unity3dusercontent.com/client_api/v1/environments/[...]/release_by_badge/latest/entry_by_path/content/?path=/[...].bundle UnityWebRequest result : ProtocolError : HTTP/1.1 404 Not Found

What I know:

• The catalog from my local addressables build (the one in the aa folder) is not the same as the remote one (hash and bin), so if it uses this one, it will indeed find the wrong addressables
• If I copy the content of the remote catalog_version.bin and catalog_version.hash to the local catalogs, everything works as expected, including the small shadergraph change, but I shouldn’t have to do this manually
• Addressables.CheckForCatalogUpdates and Addressables.UpdateCatalogs should fetch the remote catalog if the hash of the one currently in use (seemingly the local one) is different from the remote one, but when I run Addressables.CheckForCatalogUpdates, I get nothing back so the code thinks the catalog is up-to-date.

This is my CheckCatalog Function:

private static IEnumerator CheckCatalogs()
{
    var catalogsToUpdate = new List<string>();
    var checkForUpdateHandle = Addressables.CheckForCatalogUpdates();
    checkForUpdateHandle.Completed += op => catalogsToUpdate.AddRange(op.Result);

    yield return checkForUpdateHandle;

    if (catalogsToUpdate.Count > 0)
    {
        Debug.Log($"Updating {catalogsToUpdate.Count} catalogs...");
        var updateHandle = Addressables.UpdateCatalogs(catalogsToUpdate);
        yield return updateHandle;
    }
    else
    {
        Debug.Log("All catalogs up to date");
    }
}

So basically I’m looking for the answer to the following question:
How can I make absolutely sure that a new Player build uses the latest Remote catalog for loading the addressables?

Any help would be greatly appreciated and if you are missing any information, please let me know!

Kind regards,
Jordi

EDIT: it turned out that the new player build updated the hash of local addressables which means that when downloading the remote catalog, some remote assets referenced old local addressables which cannot be found anymore. When using CCD, make sure to reupload/update your remote assets every time you make/deploy a new player build!

[timtunity3d]

So there was a bug that was resolved in 2.3.7 that would not properly detect catalog updates. I would recommend using at least 2.4.1 as that solves the Group sorting issue was well. Does the issue still appear in those versions?

The short answer to:

How can I make absolutely sure that a new Player build uses the latest Remote catalog for loading the addressables?

is that you can clear out the value in Player Version Override in the Addressables Settings and you’ll always get a unique catalog filename. My hunch is that isn’t what you’re looking for, however, as it sounds like you have an almost completely working solution here.

[jordivanduijn]

Hi @timtunity3d ,

Thanks for your reply! I will look into updating to the latest version (at least 2.4.1) to see if the issue is resolved. In the meantime I have another question:

I have experimented with the Player Version Override, but I don’t remember that solving my issue. The only thing that setting does is decide the naming for the remote catalog right? Or for the local catalog as well?

In any case, as far as I understand, the naming shouldn’t really matter, because Addressables.CheckForCatalogUpdates() should just fetch the most recent (?) remote catalog and use that one to load addressables. Is this how it works or have I misunderstood something?

Thanks again!

[timtunity3d]

Player Version Override is just used in your catalog filename:

catalog_[Player Version Override Value].json

If you clear out that field a timestamp will be added to the filename instead:

catalog_[timestamp].json

which is useful when developing to ensure you always have exactly the content you want, but is not very helpful when you want to keep pushing out builds that consume the same content.

The URL that is used to download the remote catalog is in settings.json that is bundled with the player. So when troubleshooting what’s going on, it’s useful to check that file first and then make sure the filename matches what you expect to be downloading remotely.

[jordivanduijn]

Hi Tim, thanks again for taking the time to help me out!

I have updated to Addressables 2.4.1 and that seems to have resolved the issue!

If I compare the settings.json generated from 2.2.2 with the one generated from 2.4.1, the only difference is m_AddressablesVersion, so the remote url is the same in both versions. I have confirmed that that remote url is correct, so it was already correct in 2.2.2 which leads me to believe that I was dealing with a bug in the Addressables package.

A couple of things to note:

1. I couldn’t upgrade to Addressables 2.4.1 from the package manager, so I had to manually edit manifest.json to update it. It then shows 2.2.2 as recommended version in the Package Manager which worries me a bit.
2. In the Addressables report, it says that there are multiple catalogs, but shouldn’t there only be one?
   
3. In the Addressables report, it says that these catalogs are in JSON format, while they are uploaded in BIN format (I have Enable Json Catalog disabled in the Addressable Asset Settings). Here is a screenshot of the catalog that exists in my bucket:
   
4. When inspecting the settings.json file that is bundled with my .apk Android build, I noticed that m_CatalogLocations has multiple entries, is this correct?

    "m_CatalogLocations": [{
            "m_Keys": ["AddressablesMainContentCatalogRemoteHash"],
            "m_InternalId": "https://[...].client-api.unity3dusercontent.com/client_api/v1/environments/development/buckets/[...]/release_by_badge/latest/entry_by_path/content/?path=/catalog_0.28.0.hash",
            "m_Provider": "UnityEngine.ResourceManagement.ResourceProviders.TextDataProvider",
            "m_Dependencies": [],
            "m_ResourceType": {
                "m_AssemblyName": "mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089",
                "m_ClassName": "System.String"
            },
            "SerializedData": [7, 76, 85, 110, 105, 116, 121, 46, 82, 101, 115, 111, 117, 114, 99, 101, 77, 97, 110, 97, 103, 101, 114, 44, 32, 86, 101, 114, 115, 105, 111, 110, 61, 48, 46, 48, 46, 48, 46, 48, 44, 32, 67, 117, 108, 116, 117, 114, 101, 61, 110, 101, 117, 116, 114, 97, 108, 44, 32, 80, 117, 98, 108, 105, 99, 75, 101, 121, 84, 111, 107, 101, 110, 61, 110, 117, 108, 108, 75, 85, 110, 105, 116, 121, 69, 110, 103, 105, 110, 101, 46, 82, 101, 115, 111, 117, 114, 99, 101, 77, 97, 110, 97, 103, 101, 109, 101, 110, 116, 46, 82, 101, 115, 111, 117, 114, 99, 101, 80, 114, 111, 118, 105, 100, 101, 114, 115, 46, 80, 114, 111, 118, 105, 100, 101, 114, 76, 111, 97, 100, 82, 101, 113, 117, 101, 115, 116, 79, 112, 116, 105, 111, 110, 115, 50, 0, 0, 0, 123, 0, 34, 0, 109, 0, 95, 0, 73, 0, 103, 0, 110, 0, 111, 0, 114, 0, 101, 0, 70, 0, 97, 0, 105, 0, 108, 0, 117, 0, 114, 0, 101, 0, 115, 0, 34, 0, 58, 0, 116, 0, 114, 0, 117, 0, 101, 0, 125, 0]
        }, {
            "m_Keys": ["AddressablesMainContentCatalogCacheHash"],
            "m_InternalId": "{UnityEngine.Application.persistentDataPath}/com.unity.addressables/catalog_0.28.0.hash",
            "m_Provider": "UnityEngine.ResourceManagement.ResourceProviders.TextDataProvider",
            "m_Dependencies": [],
            "m_ResourceType": {
                "m_AssemblyName": "mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089",
                "m_ClassName": "System.String"
            },
            "SerializedData": [7, 76, 85, 110, 105, 116, 121, 46, 82, 101, 115, 111, 117, 114, 99, 101, 77, 97, 110, 97, 103, 101, 114, 44, 32, 86, 101, 114, 115, 105, 111, 110, 61, 48, 46, 48, 46, 48, 46, 48, 44, 32, 67, 117, 108, 116, 117, 114, 101, 61, 110, 101, 117, 116, 114, 97, 108, 44, 32, 80, 117, 98, 108, 105, 99, 75, 101, 121, 84, 111, 107, 101, 110, 61, 110, 117, 108, 108, 75, 85, 110, 105, 116, 121, 69, 110, 103, 105, 110, 101, 46, 82, 101, 115, 111, 117, 114, 99, 101, 77, 97, 110, 97, 103, 101, 109, 101, 110, 116, 46, 82, 101, 115, 111, 117, 114, 99, 101, 80, 114, 111, 118, 105, 100, 101, 114, 115, 46, 80, 114, 111, 118, 105, 100, 101, 114, 76, 111, 97, 100, 82, 101, 113, 117, 101, 115, 116, 79, 112, 116, 105, 111, 110, 115, 50, 0, 0, 0, 123, 0, 34, 0, 109, 0, 95, 0, 73, 0, 103, 0, 110, 0, 111, 0, 114, 0, 101, 0, 70, 0, 97, 0, 105, 0, 108, 0, 117, 0, 114, 0, 101, 0, 115, 0, 34, 0, 58, 0, 116, 0, 114, 0, 117, 0, 101, 0, 125, 0]
        }, {
            "m_Keys": ["AddressablesMainContentCatalogLocalHash"],
            "m_InternalId": "{UnityEngine.AddressableAssets.Addressables.RuntimePath}/catalog.hash",
            "m_Provider": "UnityEngine.ResourceManagement.ResourceProviders.TextDataProvider",
            "m_Dependencies": [],
            "m_ResourceType": {
                "m_AssemblyName": "mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089",
                "m_ClassName": "System.String"
            },
            "SerializedData": []
        }, {
            "m_Keys": ["AddressablesMainContentCatalog"],
            "m_InternalId": "{UnityEngine.AddressableAssets.Addressables.RuntimePath}/catalog.bin",
            "m_Provider": "UnityEngine.AddressableAssets.ResourceProviders.ContentCatalogProvider",
            "m_Dependencies": ["AddressablesMainContentCatalogRemoteHash", "AddressablesMainContentCatalogCacheHash", "AddressablesMainContentCatalogLocalHash"],
            "m_ResourceType": {
                "m_AssemblyName": "Unity.Addressables, Version=0.0.0.0, Culture=neutral, PublicKeyToken=null",
                "m_ClassName": "UnityEngine.AddressableAssets.ResourceLocators.ContentCatalogData"
            },
            "SerializedData": []
        }

5. One more unexpected thing is happening which needs a bit of context to explain:
   .

• I have remote addressables that may or may not have a reference to a material, these materials are local addressables
• These materials all use the same shader (shadergraph), also a local addressable
• The materials and shader are inside a folder that is marked as addressable, so the assets themselves do not have the Addressable checkbox checked (not sure if this has anything to do with it, but might be worth noting):
  
• If, at runtime, one of these assets does not have a material specified, it loads a default material (NOT marked as addressable, but it is using the same local addressable shader as the other materials)
• When I change the shader used by all these materials to only output pure red as the base color and I make a new build, this is reflected on all of the remote assets that did not specify a material (so they are using the default material), but all assets with a specific material assigned to them (again, these materials are all local addressables), the shader change is not visible.
  .
  This must mean that 2 different shaders are used at runtime, one that correctly outputs pure red, and one that doesn’t have this change. Even with the default material not being marked as addressable, I expected all materials to reflect the shader change. Can you explain this behaviour or do you think this might also be a bug?

As mentioned before, my main issue seems to be resolved, but if you could take the time to clear up some or all of the 5 points mentioned before, that would be greatly appreciated!

Thanks for your time!

[timtunity3d]

1. You should be able to use “Install by Name” from Package Manager with:

com.unity.addressables
2.4.1

Getting the automatic upgrade button working takes us quite a bit longer. But editing the manifest works fine.

3. Is definitely a bug. Clearly we just hardcoded “json” in there.

2 and 4 is expected, but a bit confusing. Basically when you have a remote catalog we have 3 places we check for changes:

1. Remote
2. Cached
3. Local

Remote is the remote file that you upload to your CDN. The second is the cached copy from the last download, and local is a version of the remote file that is shipped with the build. This is in place to keep from downloading large catalogs when they haven’t changed from what’s shipped with the game.

5. Do you have any shaders marked as duplicates in your Addressables Report? Addressables can pull in dependencies unexpectedly and teasting that out can be difficult. It sounds like in one case it has pulled in a copy, and in the other they’re all referring to the Addressables shader.

[jordivanduijn]

Hi @timtunity3d ,

Thanks for clearing up points 1-4, really appreciate it!

As for point 5:

• The Duplicated Assets view in the Potential Issues tab of my Addressables Report is empty, so I believe there should not be any copies:
  
  
• In the Explore tab, the shader and the materials (excluding the default material I mentioned in my previous post) are all visible and have expected Refs To and Refs By counts:
  
  

I think this means that there shouldn’t be any duplicates of the shader, but looking at the runtime behaviour in my player build, there has to be at least one duplicate.

Regardless of the duplicates, I expect the shader change to work on all local versions of that shader (even though there shouldn’t be any duplicates), because the shader is definitely not a remote addressable so when making a new player build, the shader changes should exist on all local versions. It is almost as if a local version is cached to be used for the local addressable materials and only the non-addressable material uses it’s own updated/correct and non-cached version of the shader.

Note: I have tried to clear the addressables build cache through Addressables Groups > Build > Cear Build Cache > All, but my player build still showed the shader change (pure red base color) on the assets that use the default material, and no shader change on assets that use a local addressable material.
.
.
Another note: as I was testing out some stuff to create this reply and made the first player build, I was back to square 1: no remote assets could be loaded because a seemingly wrong catalog was used (maybe the local copy of the remote catalog that you mentioned in your previous post?). For the past couple of days I had made some script changes and when I stashed these and made a new build, the remote addressables were found again. I then applied the stashed changes and made a new build to check if these script changes were indeed the culprit, but this player build was also fine (?!). I have also tried introducing scene changes, but they also didn’t break the player build, so now I’m not able to reproduce what happened on my first player build this morning…

A question to which the answer might clear up what kind of changes break my build:
Are there specific kinds of changes that sort of “invalidate” addressables? (An obvious one might be a script change to the Scriptable Object definition of remote assets of that type of Scriptable Object)

Whenever my player build breaks (remote assets cannot be found) it really feels like the Addrssables system is using that local copy of the remote catalog, but the locations in this catalog do not match with what is in the ACTUAL remote catalog.

Could it be that my player build simply ALWAYS uses the local copy of the remote catalog, and if it happens to be the same as the actual remote catalog, everything is fine, but if it is different, the remote assets cannot be found anymore?

I have Only update catalogs manually turned off:

I hope you will find the time to help me out a bit more, I must confess that I’m getting a bit frustrated about this whole situation and I’m honestly not sure anymore if it is my own stupidity or bugs in the Addressables package that I’m dealing with …

Thanks a lot in advance!

[timtunity3d]

Ok. Staring at this a bit more. One thing I noticed is you’re using CCD. CCD can take up to 5 minutes to update badges to point at new code. Is it possible that’s the issue you’re seeing with missing content?

[jordivanduijn]

Hi @timtunity3d,

That sounds pretty plausible, as soon as I have a couple of hours I will experiment to see if my impatience (+ not knowing about these 5 minutes) is the cause of the missing content. (Me and my wife just had our first child, so I’m having a bit of trouble to find some dedicated work time)

In the meantime, do you have any extra thoughts about the shader? On a previous project where we used Addressables I’ve see similar behaviour where changes to a local shader didn’t show up at runtime because a remote addressable version of that shader was being used instead. But in this case I am 100% sure that the shader is a local addressable, so I’m really confused about why I am not seeing my shader changes.

Could it be that a non-addressable material just drags in the shader it needs as a regular asset regardless of that shader being addressable or not and that addressable assets use the actual addressable “version” of that shader? I think that would explain why the shader doesn’t show up as a duplicate in the addressables report. If this is the case, that still doesn’t explain why the shader changes are not visible in a new player build because that shader is a local addressable and should have been re-build into the player build with the new changes right? Unless the local addressable shader is cached somewhere and the system doesn’t detect the changes (which would mean that Addressables Groups > Build > Clear Build Cache > All wouldn’t clear that cache, because I’ve tried this)?

If you have a moment to try and answer some of my questions, that would be greatly appreciated! If you suspect this is a bug, I could try to invest some time to make a minimal working example, but I really hope I haven’t spent so much time on a bug.

I really appreciate you engaging with me on this issue, it helps a lot and I feel like I have learned quite some things that weren’t (cearly) documented somewhere.

Thanks in advance for your time!

[timtunity3d]

So I think it is the material issue. So I built a project with a remote Addressable Sphere prefab that references a local Addressable material that references a local Addressable shader. You can see in this screenshot that it depends on the local bundle for the material and shader.

If I simply uncheck Addressable on the material, but keep the Shader Addressable you see this:

Both the shader and material have been copied into the remote bundle. Changes to the shader in the local bundle will not impact the remote bundle since these are copies and not references.

Does that answer your question?

[jordivanduijn]

Hello again @timtunity3d,

Thanks for taking the time to create a mini repro project. Unfortunately this doesn’t answer my question. Everything you showed is exactly how I would expect it, my case is just different. This means I was probably not clear enough, but as mentioned before, my assets do or do not reference a material. If they do, it is one of the local addressable materials, but if they don’t, a default material is loaded. Here is an example of an asset that references a specific material and one that doesn’t:

This asset doesn’t need a unique material, so the material field is left empty:

So it uses the default non-addressable material because of this line of code:
m_puzzleMaterial = puzzle.material != null ? puzzle.material : Resources.Load<Material>("PieceInstance/M_Puzzle_Default");

In the addressables report, there is no reference to a material (as expected):

This is how it looks in build after the shader change (base color set to pure red, so the result is as expected):

.
.
.
And this asset needs a unique material:

The material (local addressable):

In the addressables report, a bundle that contains the local addressable material is referenced (as expected):

This is how it looks in build after the shader change (the pure red shader change is not visible, not expected):

A detail that I didn’t mention before is that this material is loaded from the Resources folder. There is actually not a very good reason that this default material is not marked as addressable, so I can definitely change that, but that doesn’t change the issue: my shader changes don’t show up on the local addressable materials.

So again, it is only this default non-addressable material (loaded from the resources folder) that shows the shader changes on a new player build, all of the other local addressable materials do not show the shader change. Any more ideas? Cache issue? Bug?

Thanks again for taking the time to help me out!

PS: the screenshots are from an Android player build in which the remote addressables are still loaded successfully, even after a local shader change (this didn’t work before upgrading to the new version of Addressables and was my original issue for this post which you helped resolve). I had to make a player build though, because in-editor, the remote addressables cannot be found again when going into playmode. The Play Mode Script is set to:

So when it comes to loading addressable assets, I expect no difference between editor and build, but there is.. A screenshot of my console debug log probably won’t help a lot, but I’ll leave it here anyway, just in case:

Please let me know if you need more information, or if you need me to try something out!

[timtunity3d]

It definitely feels like the issue here is around catalogs and filenames. If you look in your log it’s not able to find an asset bundle in the build folder inside the Library. That folder is cleared out on each content build so there shouldn’t be a way for that to happen. The player should always be able to find local bundles.

What can happen is related to remote catalogs. Lets say I have a remote catalog deployed that refers to a local asset bundle and remote asset bundle:

remote_abcdef.bundle
local_ghjijkl.bundle

Lets says local_ghjijkl.bundle has a shader inside with the Addressable key myShader.

If I change the shader locally and rebuild my code the hash on the local bundle will change. Let’s call it local_mnopqrs.bundle. When I start my app the local catalog that was built with my player will say that myShader refers to bundle local_mnopqrs.bundle.

But the first thing we do in Addressables is update the remote catalog. We download the catalog again and overwrite the catalog you just built locally. Now that catalog says myShader is in bundle local_ghjijkl.bundle and we’ll get an error that it can’t be found when we try to load it.

With Unity’s Cloud Content Delivery product we recommend always uploading all your remote bundles and remote catalog on each build because it makes this sort of issue less likely to happen. We recommend using the Addressables profile system to deploy into a per user bucket for local development, and then switch to a separate profile with your QA and production content that is built by a CI.

Does that make sense? Does it seem like this could be what’s happening on your end?

[jordivanduijn]

Hi @timtunity3d ,

Thanks again for taking the time to look into my issues! What you are describing actually makes a lot of sense. I have been assuming that remote addressables couldn’t be found, but as you explain it, it’s local addressables that cannot be found because old hashes from the remote catalog don’t exist anymore on the new build. I will look into this shortly and verify that this was the issue, but it seems highly likely!

I missed/forgot the part of the documentation that recommends always uploading all the remote bundles on each build, but this would definitely fix the issue! I do think that most QOL-kind of shader changes would not need to update the hash and therefore don’t require a reupload of the remote bundles on a new build, but it is understandable that they do from a complexity standpoint.

This doesn’t explain why I seem to have a duplicate of my shader though. With what you’ve just explained, shouldn’t a shader change to a local addressable shader either result in that shader not being found or the shader change being visible? When all local addressables are still being found (hash wasn’t updated?) AND the shader change is not visible, shouldn’t that mean that an OLD version of that shader is being used? Looking at the addressables report, it wasn’t uploaded as a remote addressable, so could it be that a cached version of a local addressable is used?

One last thing: is it easy to check if local hashes have changed after making changes to a shader?

Thanks again, I really appreciate it!

[jordivanduijn]

This post is very suspicious. It feels like it’s generated with AI and some weird words have been placed in various spots and a suspicious URL at the end.