Loading content with Signed Urls

Our current setup is along the lines of this:

  • Request list of all asset bundles for a particular build from our server
  • List is returned with signed urls (google cloud storage)

Desired Outcome:
Load the content catalog, hash file, and asset bundles using the new signed urls.

Issue: How do I override the Content Catalog and Hash loading to use the new signed url?

I believe I can create my own AssetBundleProvider and simply replace the urls with the new signed ones when it tries to download. But can I create my own ContentCatalogProvider?

The reason for this is we don’t want people to learn the url scemes and try to snoop prying for access to files that aren’t ready for public yet. Signed urls allow per-file authentication process to avoid this.

Thanks,
Quinn

Why not use remote catalog, then convert all urls in the json file to signed urls, then upload to your RemoteLoadPath. So you won’t need to hack the system, though I guess you will find a way eventually.

yes, you can do both of those things, and we are working on an example (or perhaps built in provider) to do that.

As to @Favo-Yang 's suggestion, that could work too (editing the catalog at build time to include the correct paths). This obviously won’t work if your URLs are dynamic.

Thanks @Favo-Yang and @unity_bill .

The signed urls are dynamically generated and expire after 15 minutes. So there’s no way to just compile them into the build sadly. I am currently using a Remote Catalog.

I saw there’s an Addressables.LoadCatalogAsync(string path) which I’ve tried to send it the signed-url path to the catalog on the server but was having issues still… Maybe having a remote catalog isn’t necessary though. The way I’m storing bundles is in a per-build specific folder (to keep backwards compatibility for testing previous builds). So maybe I just have a local catalog and append the signed urls when downloading the bundles?

To actually get the bundles, yeah, I’d recommend a custom provider that appends the signed url as needed.
To get the remote catalog, you may need to derive from JsonAssetProvider and override the Start() method.

A remote catalog is only necessary if you want to be able to update content without publishing a new build. If every time you change something, you ship a new full build, then the remote catalog is not needed.

An update on this, I successfully am using runtime generated signed urls to load asset bundles. Turns out was actually fairly simple. Here’s how I’m achieving this:

Step 1: Download list of all the signed urls
For us this is easy, we’re dumping all the bundles into 1 folder, so our server engineers setup a separate API call for me to get all signed urls for a folder in a Google Bucket. (game specific, aka not a unity call here)

Step 2: Create your own AssetBundleProvider
I simply found the Addressables default AssetBundleProvider and AssetBundleResource, copied them into my own ProjectAssetBundleProvider and made the following changes:

UnityWebRequest CreateWebRequest(IResourceLocation loc)
{
    // This is your Step 1 which contains all of your signed urls
    _assetBundleUrlResolver = AssetBundleUrlResolver.Instance;
   // Get the signed url for this particular asset
   // The default loc.InternalId here would be the original unsigned url (http://www.storage.google.com/yourbucket/blah.bundle)
    string signedUrl = _assetBundleUrlResolver.GetSignedUrl(loc.InternalId);
    if (m_Options == null)
        return UnityWebRequestAssetBundle.GetAssetBundle(signedUrl);

    // Note the rest of this is untouched from original CreateWebRequest function.
    var webRequest = !string.IsNullOrEmpty(m_Options.Hash) ?
        UnityWebRequestAssetBundle.GetAssetBundle(signedUrl, Hash128.Parse(m_Options.Hash), m_Options.Crc) :
        UnityWebRequestAssetBundle.GetAssetBundle(signedUrl, m_Options.Crc);

    if (m_Options.Timeout > 0)
        webRequest.timeout = m_Options.Timeout;
    if (m_Options.RedirectLimit > 0)
        webRequest.redirectLimit = m_Options.RedirectLimit;
#if !UNITY_2019_3_OR_NEWER
    webRequest.chunkedTransfer = m_Options.ChunkedTransfer;
#endif
    if (m_ProvideHandle.ResourceManager.CertificateHandlerInstance != null)
    {
        webRequest.certificateHandler = m_ProvideHandle.ResourceManager.CertificateHandlerInstance;
        webRequest.disposeCertificateHandlerOnDispose = false;
    }
    return webRequest;
}

The AssetBundleUrlResolver.GetSignedUrl function is simple, just doing a simple string.Contains search of the signed urls using the original url.

public string GetSignedUrl(string originalUrl)
{
    // SignedUrls is simply list of string urls that were signed by the server.
    foreach (var url in SignedUrls)
    {
        if (url.Contains(originalUrl))
            return url;
    }

    return null;
}

Step 3: Set your bundle provider in Addressables window to be your own

I hope this helps someone!

EDIT: Btw, I’m also still using a remote catalog. Turns out it runs through my provider to get the remote catalog so it’s checking the signed url version of the catalog.

4 Likes

@unity_bill This approach works above, but one concern here is if there are any updates to how Addressables BundleProviders work, i’ll have to copy/paste this code back in with each update to be sure we’re using the latest code. Which I assume is a problem for anyone writing custom providers for things.

That’s not as bad an issue as you think. We have custom stuff in several packages, Addressables, TextMeshPro, LWRP, etc. If you are using custom stuff you should:

  • Move the package from Library/PackageCache/ to Packages/
  • Rename the package not to have a version suffix.
  • Commit it to your repo without your changes.
  • Commit your changes.

To update the package:

  • Update your working directory back to the revision you committed the vanilla package in.
  • Delete the package.
  • Update using the package manager.
  • Move the package from Library/PackageCache/ to Packages/
  • Rename the package not to have a version suffix.
  • Commit.
  • Merge your changes in.

We use Mercurial for this, where I have a branch for import of each package. It works a treat and you can easily inspect what changes are made to the packages by the team this way. If you have a custom provider, it’s very easy to check the diff of the original one and make the same mods on yours.

2 Likes

We host our assets files on a Playfab CDN and to access them we must first call GetContentDownloadUrl API (a HTTP request to get a signed URL for accessing that file) and then make the GET request to that signed URL to actually get the file. But The URL expires in x time, so to make sure everything is still valid, we must call GetContentDownloadUrl API before every attempt to download a file.
We managed to implement our own AssetBundleProvider and set it to the AssetBundleProviderType in our remote group asset, as @CodeBombQuinn suggested. The problem is we don’t have the list of all the signed URLs and we must get each one before making each get resource request. Does anyone knows how inject another HTTP request before the one that gets the asset using a custom AssetBundleProvider? Or how to try another approach?

One way of doing that would be modifying AssetBundleProvider::BeginOperation() to detect whether you are going to need to sign the Url and chain the operations if so, setting the creation and sending of the AssetBundle web request to be send by the completion handler of your GetContentDownloadUrl() operation. You need to be careful with caching, you don’t want to end up doing this even if the file is available locally.

The only thing on our roadmap to do to the AssetBundleProvider eventually is “add signed url support”. So with any luck, the first change we throw at you that conflicts with your work will also make it unneeded.

But as @AlkisFortuneFish has pointed out, it is a manageable problem.

Hey any update on this thread its been year but i havent found anything helpful to change my url path after lazy init

1 Like

@unity_bill

Just curious. I see you’ve posted a lot in threads about this issue which makes private usage of addressables kind of useless. Has there ever been a demo made for this or any addressable updates to account for a custom provider or signed urls?

Hey all, I just wanted to post something that might help. There’s a Addressables.ResourceManager.InternalIdTransformFunc that you can set on the ResourceManager that will let you alter the url that gets used before Addressables sends the UnityWebRequests for the AssetBundle(s).

I think we have this documented but I’ll double check.

Let us know if there’s still any issues. Thanks!

just checking to see if there is any example to reference for this - setting out to implement a solution soon :slight_smile:

This worked for me! Was able to remove my custom Addressables code completely!

My Implementation (hope it helps someone):

Addressables.ResourceManager.InternalIdTransformFunc += MyTransformAddressableIdFunc;
public string MyTransformAddressableIdFunc(IResourceLocation location)
{
    // If we're trying to load an asset from online, swap the URL for signed version.
    if(location.InternalId.StartsWith("http") &&
        !location.InternalId.Contains("localhost")) // Don't swap if we're running Editor Hosting
    {
        // This function is all on you to determine what to do with this.
        return GetSignedUrl(location.InternalId);
    }

    // Return default location if not a server url
    return location.InternalId;
}

My GetSignedUrl is simply swapping out that URL for my signed version, notice I’m ignoring Editor Hosting swaps since those don’t need Signed Urls.

And here’s official Documentation on that func: https://docs.unity3d.com/Packages/com.unity.addressables@1.14/manual/TransformInternalId.html

2 Likes

Absolutely useless if you need to make some 3rd party API request (i.e. PlayFab CDN) in order to acquire the bundle URL.
Moreover there are NO working solutions out there as of right now. This must be a joke.

1 Like

I’m not familiar with the PlayFab workflow specifically but what would you normally do to query for that url? Would it be possible to obtain the url prior to making your Addressables requests and use that when TransformInternalId is called to modify your url to the correct bundle location? It’s an interesting problem.

there has to be some way using the transform func. It works pretty well… I am using it extensively in both setting my remoteload path and signed url.

My remote location link completely changes depending on which section they want to look download so its not as easy as just adding a signed url to the end. Since we have 30 different procedures in the app we didnt want to download them all at once because they can be big. This gives them the option to only download content they wish to view. Because of this the different catalogs are stored in different directories on private Azure Blob Storage. The TransformInternalId func actually works great for this if i set the content link right before downloading the addressabels so it applies both the remote location url and adds the private blobs signed policy to the url to allow access to the private content. So the Remote Location url is completely dynamic for me in this way. And looks similar to this in the Addressables Profile for Remote Load Path:

{AddressableURL.remoteLoadPathUrl}/[BuildTarget]/{AddressableURL.remoteLoadPathVersion}
AddressableURL.remoteLoadPathUrl =
*https://azurestorageaccountname.blob.core.windows.net/release/MyApp/MyProcedureName/*_
[BuildTarget] = Android_

AddressableURL.remoteLoadPathVersion = 1.0
catalog_2021.xx.xx.xx.xx.xx.json
and then ?my_signed_url_to_add_to_the_end

Final Transformed RemoteLoadPath:
https://azurestorageaccountname.blob.core.windows.net/release/MyApp/MyProcedureName/Android/1.0/*catalog_2021.xx.xx.xx.xx.xx.json?my_signed_url_to_add_to_the_end*

The biggest hurdle was new builds with new addressable content without affecting previous builds addressables. So I used the app build version which is in the url to download the new correct content for that build and found in the the addressable profile the Cache Clear Behaviour which allows me to Clear When New Version Loaded. (Although in atleast in addressables version1.17.17 its called Clear When When New Version Loaded) :eyes:

So far everything I am doing between these has addressables, versioning and url signing for private blobs working like a charm.

1 Like

Hi all just wanted to mention an approach that worked for my case. I wasn’t using signed urls, but I needed to replace my load paths at runtime. I used a combination of a custom AssetBundleProvider and InternalIdTransformFunc.

In my custom AssetBundleProvider.Provide I retrieve the url & save it, and do any other prerequisite actions before the bundle can be downloaded (these were typically async calls that couldn’t be “waited until completion” in InternalIdTransformFunc). Then in InternalIdTransformFunc the saved url is returned instead of the default load path.