Hi all,
I am trying to work out a system for downloading when the game is loaded up. I am currently using UnityWebRequest.GetAssetBundle (I am using 2017.4) to cache the assetbundle in the system. I have an asset manager that tracks everything loaded, but I’m wondering which way would be better to handle the bundles. Am I right in thinking I cannot use AssetBundle.LoadFromFile if I have downloaded it from the cache?
Our current process is that the app gets the bundle crc from the server which replaces the bundle in the cache if so with the UnityWebRequest.GetAssetBundle. It doesn’t store the bundle anywhere, it just caches it.
How is the best way to get the bundles after this? E.g. I want to load a building model, I will want to load the building bundle from the cache, get the asset, store it in the manager then unload the bundle. Would it be best just to use the web request each time?
Would anyone recommend me doing it a different way or is this a reasonable approach?
One I thing I suggest you to do is to figure out what you need exactly and see if you aren’t reinventing the wheel.
UnityWebRequest.GetAssetBundle() (the overloads with CRC/version arguments) is loading AssetBundle from cache, if given AssetBundle with given crc/version/hash is in the cache, otherwise it downloads and caches it for future use. As such you just do UnityWebRequest.GetAssetBundle() and then take the bundle via DownloadHandlerAssetBundle.GetContent(), this way you don’t need to bother, whether tahe bundle was downloaded or loaded from the cache.
If you are looking into using AssetBundle.LoadFromFile, the you shouldn’t use UnityWebRequest.GetAssetBundle, you need to assign DownloadHandlerFile instead to bypass AssetBundle system at download time. But by doing so you also lose the builtin caching etc. and reinvent them yourself. So it sounds you should first make sure the builtin caching/loading is not enough for you.
Ah okay, so the only real difference would be I would need to create a system for caching it manually? I guess the UWR method would be much better for me then. In terms or memory usage is there much in it between UWR and LoadFromFile?
UWR caching stores AssetBundle on disk decompressed, so the load will be faster. In terms of memory usage there should be no significant difference between the two, only that in case of LoadFromFile you’d have to download and store the bundle first and ensure efficient memory usage during download (such as DownloadHandlerFile).
@Aurimas-Cernius What about if the app is offline with no internet - UnityWebRequest would fail because there is no internet, but will it still download from the local cache if it exists? having a problem with it when offline on Android.
If AssetBundle is cached, UnityWebRequest should loading without even attempting to download anything. It should only go to internet if the bundle is not cached or if cached bundle is of different version/hash/crc than requested.
Hi Aurimas,
We implemented the assetbundle download via a custom DownloadHandlerScript - this works fine aside from being a bit slow.
All our assetbundles are LZ4 which is fine once on disk but we’d like to take advantage of the extra compression for download and decompress as it is downloading rather than at the end.
The options we see are:
Gzip the LZ4 and decompress ourselves - works but is clunky, bit slow compression still not as good as LZMA + compression occurs after entire download.
Build assetbundles as LZMA and use the GetAssetBundle to decompress straight to disk
is a problem as there is no way to pipe the result to a particular file/folder
Is there anyway to get the benefits of Unity AssetBundle decompression and output the file somewhere?
Hello, I don’t understant one thing. How UnityWebRequest understands if bundle is cached or not? If I use version parameter, does it just use Caching.currentCacheForWriting’s version and compares it to the one I specified in UnityWebRequestAssetBundle.GetAssetBundle() or what? For example if I have two bundles - bundle1 and bundle2. After first laucng they were downloaded and cached in AssetBundlesCache/bundle1, AssetBundlesCache/bundle2 acordingly. Now I restart my game with no internet connection. Usually I try to load bundle on server response. It sends me uri and version. I don’t have connection, so can I just pass empty uri and version of the current cache to UnityWebRequestAssetBundle.GetAssetBundle() and specify currentCacheForWriting accordingly for each bundle? Also I don’t know how to get version from the cache. Caching.GetVersionFromCache() is obsolete.
Also I checked if UnityWebRequestAssetBundle.GetAssetBundle() throws an exception with no Internet. It doesn’t. But the problem is DownloadHandlerAssetBundle.GetContent() throws InvalidOperationException: Cannot resolve destination host. So I can’t load bundle from cache.
Another thing I tried was to directly assign downloadHandler property of my UWR with DownloadHandlerAssetBundle object. This time there was no error but assetBundle itself was null. Why is it so hard to just load bundle from cache?! Looks like I’d have to use AssetBundle.LoadFromFileAsync() for this.
But why do you want to save bundles on disk? When you use GetAssetBundle() with additional parameters, it will use caching system and will avoid downloading an already downloaded bundle.
As for additional compression, first of all make sure it does reduce the size significantly. If using GZip does benefit you, first thing to do is to check the request headers and configure server to use gzip if cleant accepts it. The gzip compression will work out of the box if device has builtin support for it.
Cached asset bundles are identified by name, version, crc and hash. For UnityWebRequest name is taken from url and you have to pass at least one of other three for the cache to be used. If you pass only url, it will always download bundle from the internet. Passing the same url and the same crc/hash/version like previously should load asset bundle from cache (if it is cached and you haven’t cleared it) and this should work without internet connection.
Oh, OK. So in case I get URL from the server I have to save it somewhere to be able to pass it even without connection. I decided to load it frrom cache folder with the use of LoadFromFile functionality.
Hi Aurimas, we wrote our own caching system based on prior experiences with earlier versions of Unity caching - our understanding was that there is no guarantee the cache is available or enough space is available to cache, the side result is it loads into memory instead or ejects out other assetbundles in the cache? Is that still the case? We do all our checks up front and ensure we have space and decompress into our persistentDataFolder so we put up disk space requirements etc etc - it all works fine.
Unless things have changed then we could switch if you feel there are no issues now.
As for gzip and header re: yes we already did this, we were just wondering if it was possible to just utilize the decompression from LZMA → LZ4 and stream that into our location that was all.
Ok, so that all sounds good and well, but it does not actually play out that way.
In my case I want the bundles to update when they reach the server if the server file has changed, good so far.
Then if it doesn’t reach the server I want them to use the locally cached files, so I need to use the matching hash from the last downloaded version. Accept the manifest files with those hashes come from the server and don’t make it into cache, and the AssetBundleManifest file does not even seem to get a hash.
I tried taking a step back and tracking an internal version number that I update whenever I get a positive read from the server, but the version number based UnityWebRequest.GetAssetBundle does not seem to pull correctly from the cache. Even checking the cache manually for version numbers seems to have been marked obsolete. There also does not seem to be any methodology for looking up the bundles in cache manually and pulling them out directly.
I have found a somewhat Frankenstein solution, but while it works, it has a lot of potential for failure cases. Until there is a built in Unity solution, here you go.
AssetBundle.manifest doesn’t contain a hash, but it does contain a CRC value, so if you download a manifest file that has a CRC, but no hash, you know you have a good line on a fresh version of the AssetBundle file. Clear all cached versions of the assetBundle file and then download a new one from the same source.
Unity won’t cache the bundle unless you give it a version number or a version hash, so you have to make one up when asking for it. At the same time, you need to know what version it was next time in case you can’t hit the server. This is why I clear the cache when I am about to get a fresh version and use 0 as the version number. This means that the assetbundle file always has to be downloaded, even if it is current, but as far as I can tell there is not a way to find out what the hash should be for it. Oh, did I mention that version numbers do not update in the manifest files either?
Anyway, after I’ve gotten a CRC and a hash, just a CRC, or just nothing at all, I now can try to download the actual bundle file. This logic could probably be simplified if I handled the main assetbundle file through a different flow, but they are largely the same logic outside of this point.
UnityWebRequest download;
if( config.hash == default(Hash128) )
{
if( config.crc == 0 )
{
// file not found
List<Hash128> listing = new List<Hash128>();
Caching.GetCachedVersions( config.name,listing );
if( listing.Count > 0 )
{
// No manifest, but previous versions were cached
download = UnityWebRequest.GetAssetBundle( Path.Combine( config.path, config.name ), listing[listing.Count-1], config.crc );
Log.LogL( "{0} did not have a hash key, attempting to use most recent version {1}",
config.name,
listing[listing.Count - 1] );
}
else
{
// No manifest, no cache, trying to get the bundle anyway just in case
download = UnityWebRequest.GetAssetBundle( Path.Combine( config.path, config.name ), config.crc );
Log.LogL( "{0} did not have a hash key and no previous versions found", config.name );
}
}
else
{
// was manifest bundle
download = UnityWebRequest.GetAssetBundle( Path.Combine( config.path, config.name ),
0,
config.crc );
}
}
else
{
// have everything we need, just grab the file
download = UnityWebRequest.GetAssetBundle( Path.Combine( config.path, config.name ), config.hash, config.crc );
Log.LogL( "{0} is cached : {1}", config.name, Caching.IsVersionCached( download.url, config.hash ) );
}
download.SendWebRequest();
Sorry that this bit is such a mess, I just got it working really. I’m going to walk through it backwards because that is actually more the order of case regularity.
The last else is processing a case where you have a good hash. This means it is a normal asset bundle and we got the manifest file alright. Basically, this is the call just working as intended.
Next to last case we have no hash, but we do have a CRC. This is a properly loaded manifest file for a AssetBundleManifest bundle. Here we set the version number to zero and check the CRC. Still mostly normal.
Above that we have no CRC or hash, plus we didn’t have a version in cache. This is the worst case scenario and there isn’t really anything Unity can do to help you here. I try downloading the file anyway, just for giggles. In truth, I have logic elsewhere that if the download fails tries to pull it from streaming assets instead, but that’s not really relevant here.
Finally we start getting to the workaround, sorry about that. I pull down a list of all cached version hashes for this bundle. We can’t get a list of the actual bundles, the version numbers, or the bundles themselves, but we can get this list. If there are items in the list, we have cached versions to work with. This list is in order from oldest downloaded to newest, so I grab the last hash in the list and use it as the download hash, effectively loading that version from cache. As a note, it looks like I pass it a CRC, but it’s really zero so I’m basically just ignoring it.
This all works, but it still has a gaping hole. Lets say you have character sprites in an asset bundle, and you want to have them wear Santa hats for a Christmas event. You upload a new asset bundle for Christmas, and then afterwords you switch back to the old bundle. Here in we have a problem. the list is in order of oldest to newest DOWNLOADED, not used. This means that the Christmas event bundle is the last in the list. So any user who played before and through Christmas will have the Christmas event bundle when they are offline even after the event has ended and they’ve received a more recent bundle.
You can work around this by manually clearing items out of the list when you get new ones, forcing the order of cached items, or a few other methods, but honestly? It’s a mess.
The point is, this feature doesn’t quite let you do what it says on the box, and the unity API documentation on it is both out of date AND marks it as obsolete. Hopefully this helps someone else to not spend their whole weekend combing through documentation and forums like I did.
Thanks for showing all that you did. This is what I am having to use because Addressable Assets are not mature enough to do this type of thing as I’ve found and been told.
Could you help me understand a bit about how you are implementing this? What the code outside of this snippet looks like?
For example - what the config looks like, and the manifest - and how these things are declared and called?
It seems like this is set up to just download one asset bundle, but do you have this in a loop to grab many which are declared in the AssetBundle.manifest?
I am so sorry, I didn’t see this for a while. You likely aren’t looking anymore, but for anyone who is I can post this. It is probably not the best solution, but it is the one I’ve been using.
/// <summary>
/// Facilitates loading, caching, downloading, and managing AssetBundles
/// </summary>
public static class AssetBundleManager
{
/// <summary>
/// check this value once a <see cref="DownloadAllWithManifest"/> has been started to see how close it is to finishing
/// </summary>
public static float Progress => m_total == 0 ? -1 : (float)m_done / m_total;
/// <summary>
/// check this value once a <see cref="DownloadAllWithManifest"/> has been started to see if it has finished
/// </summary>
public static bool IsDone => m_total > 0 && m_done == m_total;
private static readonly Dictionary<string,AssetConfig> m_configs = new Dictionary<string, AssetConfig>();
private static int m_done;
private static int m_total;
/// <summary>
/// Retrieves and <see cref="AssetBundle"/> that has already been loaded into memory
/// </summary>
/// <param name="bundleName">name of the <see cref="AssetBundle"/></param>
/// <returns>null if the bundle is not currently loaded into memory</returns>
public static AssetBundle GetBundle( string bundleName)
{
if( !m_configs.ContainsKey( bundleName ) )
{
Log.Warn( "Bundle {0} not loading", bundleName );
return null;
}
AssetConfig config = m_configs[bundleName];
if( config.failed )
{
Log.Warn( "Bundle {0} Failed to load", bundleName );
return null;
}
if( !config.succeeded )
{
Log.Warn( "Bundle {0} is still loading", bundleName );
return null;
}
if( !config.isLoaded )
{
Log.Warn( "Bundle {0} has been unloaded", bundleName );
return null;
}
return config.bundle;
}
/// <summary>
/// Tries to get an asset from a given <see cref="AssetBundle"/>
/// </summary>
/// <returns>null if not found</returns>
public static T GetAsset<T>( this AssetBundle bundle, string assetName ) where T : UnityEngine.Object
{
if( bundle == null )
{
Log.Warn( "Bundle is null" );
return null;
}
if( !bundle.Contains( assetName ) )
{
Log.Warn( "Bundle {0} does not contain asset {1}", bundle.name, assetName );
return null;
}
return bundle.LoadAsset<T>( assetName );
}
/// <summary>
/// Tries to get an asset from the named <see cref="AssetBundle"/> if it is loaded into memory
/// </summary>
/// <returns>null if not found</returns>
public static T GetAsset<T>( string bundleName, string assetName ) where T : UnityEngine.Object
{
AssetBundle bundle = GetBundle( bundleName );
return bundle.GetAsset<T>( assetName );
}
/// <summary>
/// Unloads the bundle from memory and marks it as having been unloaded
/// </summary>
/// <param name="unloadAllLoadedObjects">should references to data in this bundle be cleared when it unloads</param>
public static void UnloadAssetBundle(string bundleName, bool unloadAllLoadedObjects = true)
{
AssetBundle bundle = GetBundle(bundleName);
if( bundle == null ) return;
m_configs[bundleName].isLoaded = false;
bundle.Unload( unloadAllLoadedObjects );
}
/// <summary>
/// Get all bundle names and their current loaded state
/// </summary>
public static Dictionary<string, bool> GetBundleActivity()
{
Dictionary<string,bool> bundles = new Dictionary<string, bool>();
foreach( AssetConfig config in m_configs.Values )
{
bundles[config.name] = config.isLoaded;
}
return bundles;
}
/// <summary>
/// Quick loads an unloaded <see cref="AssetBundle"/> back into memory. also can be used to try and restart a failed load.
/// </summary>
/// <param name="bundleName">name of the bundle including variant</param>
/// <param name="onComplete">called if the load succeeds</param>
/// <param name="onFailure">called if the load fails</param>
public static void ReloadAssetBundle(string bundleName, Action<AssetBundle> onComplete = null, Action onFailure = null )
{
if( !m_configs.ContainsKey( bundleName ) )
{
onFailure?.Invoke();
return;
}
AssetConfig config = m_configs[bundleName];
if( config.isLoaded )
{
Log.LogL( "Bundle {0} Already Loaded", bundleName );
onComplete?.Invoke( config.bundle );
return;
}
config.onComplete += onComplete;
config.onFailure += onFailure;
config.failed = false;
config.succeeded = false;
if( File.Exists( Path.Combine( config.path, bundleName ) ) )
{
Log.LogL( "Bundle {0} is local", bundleName );
LoadFromFile( config );
return;
}
Log.LogL( "Downloading bundle {0}", bundleName );
Tool.StartCoroutine( Download( config ) );
}
/// <summary>
/// Tries to load an individual <see cref="AssetBundle"/> into memory. If the bundle needs to be downloaded to cache first it does this as well.
/// Will try to load a manifest file first and validate the data if it finds one.
/// </summary>
/// <param name="path">URL where the bundle is located</param>
/// <param name="bundleName">name of the bundle including variant</param>
/// <param name="onComplete">called if the load succeeds</param>
/// <param name="onFailure">called if the load fails</param>
/// <param name="restartIfFailed">if this bundle has already started loading, but failed, should it try again now?</param>
public static void LoadAssetBundle(string path, string bundleName, Action<AssetBundle> onComplete = null, Action onFailure = null, bool restartIfFailed = false)
{
if( m_configs.ContainsKey( bundleName ) )
{
if( onComplete == null ) return;
if( m_configs[bundleName].isLoaded )
{
Log.LogL( "Bundle {0} Already Loaded", bundleName );
onComplete.Invoke( m_configs[bundleName].bundle );
return;
}
if( m_configs[bundleName].failed )
{
if( !restartIfFailed )
{
Log.LogL( "Bundle {0} Already Failed", bundleName );
onFailure?.Invoke();
return;
}
}
else
{
Log.LogL( "Bundle {0} Still Loading", bundleName );
m_configs[bundleName].onComplete += onComplete;
m_configs[bundleName].onFailure += onFailure;
return;
}
}
AssetConfig config = new AssetConfig
{
path = path,
name = bundleName,
onComplete = onComplete,
onFailure = onFailure
};
m_configs[bundleName] = config;
if( File.Exists( Path.Combine( path, bundleName ) ) )
{
Log.LogL( "Bundle {0} is local", bundleName );
LoadFromFile( config );
return;
}
Log.LogL( "Downloading bundle {0}", bundleName );
Tool.StartCoroutine( Download( config ) );
}
/// <summary>
/// Attempts to download the AssetBundleManifest from the location and if it succeeds it does the same for all bundles listed and caches them.
/// If the a bundle's hash(based on its contents) has not changed since it was last cached then it will use the cached version instead of re-downloading.
/// </summary>
/// <param name="url">URL to find the assetBundles for download(if null will skip this step and try falling back to streaming assets)</param>
/// <param name="streamingAssetsSubFolder">Subfolder in streaming assets to load assetBundles from if they were not successfully downloaded from the server. Skipped if null or empty</param>
public static void DownloadAllWithManifest( string url, string streamingAssetsSubFolder = null )
{
m_total = 1;
m_done = 0;
Action fallBack = OnFailureEach;
if( !string.IsNullOrEmpty( streamingAssetsSubFolder ) )
{
fallBack = () => { CheckStreamingAssets( streamingAssetsSubFolder ); };
}
if( string.IsNullOrEmpty( url ) )
{
fallBack.Invoke();
}
else
{
LoadAssetBundle( url, "AssetBundles", OnCompleteManifest, fallBack );
}
}
private static void CheckStreamingAssets(string url)
{
m_configs.Clear();
m_total = 1;
m_done = 0;
LoadAssetBundle( Path.Combine( Application.streamingAssetsPath, url ),
"AssetBundles",
OnCompleteManifest,
OnFailureEach );
}
private static void OnCompleteManifest( AssetBundle bundle )
{
AssetBundleManifest manifest = bundle.LoadAsset<AssetBundleManifest>( "AssetBundleManifest" );
string[] bundles = manifest.GetAllAssetBundles();
m_total += bundles.Length;
m_done++;
string path = m_configs["AssetBundles"].path;
string output = "bundles :";
foreach( string child in bundles )
{
LoadAssetBundle( path, child, OnCompleteEach, OnFailureEach);
output += " '" + child + "'";
}
Log.LogL( output);
}
private static void OnCompleteEach( AssetBundle bundle )
{
m_done++;
}
private static void OnFailureEach()
{
m_done++;
}
private static IEnumerator Download(AssetConfig config)
{
UnityWebRequest manifest = UnityWebRequest.Get (Path.Combine( config.path, config.name )+ ".manifest");
yield return manifest.SendWebRequest ();
if( !string.IsNullOrEmpty( manifest.error ) )
{
Log.Warn( "Failed to download manifest file for {0} from URL '{2}' :\n\t{1}",
config.name,
manifest.error,
manifest.url );
}
else
{
string[] lines = manifest.downloadHandler.text.Split('\n');
if( lines[1].Contains( "CRC:" ) )
{
Log.LogL( "Bundle {1} CRC: {0}",lines[1].Substring( 5, lines[1].Length - 5 ), config.name );
uint.TryParse( lines[1].Substring( 5, lines[1].Length - 5 ), out config.crc );
}
config.hash = default(Hash128);
if( lines[5].Contains( "Hash:" ) )
{
Log.LogL( "Bundle {1} Hash: {0}",lines[5].Split (':') [1].Trim (), config.name );
config.hash = Hash128.Parse(lines[5].Split (':') [1].Trim ());
}
else if(config.crc != 0)
{
// new manifest bundle found
Caching.ClearAllCachedVersions( config.name );
}
}
UnityWebRequest download;
if( config.hash == default(Hash128) )
{
if( config.crc == 0 )
{
// Manifest file not found
List<Hash128> listing = new List<Hash128>();
Caching.GetCachedVersions( config.name,listing );
if( listing.Count > 0 )
{
download = UnityWebRequestAssetBundle.GetAssetBundle( Path.Combine( config.path, config.name ), listing[listing.Count-1], config.crc );
Log.LogL( "{0} did not have a hash key, attempting to use most recent version {1}",
config.name,
listing[listing.Count - 1] );
}
else
{
download = UnityWebRequestAssetBundle.GetAssetBundle( Path.Combine( config.path, config.name ) );
Log.LogL( "{0} did not have a hash key and no previous versions found", config.name );
}
}
else
{
// was manifest bundle
download = UnityWebRequestAssetBundle.GetAssetBundle( Path.Combine( config.path, config.name ),
0,
config.crc );
}
}
else
{
// proper and standard loading of file
download = UnityWebRequestAssetBundle.GetAssetBundle( Path.Combine( config.path, config.name ), config.hash, config.crc );
Log.LogL( "{0} is cached : {1}", config.name, Caching.IsVersionCached( download.url, config.hash ) );
}
download.SendWebRequest();
while( !download.isDone )
{
config.progress = download.downloadProgress;
Log.LogL( "Bundle {0} progress {1}", config.name, config.progress );
yield return null;
}
if( !string.IsNullOrEmpty( download.error ) )
{
Log.Error( "Failed to download bundle {0} from URL '{2}' :\n\t{1}",
config.name,
download.error,
download.url );
config.failed = true;
config.onFailure?.Invoke();
config.onComplete = null;
config.onFailure = null;
yield break;
}
Log.LogL( "Bundle {0} downloaded", config.name );
config.bundle = DownloadHandlerAssetBundle.GetContent(download);
config.succeeded = true;
config.isLoaded = true;
Log.LogL( "Bundle {0} loaded", config.name );
config.onComplete?.Invoke( config.bundle );
config.onComplete = null;
config.onFailure = null;
List<Hash128> cacheListing = new List<Hash128>();
Caching.GetCachedVersions( config.name,cacheListing );
if( cacheListing.Count > 1 )
{
Caching.ClearCachedVersion( config.name, cacheListing[0] );
}
}
private static void LoadFromFile(AssetConfig config)
{
if( !File.Exists( Path.Combine( config.path, config.name ) ) )
{
Log.Error( "Bundle {0} was not local or failed to download", config.name );
config.failed = true;
config.onFailure?.Invoke();
config.onComplete = null;
config.onFailure = null;
return;
}
config.bundle = AssetBundle.LoadFromFile( Path.Combine(config.path,config.name) );
config.succeeded = true;
config.isLoaded = true;
Log.LogL( "Bundle {0} loaded", config.name );
config.onComplete?.Invoke( config.bundle );
config.onComplete = null;
config.onFailure = null;
}
}
public class AssetConfig
{
public string name;
public string path;
public uint crc;
public Hash128 hash;
public bool succeeded;
public bool failed;
public bool isLoaded;
public float progress;
public AssetBundle bundle;
public Action<AssetBundle> onComplete;
public Action onFailure;
}