How to check the hash code of an Assetbundle to validate

Unity Engine
1

Hi guys.

The topic may not totally related to the Assetbundle Graph Tool, but it will still affect the people, so I hope we can get the answer here.

When building the assetbundles, the manifest will give out the computed hash128:

                    Hash128 myHashCode = aManifest.GetAssetBundleHash(aAssetbundleBuildArray[i].assetBundleName);
                    string myHashCodeString = myHashCode.ToString();

That’s nice and we can also use that hash value if it is already present in our download cache:

Caching.IsVersionCached(aWww.url, myHashCode)

But how are we supposed to get the hashcode from the assetbundle to validate? Afaik, the hash code is not present in any form in the assetbundle, but we have to get the hash code from outside (file downloaded from server is our solution).

That doesn’t help to validate that the downloaded data is correct though and here we come to my problem. Even with the hash code, I can’t validate that the data my Client has downloaded is actually the correct one.

Am I missing something here? Would it be possible that Unity adds the hash code into the Assetbundle when creating the bundles? Is there any other way to validate the data (I’ve not found any way yet, as WWW.bytes is not readable for me and the Assetbundle property is not serializable to do a MD5 check by myself).

Thanks 4 help,
Patrick

2

You want to use the CRC generated at build time, which can be found in the .manifest file for each asset bundle. Pass that CRC to WWW.LoadFromCacheOrDownload, or UnityWebRequest.GetAssetBundle. A CRC will be calculated for the downloaded bundle and compared to the given CRC parameter.

-Stephen

4 Likes
3

Ok, thanks, I’ll try that and report :slight_smile:

4

Curious if you had a chance to try this?

-Stephen

1 Like
5

@Reichert

Yes, I tried and it works, but not in the way I would like it to have. I thought about writing about it several times now, but I wanted to talk first to one of the Assetbundle System engineers if my implementation of our system is using the AB system wrong or if we missed something.

tl:dr
I assume the AB system suggests to have a “preloading mechanism” which loads and verifies all assetbundles, before the actual game starts, rather than check on the fly if an assetbundle is up2date or not (which is what we are doing at the moment).

long story:
We use the assetbundles to load battleground (base) and assets (custom additions) when the player is entering a mission, as well as for static assets (UI atlasses, music,…). While the former is only loaded when used, the later is loaded when the player starts the game.

Now the biggest problem we face is, that the player has to get from somewhere outside that an assetbundle has been updated. We use a small textfile (well, kind of small, but still 300 kb for us) which contains all data we need to tell the client that the assetbundle is updated: CRC, hash value and a size for our loading screen display. The progress of downloading and updating looks like:

  1. player starts game, downloads “assetbundle_version.cfg”.
  2. game parses config and knows now the latest and correct CRCs, hash values and sizes for the assetbundles.
  3. game gets on the preloading screen, checks and downloads static assetbundles (UI atlas, music,…) if needed.
  4. player can login into game and decides what he would like to do.
  5. player wants to play a mission, enters the mission and comes to a loading screen.
  6. game checks which assetbundles are actual needed for that mission and tries to download the assetbundles (battleground, models, textures, materials,…) if needed (client knows the CRCs and hashvalues from the start of the game).

Now in the last step, we face our problem: if we update our assetbundles on our ftp while the player is playing, he could potentially download assetbundles with a different CRC than the client expects: the client downloaded the configuration file only at start of the game and we updated the assetbundles after that.

Now to fix that, we could download the configration file each time we want to check if we need to download assetbundles, but that would mean to download 300 kb each time the player wants to enter a mission (which happens quite often and is not suitable for a mobile game). Maybe we can change our WWW.LoadFromCacheOrDownload to not check the CRC and hashvalues later, after the game started.

In my conclusion, the current main problem in the assetbundle system is, that the hashvalues and CRCs are given in from outside. In a perfect world, we would simply tell the game to download an assetbundle from our ftp and it checks from a small download header if that is already the data the game knows and if it is not corrupted. No passing in an external configuration file.

Well, now I wrote the story before talking to the pros, but I’ll post a follow up to this :wink:

Patrick

6

Thanks for the write-up! super useful to hear your use case. It sounds like you have a pretty sophisticated system that works well, apart from the issue you raise. To fix this I suggest you calculate and store a hash of the assetbundle_version.cfg file every time it changes in a separate file (assetbundle_version.cfg.hash) which would be small enough to download at mission start and compare to the previously saved value. If it’s different, only then do you need to download a new assetbundle_version.cfg file.

We’re in the planning phases for a possible asset bundle hosting service, which could easily support the kind of solution you are talking about by providing the CRC and hash in an HTTP HEAD request, for instance. Anyhow, good feedback, thanks.

-Stephen

7

Ok, talked to Ryan at the Unite Europe and he also suggested something like the small version file. It sounds like a solution we can try.

I’m very happy that you are already working on some solutions for the problem. I’ve to admit though, if that will be some kind of Unity Service, we most probably have to stay away from it (legal stuff etc). But I’m looking forward to it anyway :slight_smile:

Thanks again,
Patrick

8

@PAHeartBeat Your asset bundle issue is EXACTLY the same issue we ran in our company’s game. The only difference is that I store our asset bundles’ version/CRC hashes in a database, and I have a Python script return the version/CRC pair for that asset bundle for the specified platform. It’s tedious and annoying because I have to make a call to our REST API to get the version/crc pair before I load an asset bundle. Not only that, but I have to manually open up all of the manifest files, so that I can copy and paste them into the database until I write a tool to do that for me.

I create a UnityWebRequest object with UnityWebRequest.GetAssetBundle(), and call req.Send(). Then, if an updated asset bundle is found, all Unity does to let you know is throw an exception saying there’s a CRC mismatch as if it downloaded incorrectly. It’s nice that there’s a way to check if it downloaded correctly, but the problem is that this is the EXACT same mechanism needed to update the asset bundle too… I’m not checking for the AssetBundle’s integrity to see if it downloaded correctly in this context. I literally want to just download the latest version from the server, or pull from the cache. I haven’t found much documentation on what that “version” code is for, but it seems to be Unity’s version code for the asset bundle’s file format for forward compatibility in the future (from what I’ve gathered).

9

Hello guys. I want to ask it to you, because I haven’t found working examples on the internet. I’m also interested in AssetBundle’s hash which is stored in a file with the .manifest extension. I always have to check if there is an update available on the server (check if the file have changed). If I’m not mistaken, there is a class AssetBundleManifest and a method GetAssetBundleHash, but my attempts to use it result in failure. I cannot get AssetBundleMaifest.

I used this source as an example: Unity - Scripting API: Caching.
Despite using local loading in the example, I still cannot do it. I tried to load both the file with the .manifest extension and the AssetBundle file itself, what could be the problem?

May be you could show me a working example of getting AssetBundleManifest and calling GetAssetBundleHash.

10

@ViktorAcadem
You get the Hash from AssetBundleManifest.GetAssetBundleHash which you store in a file. You have to download the file later then, so your client will know which Hash values are the current ones on the server.

The AssetBundleManifest comes from your BuildPipeline.BuildAssetBundles call.

Pretty much straight forward on creation side.

On client side later, you use WWW.LoadFromCacheOrDownload and use the Hash value as parameter.

11

@pahe
WWW.LoadFromCacheOrDownload TO BE DEPRECATED (Use UnityWebRequest)

Following the recommendations I use UnityWebRequest, but the problem is not here.

The problem is that I don’t correctly use AssetBundleManifest.GetAssetBundleHash, it always leads to the error. I would like to see the applying of this method on the working example. As a result of calling BuildPipeline.BuildAssetBundles, we get two files, one with the .manifest extension, the other main one without the extension. Which one should be loaded to get hash using AssetBundleManifest.GetAssetBundleHash? The thing is that I need to track the updates on the server and, if necessary, load AssetBundle. For this purpose, I store not only the AssetBundle file on the server, but also a text file with the following data: crc, version, hash.

I read the saved file as follows:

public IEnumerator LoadManifest(string path)
   
{
       
   using (WWW www = new WWW(path))
       
   {
          
      yield return www;
         
      if (!string.IsNullOrEmpty(www.error))
          
      {
               
           errorMessage = www.error;
               
           yield break;
           
      }
           
      ReadFile (www.text);
       
    }
   
}
    

private void ReadFile(string text)
{
       
   string[] readtext = text.Split ("\ n"[0]);
       
   uint crc = uint.Parse(readtext [0]);
       
   int version = int.Parse(readtext [1]);
       
   string hash = readtext [2];
       
   Manifest = new AssetInfo (crc, version, hash);
   
}

Despite this solution, I would like to learn how to get hash correctly using AssetBundleManifest.GetAssetBundleHash. I would be very grateful if you would show by example how AssetBundleManifest.GetAssetBundleHash works, when downloading a new file from the server, or at least locally, because I didn’t find working examples.

12

Thanks for the info, didn’t know that. I’ll have a look into that. I guess that with 2017 the function is deprecated, as we’re on 5.6.x and I don’t get that warning. Anyway, it should work both ways.

13

Do you have a working example of using AssetBundleManifest.GetAssetBundleHash in order to get Hash? Could you, please, show this example to me? Thank you in advance.

14

Well, you mean something like this?

Hash128 aHashCode = aManifest.GetAssetBundleHash(aAssetbundleBuild.assetBundleName);
var aHashCodeString = aHashCode.ToString();
Debug.Log(aAssetbundleBuild.assetBundleName + " Hash: " + aHashCodeString + " CRC: " + aCrc + " filesize: " + aFileLength);
15

Yes, right, but I have a problem with lines earlier, when getting AssetBundleManifest (LoadAsset returns null).

AssetBundle manifestBundle = AssetBundle.LoadFromFile(manifestBundlePath);
AssetBundleManifest manifest = manifestBundle.LoadAsset<AssetBundleManifest>("AssetBundleManifest");

manifestBundlePath - is this a path to the file with a .manifest extension? There are some who points the way to the main file without an extension. Which file does manifestBundlePath point to? I tried different options, but it doesn’t work. I can’t understand how to get AssetBundleManifest correctly.
How did you get aManifest? Which file did you download?

16

Hm. Do you need the manifest from runtime or to editor time?

Editor time (when creating the assetbundles) will give you the manifest when creating the Assetbundle:

var aManifest = BuildPipeline.BuildAssetBundles(anAssetbundleOutputPath, aAssetbundleBuildArray, aBuildOptions, aBuildTarget);

For runtime, you have to download the manifest file and unpack it:

assetBundleManifest = aDownload.assetBundle.LoadAsset<AssetBundleManifest>("AssetBundleManifest");

Hope this helps. If not, please add some more info at which point (editor/runtime) you need the manifest file.

17

I’m interested in getting the manifest file at run time by downloading the file from the server. But I can’t even download it from the device’s memory. Here is an example of how my logic works:

As a result of the BuildPipeline.BuildAssetBundles work, I got the main file (filename) and the manifest file (filename.manifest). Then I move the received files to the project folder (StreamingAssets) and try to download them using the following lines:

string path = Path.Combine(Application.streamingAssetsPath, "filename.manifest");
   
AssetBundle manifestBundle = AssetBundle.LoadFromFile(path);
   
AssetBundleManifest manifest = manifestBundle.LoadAsset<AssetBundleManifest>("AssetBundleManifest");

Hash128 hash = manifest.GetAssetBundleHash(«bundleName»);

When I try to download “filename”, I have an error in the 3rd line (manifest = null)

When I try to download “filename.manifest”, I have an error “Unable to read header from archive file”

Where is the error?

How to download correctly a manifest file?

18

“filename.manifest” is no assetbundle, but the manifest file in text format. You need to download “filename”, which is an assetbundle which contains the manifest file as AssetBundleManifest.

So, although you get an error when downloading “filename”, that would be the correct way. I’m not using AssetBundle.LoadFromFile, but that should still work the same way. I’d suggest you debug that assetbundle by manifestBundle.LoadAllAssets() and give out what name and type is in it. Everything else looks fine in my opinion.

Edit: if you can’t get it to work, I’ll try to find some time at the weekend to create a showcase project for it.

19

Using the function manifestBundle.LoadAllAssets (), we get an array that contains only compressed GameObject. There is no AssetBundleManifest class in this array.

Here is an example of viewing all the downloaded resources:

string path = Path.Combine(Application.streamingAssetsPath, "filename");
AssetBundle manifestBundle = AssetBundle.LoadFromFile(path);
object[] objects = manifestBundle.LoadAllAssets();
foreach(var obj in objects){
     Debug.Log(obj.GetType().ToString());
}

The contents of the console:

UnityEngine.GameObject
UnityEngine.Debug:Log(Object)

Maybe I need to use special settings for AssetBundle build?

I build using the following lines:

BuildPipeline.BuildAssetBundles (path, BuildAssetBundleOptions.None, EditorUserBuildSettings.activeBuildTarget);
20

Alright, I’ve created a small test project now, to make sure that I’m not talking about project specific stuff :slight_smile:

Testproject
My project is very simply, only containing a Material, and empty folder called “Assetbundles” and an editor script to build the assetbundles. The editor script is under the Editor folder and looks like:

public class abCreator
{
    [MenuItem("Assetbundles/Create Assetbundles")]
    public static void createAssetbundles () {
       
        BuildPipeline.BuildAssetBundles("Assetbundles", BuildAssetBundleOptions.None, BuildTarget.StandaloneWindows64);
    }
}

The material is created empty and I setup the assetbundle settings for it via inspector (in our project we do that via code, but it works the same way).

3247432--249864--assetbundle.png

Now, when I use the editor script to create the assetbundles, I get the following files:

3247432--249866--assetlist.png

mymats is the assetbundle with my material.
mymats.mats (mymats.mats.manifest is the real name, .manifest is only removed in the project view) is the manifest file in text format for my assetbundle.
The first Assetbundles is the assetbundle with the manifest for my assetbundles which are contained in the output folder. (This is the assetbundle you need to download later, to get the manifest from!)
The second Assetbundles is the manifest file (again, .manifest is not shown in the project view) in text format, so you can see what is inside (afaik only for you to see what is actually inside later and if you want to check if GUIDs are set correctly, dependecies are set correctly, …).

What do you need in the end?
You only need the two assetbundles. The Assetbundles assetbundle contains the manifest you need later. The mymats assetbundle contains your material.
The other two files (both manifest files in text format) which were created are AFAIK solely for debugging, if you want to check what is inside and if everything is correct referenced or you just want to see how it looks. (Please correct me if I’m wrong here).

Hope that helps :slight_smile: