Asset Bundle Dependencies

Hello,

I’m currently working on a modding system for my game and I’m having a little bit of trouble. I’m trying to reduce the memory usage by making use of dependencies within asset bundles, but the method that I’m trying to utilize doesn’t seem to work. In the documentation page (http://docs.unity3d.com/Manual/managingassetdependencies.html) it shows a method of building shader dependencies using a temp object, and I’m wondering if there’s a way to do the same for other types of objects. The code I’m using is:

// iterate over all of the objects
GameObject goObject = null;
foreach (GameObject go in gameobjects)
{
    // don't allow standard assets to be added
    // we do this because they are already part of Unity
    if (go.IsStandardAsset())
        continue;

    // try to get the prefab
    goObject = (GameObject)PrefabUtility.GetPrefabParent(go);
    // if no prefab was found, assume that this is either not a prefab, or is already a prefab
    if (goObject == null)
        goObject = go;

    // get all of the mesh filters and renderers
    SkinnedMeshRenderer[] skinrenderers = goObject.transform.GetInactiveComponentsInObjectAndChildren<SkinnedMeshRenderer>();
    MeshRenderer[] renderers = goObject.transform.GetInactiveComponentsInObjectAndChildren<MeshRenderer>();
    MeshFilter[] filters = goObject.transform.GetInactiveComponentsInObjectAndChildren<MeshFilter>();

    // iterate over all of each type and add the proper sub-objects to the correct assets
    foreach (SkinnedMeshRenderer skinrenderer in skinrenderers)
    {
        foreach (Material material in skinrenderer.sharedMaterials)
        {
            shaders.AddAsset(material.shader);
            materials.AddAsset(material);
        }
    }
    foreach (MeshRenderer renderer in renderers)
    {
        foreach (Material material in renderer.sharedMaterials)
        {
            shaders.AddAsset(material.shader);
            materials.AddAsset(material);
        }
    }
    foreach (MeshFilter mesh in filters)
        meshes.AddAsset(mesh);
}

The object I am attempting to serialize them to is:

#if GAME_TOOLS

using System;
using UnityEngine;
using System.Collections.Generic;
using Game.Modding.Shared;

#if UNITY_EDITOR
using UnityEditor;
using Game.Modding.Editor;
#endif

namespace Game.Modding.Shared
{
    public class ModPrerequisites : ScriptableObject
    {
        [SerializeField]
        private List<UnityEngine.Object> m_lAssets = new List<UnityEngine.Object>();

        public IEnumerable<UnityEngine.Object> Assets { get { return m_lAssets; } }

#if UNITY_EDITOR

        public void AddAsset(UnityEngine.Object scene)
        {
            // don't allow standard assets to be added
            if (scene.IsStandardAsset())
                return;

            // add it
            m_lAssets.Add(scene);

            EditorUtility.SetDirty(this);
            AssetDatabase.SaveAssets();
        }

        public void RemoveAsset(UnityEngine.Object scene)
        {
            m_lAssets.Remove(scene);

            EditorUtility.SetDirty(this);
            AssetDatabase.SaveAssets();
        }

#endif
    }
}

#endif

Unfortunately, EditorUtility.SetDirty(); returns an exception on the object (the ModPrerequisite object, not the object that is being added to it), stating that it has been destroyed. I have checked thoroughly, and neither the object itself nor any of the components listed inside are null, and the asset I’m working on shows up fine in the scene view. My first thought was that this is impossible, but attempting to put the prefab gameobject itself into my object caused the same exception. Of note, I AM able to drag/drop these assets into the list in the inspector for the asset.

To further assist, the end goal is to create an asset bundle dependency chain that looks like:
Shaders > Materials > Meshes > Mod Asset. The largest reason for this is that one of the mod assets are world modules, complete with artwork and lighting. I’m making use of scenes as prefabs for these, that way I can load the lightmaps additively and make use of GI with the world module. I’d like to set aside the materials and meshes that way I don’t have to include them in every single bundle. (It will also allow for artless variants in the case of distributing a dedicated server!)

Any thoughts?

Bump?

Have you tried calling SaveAssets after completing all edits on the object, instead of after adding each individual asset to the list?

(Not sure if that will fix it but it seems possible to me)

Thanks for the reply!

Unfortunately, that still causes the exception :frowning: Out of curiosity, how does dependency generation even work with the new AssetBundle system in 5.X? (Using the BuildPipeline.BuildAssetBundles() api instead of the messy 4.X api) Right now I’m hoping it will work as I’d imagine (build my array of bundles, and maybe early bundles build dependency chains to later bundles?)

Hmm. In that case I think we need to see how you are populating ‘shaders’, ‘materials’ and ‘meshes’ in your first snippet.

The basic rules, at least as far as the automatic bundle asset gathering goes, are:

  • Any asset that has been explicitly marked as being packed into a particular bundle will always be packed into that bundle.
  • Any asset which is being packed into a bundle will cause assets that it references (i.e. depends on) to be packed into the bundle as well, EXCEPT when the asset it references is marked to be packed into a different bundle.
  • If an asset being packed into bundle A depends on an asset which is marked as being in bundle B, then bundle A has a dependency on bundle B, and this will be recorded in the manifest.

Crucially, this means that an asset which is not marked as being in any specific bundle, but is referenced by multiple other assets that are marked as being in different bundles, then the asset will be duplicated into each bundle as it is packed alongside the explicitly-marked assets. We’re working on tooling to help you identify when this is happening so that you know when data is being duplicated.

I don’t quite understand the bundle setup you’re trying to achieve so I can’t say whether it’s going to work or not.

Then I guess it’s time to go into a little more detail :smile:

With this game, I’m trying to develop as robust a mod system as possible. Essentially, every mod “asset” is a Unity scene file, which gets piled into an asset bundle, and then loaded at run-time. In my initial design, I was simply using prefabs, but after reviewing how that made the environment modules look, I decided that I wanted to use streamed scenes, that way I could make use of GI. A side-effect of this however, is that there is a lot of material and mesh re-use, when populating environment modules. My thought was that I could potentially package these assets into a “prerequisite” bundle, allowing them to be used interchangeably in the scene.

My prerequisite class:

public class ModPrerequisites : ScriptableObject
{
    [SerializeField]
    private List<UnityEngine.Object> m_lAssets = new List<UnityEngine.Object>();

    public IEnumerable<UnityEngine.Object> Assets { get { return m_lAssets; } }

    public int Count { get { return m_lAssets.Count; } }

#if UNITY_EDITOR

    public void AddAsset(UnityEngine.Object scene)
    {
        if (scene == null)
            return;

        // don't allow standard assets to be added
        if (scene.IsStandardAsset())
            return;

        // add it
        m_lAssets.Add(scene);


    }

    public void RemoveAsset(UnityEngine.Object scene)
    {
        m_lAssets.Remove(scene);

        EditorUtility.SetDirty(this);
        AssetDatabase.SaveAssets();
    }

#endif
}

Which is used here:

private static bool ProcessScene(string szAssetName, ModPrerequisites shaders, ModPrerequisites materials, ModPrerequisites meshes, ModPrerequisites luascripts)
{
    // get a list of every game object in the scene
    GameObject[] gameobjects = Resources.FindObjectsOfTypeAll<GameObject>();

    // iterate over all of the objects
    GameObject goObject = null;
    foreach (GameObject go in gameobjects)
    {
        // don't allow standard assets to be added
        // we do this because including standard assets in the serialization will break it
        if (go.IsStandardAsset())
            continue;

        // try to get the prefab
        goObject = (GameObject)PrefabUtility.GetPrefabParent(go);
        // if no prefab was found, assume that this is either not a prefab, or is already a prefab
        if (goObject == null)
            goObject = go;

        // if there is no parent object to this, we have a slight concern (some assets are considered 'excluded' because they are simply part of Unity and not really part of the scene)
        if (goObject.transform.parent == null)
        {
            // find the mod controller
            ModAssetController controller = goObject.GetComponentInChildren<ModAssetController>();
            if (controller == null)
            {
                EditorUtility.DisplayDialog("Mod Build Failed", "Mod assets only allow for one top-level parent in the scene hierarchy! Please make sure everything is a child to the object with the controller script!", "Ok");
                return false;
            }
        }

        // get every single component in the scene
        Component[] comps = goObject.transform.GetInactiveComponentsInObjectAndChildren<Component>();

        // iterate over all the components
        bool bHasController = false;
        foreach (Component component in comps)
        {
            // make sure it is an allowed type
            if (!s_lAllowedComponentTypes.Contains(component.GetType()))
            {
                EditorUtility.DisplayDialog("Mod Build Failed", string.Format("Scene {0} contains the disallowed component {1} on object {2}!", szAssetName, component.GetType(), component.gameObject.name), "Ok");
                return false;
            }

            // perform special handling for certain types
            if (component.GetType() == typeof(SkinnedMeshRenderer) || component.GetType() == typeof(MeshRenderer) && shaders != null && materials != null)
            {
                // get the renderer
                Renderer r = (Renderer)component;

                // add the materials and stuff
                foreach (Material material in r.sharedMaterials)
                {
                    shaders.AddAsset(material.shader);
                    materials.AddAsset(material);
                }
            }
            else if (component.GetType() == typeof(MeshFilter) && meshes != null)
                meshes.AddAsset(component);
            else if (component.GetType() == typeof(ModAssetController) || component.GetType() == typeof(PropModAssetController))
            {
                // make sure there is only a single controller
                if (bHasController)
                {
                    EditorUtility.DisplayDialog("Mod Build Failed", string.Format("Scene {0} contains more than one asset controller!", szAssetName), "Ok");
                    return false;
                }

                bHasController = true;

                if (component.GetType() == typeof(PropModAssetController) && luascripts != null)
                {
                    PropModAssetController pac = (PropModAssetController)component;
                    luascripts.AddAsset(pac.LUAScript);
                }
            }
        }
    }

    EditorUtility.SetDirty(materials);
    EditorUtility.SetDirty(shaders);
    EditorUtility.SetDirty(meshes);
    EditorUtility.SetDirty(luascripts);
    AssetDatabase.SaveAssets();

    return true;
}

After loading each prefab scene, it runs this method on it. Unfortunately, trying to serialize the resultant objects cause the exception.

Any thoughts?

It’s been so long since I posted the original that I’d forgotten I already posted that :frowning:

As an addendum, here is the method used to save all of the asset bundles.

public static void SaveModProject(string szModName, Version modVersion, UInt64 iAssetID, IEnumerable<ModProject.ProjectEntry> lAssets)
{
    // the current scene
    string szCurrentScene = EditorApplication.currentScene;

    // attempt to save the current scene before building the assets
    EditorApplication.SaveCurrentSceneIfUserWantsTo();

    // prep the manifest
    // this special manifest will allow us to detail the contents of the mod and keep track of a load order to prevent breaking dependency chains
    ModAssetManifest manifest = new ModAssetManifest(szModName, modVersion, iAssetID);

    // begin by clearing out the asset folder if there was already one
    string szModFolder = string.Format("Assets/ModStaging/{0}", szModName);
    AssetDatabase.DeleteAsset(szModFolder);

    // next, create the folders we need to make use of
    AssetDatabase.CreateFolder("Assets/ModStaging", szModName);
    AssetDatabase.CreateFolder(szModFolder, "Prerequisites");

    // build the path to the export directory
    string szExportFolder = Application.dataPath.Replace("Assets", string.Format("ModExport{0}{1}", Path.DirectorySeparatorChar, szModName));
    // delete the directory if it exists
    Directory.Delete(szExportFolder, true);
    Directory.CreateDirectory(szExportFolder);

    // first, we need to build our prerequisite objects, that way we can build dependencies reasonably well
    // the reason to do this is because otherwise Unity will include the assets in every asset bundle, creating larger asset bundles than necessary and duplicate resources
    ModPrerequisites shaders = ScriptableObject.CreateInstance<ModPrerequisites>();
    AssetDatabase.CreateAsset(shaders, string.Format("{0}/Prerequisites/Shaders.asset", szModFolder));

    ModPrerequisites materials = ScriptableObject.CreateInstance<ModPrerequisites>();
    AssetDatabase.CreateAsset(materials, string.Format("{0}/Prerequisites/Materials.asset", szModFolder));

    ModPrerequisites meshes = ScriptableObject.CreateInstance<ModPrerequisites>();
    AssetDatabase.CreateAsset(meshes, string.Format("{0}/Prerequisites/Meshes.asset", szModFolder));

    ModPrerequisites luascripts = ScriptableObject.CreateInstance<ModPrerequisites>();
    AssetDatabase.CreateAsset(luascripts, string.Format("{0}/Prerequisites/LUAScripts.asset", szModFolder));

    // we will add all of the scenes to the asset bundles
    List<AssetBundleBuild> lBundles = new List<AssetBundleBuild>();
    AssetBundleBuild bundle;

    // iterate over all of the assets and prep the prerequisites
    string szNewPath;
    ModAssetManifest.AssetEntry manifestEntry;
    foreach (ModProject.ProjectEntry obj in lAssets)
    {
        // open the scene to start
        // we do this so that we can properly load the scene
        EditorApplication.OpenScene(obj.m_ManifestEntry.m_szPath);

        // prep the new path
        szNewPath = string.Format("{0}/{1}.unity", szModFolder, obj.m_ManifestEntry.m_szName);

        // first, copy it to the mod staging folder
        AssetDatabase.CopyAsset(obj.m_ManifestEntry.m_szPath, szNewPath);

        // add an asset bundle for it
        bundle = new AssetBundleBuild();
        bundle.assetBundleName = string.Format("{0}.unity3d", obj.m_ManifestEntry.m_szName);
        bundle.assetNames = new string[] { szNewPath };
        lBundles.Add(bundle);

        // update the path in the manifest for the export
        // this way we are pointing at the correct location
        manifestEntry = obj.m_ManifestEntry;
        manifestEntry.m_szPath = bundle.assetBundleName;

        // add the asset to the manifest
        manifest.AddAsset(manifestEntry);

        // load the scene
        EditorApplication.OpenScene(szNewPath);

        // process the scene
        if (!ProcessScene(obj.m_ManifestEntry.m_szName, shaders, materials, meshes, luascripts))
            return;

        // save the scene
        EditorApplication.SaveScene();
    }

    // open the scene we started with now
    EditorApplication.OpenScene(szCurrentScene);

    //
    // if we had any, prep the prerequisites and set up bundle build objects for them
    //

    if (shaders.Count > 0)
    {
        bundle = new AssetBundleBuild();
        bundle.assetBundleName = "Prerequisites/Shaders.unity3d";
        bundle.assetNames = new string[] { "Assets/ModStaging/Prerequisites/Shaders.asset" };
        lBundles.Insert(0, bundle);

        // add the path to the prerequisite
        manifest.AddPrerequisite("Shaders", bundle.assetBundleName);
    }

    if (materials.Count > 0)
    {
        bundle = new AssetBundleBuild();
        bundle.assetBundleName = "Prerequisites/Materials.unity3d";
        bundle.assetNames = new string[] { "Assets/ModStaging/Prerequisites/Materials.asset" };
        lBundles.Insert(0, bundle);

        manifest.AddPrerequisite("Materials", bundle.assetBundleName);
    }
          
    if (meshes.Count > 0)
    {
        bundle = new AssetBundleBuild();
        bundle.assetBundleName = "Prerequisites/Meshes.unity3d";
        bundle.assetNames = new string[] { "Assets/ModStaging/Prerequisites/Meshes.asset" };
        lBundles.Insert(0, bundle);

        manifest.AddPrerequisite("Meshes", bundle.assetBundleName);
    }

    // build the asset bundles now
    BuildPipeline.BuildAssetBundles(szExportFolder, lBundles.ToArray(), BuildAssetBundleOptions.UncompressedAssetBundle, BuildTarget.StandaloneWindows);

    // serialize the manifest
    manifest.JsonSerialize(szExportFolder);

    // let them know they are done
    EditorUtility.DisplayDialog("Mod Build Complete", "Finished exporting mod!", "Ok");
}

Bump?

Hmm… I don’t see anything that jumps out at me. The only slight hunch I have is that you may need to call AssetDatabase.SaveAssets() and/or AssetDatabase.Refresh() after your CreateAsset calls, but it’s just a hunch.

What I’d try is doing an EditorUtility.SetDirty() on one of your ModPrerequisites objects at different points in the process to see if you can work out the earliest point at which the objects are no longer valid. That should give us some clue which call is causing them to become invalid.

Thanks for the help! I added some additional logging to figure out when they went null (not sure why I didn’t before :p). It turns out that EditorApplication.OpenScene() was causing them to go null. I added a load for them after the scene was opened, and the exceptions went away.

Now that they are being serialized however, I’ve discovered that what I’m doing doesn’t actually work (maybe?). In the main manifest for the asset bundle export, there are no dependencies listed. Should I try using the 4.X solution, or perhaps go about this another way?

If you’re using the variant of BuildAssetBundles that takes an array of AssetBundleBuild objects, then yeah, I don’t think it will collect dependencies automatically. You’ll need to work out the dependencies yourself and include them in the array of assetNames for each bundle.

EDIT: Alternatively - it’s probably going to be better if you can use the BuildAssetBundles overload that doesn’t take explicit bundle definitions, but just uses the assets in the project to work out the bundles to build. That should take care of all the dependency problems, as well as giving you incremental build support.

I tried out the BuildAssetBundles() alternative that doesn’t take the explicit definitions, but I was still left without any dependencies in the manifest, and the file sizes agree with the manifest’s assumption. I tried ordering it several different ways, but ultimately it doesn’t look like the system could figure out what I was trying to do. I’ll try the old system next.

The 4.X system works exactly as expected. With the prerequisite object, it exports a 15kb bundle, and without, a 719kb bundle. In addition, when loading the dependent bundle without the prerequisite, it does not load the game objects that rely on missing assets.

With that, my mod exporting/importing system is now complete. Thank you very much for the help!

I came to the same conclusion : the previous AssetBundle export 4.X system works better than the new “automatic depencies” 5.X system…