New Unity 6.6 ContentDirectory GOOD and BAD Q&A

Unity 6.6 announced new recommended way to work with assets in runtime is ContentDIrectory

I have tested them and this is what i found Good and Bad about current design and how it actually works.

What is Good

  1. Very simple API to use. 100 times simpler than Addressables! Just Build → Register → Load → Unload that’s it.
  2. Uncompressed form is really just Directory with builded assets and manifest you can check any dependency graph and any other assumption after actual build not before the one
  3. Deduplication is built-in no need to do anything
  4. Incremental rebuilds rebuild only what changed. Fast iterations
  5. Easy to migrate from any good OnDemand Loading solution (not bloated one). For my Flexy.AssetRefs it is just one config change in project and everything work from ContentDirectories
  6. Easy to see what assets become part of build and size of each. Easy to analyze build size
  7. Out of The box Ready for Different AppStores delta patch updates like steam, PlayStore, AppStore etc
  8. With simple trick it is easy to deliver simple individual assets of dictionary on demand. replace asset with empty file with same name and deliver it when you are ready just replace it in directory and load again. It works with asset dependencies too.
  9. Sources for build is not Entire files on disk but individual Assets in them and every asset result in separate file in ContentDirectory. It is amazing :slight_smile:
  10. Unloading asset Just Works without any hoops. Load prefab with material and heavy texture on it. After some time unload prefab and underlying material and texture will be unloaded too. Unlike Addressables and bundles that actually unload only if entire bundle will be unloaded! Amazing :slight_smile:
  11. If some texture is not in DIrectory loading Prefab with reference to texture fail only fail on texture all other will be loaded as expected. An unloading prefab than replacing texture with correct one and loading again will load prefab with texture. Loading Just Works :slight_smile: I really like this new system :slight_smile:

What is Bad

  1. To deliver assets on demand we need to replace them with empty files. While it is easy to do, it is weird step that must not be there. System must init based on files in manifest. Not on actual files in directory.
  2. Asset file hold only partial info about asset not all of it. Some metadata live elsewhere and replacing texture with same one but imported with another srgb mode do nothing. New build see new texture version but old one see old one even after replacing. It is looks like a BUG
  3. You can replace asset inside Directory with another one but only if it have exactly same size in bytes for texture it is mean same dimensions and compression and may be everything but some pixel changes. This is very restrictive and fell as bad design that forbid small Patches and Game Mods Support in the first place. Size of asset and other actual metadata must be part of ContentFile not stored elsewhere. So we can tweak or replace asset of a game by just changing files with new ones. Think about HD Mod that replaces all textures with bigger and nicer ones.

Currently Unknown

  1. How to build few content archives from single ContentDirectory (need deep dive into addresables to see how they do it) and test how it works overall
  2. How to create new assets into new ContentDirectory that will replace old ones in build by loading second content directory. Say new Hat styles for Hallowing without chaging original Hats. Like build new hat prefab and make system to load it instead of old asset on GUID path. There is almost no information for this.

Do anyone know or tested something more or have another questions/usecases to ask/test about ConentDirectories?

7 Likes

Hey, as someone who builds web stuff i had heck of a job with addressables, so as a result, ive not tried this if you can give a quick example of how you “build, register, load, unload” id love you for it

I have use my package Flexy.AssetRefs for managing on demand assets instead of addressables because they was bad from the beginning. That was actually the reason I have created Flexy.Assetrefs.

They collect all Refs for ondemand build and than other tasks convert them to something that can be loaded in runtime.

By default they prepare and load through Resources
So the only thing I have changed is build ContentDirectory instead of refs from resources and replace loader to ContentDirectory API instead of Resources API

so this was not ground up work. Just few tweaks to the system to support ContentDirectory

I almost have done with it and tomorrow will push new version to Github with ContentDirectory support

main pieces is:

  • Create ScriptableObject with array of references in my case map AssetRefs to Loadable
[SerializeField] AssetRef[]       _assetRefs       = Array.Empty<AssetRef>();
[SerializeField] Loadable<Object>[] _assetLoadables       = Array.Empty<Loadable<Object>>();
[SerializeField] SceneRef[]       _sceneRefs       = Array.Empty<SceneRef>();
[SerializeField] LoadableSceneId[]  _sceneLoadables       = Array.Empty<LoadableSceneId>();
[SerializeField] String[]         _sceneNames          = Array.Empty<String>();
  • Pass this one SO to BuildPipeline.BuildContentDirectory

  • On [RuntimeInitializeOnLoadMethod(RuntimeInitializeLoadType.BeforeSceneLoad)] spawn prefab with ContentDIrectory Service that will register contentdirectory with ContentLoadManager.RegisterContentDirectory(contentDirectoryPath); and load my SO with _contentCatalogs = ontentLoadManager.GetRootAssets<ContentDirectoryCatalog>(_contentDirectory);

  • Than pass any asset loading through this service

public override       T?          LoadAssetSync<T>   ( AssetRef @ref ) where T : class  
{
    var asset = GetContentLoadable(@ref).Load(); // Get corresponding Loadable<T> fro AssetRef and Load
    return GetSpecializedAsset<T>(asset); // Make it able to load Components on prefabs directly in other cases just return asset as is
}

More detailed flow you can see when I push all this into Flexy.AssetRefs tomorrow

3 Likes

Released AssetRefs 6.6 on github with support for ContentDirectories

Now you can see exact implementation of it in files:

Hi JesOb,

Thanks for taking the time to share all this great feedback! We’ve spent a lot of time designing and building this new pipeline so it’s great to see that you have identified a lot of architectural improvements. In 6.6, we are mainly targeting static ContentDirectories that ship with the player. In future releases, we will add more features to better enable dynamic content. For the time being, I’ll share a few of the implementation details that I think you might find useful based on your comments

To deliver assets on demand we need to replace them with empty files. While it is easy to do, it is weird step that must not be there. System must init based on files in manifest. Not on actual files in directory.

When you register a ContentDirectory, the available files are enumerated to create a snapshot in order to reduce file look ups and support the optional file extension. When archiving is enabled, we mount all the archives and then perform a similar enumeration. We are exploring a hook point where users can provide the raw data through the virtual file system instead of requiring files in specific folders.

To address your #2 and #3 I will try to explain a bit more of how these files are laid out.

All unity Object types go into ContentFiles and as you noticed we break each asset into 1 or more ContentFile. The ContentFile may reference other artifacts such as the raw texture data, or some other raw binary data. This binary data is always referenced from the content file via a x3 hash of the content it is referencing. ContentFiles can also reference each other, but this is not a content hash reference, it uses IDs derived from the asset source. The manifest file is responsible for mapping the IDs to the content hashes of the content files.

If I understand correctly, it sounds like you are trying to swap out the raw binary data of a texture, but keep the filename the same. This could cause problems because the texture’s content file is built with the expectation of specific data as you are seeing.

How to build few content archives from single ContentDirectory (need deep dive into addresables to see how they do it) and test how it works overall

In the 6.6 release, BuildContentDirectory can take an archiving flag, which will put all artifacts into a single archive. However, as you noticed, Addressables does its own archiving after the build. Addressables loads the ContentLayout.json file, which includes all reference information about artifacts, and then assigns them to archives manually. The archives must follow a strict naming convention in order for them to be properly mounted during content directory registration. For the time being, I would have a look at the Addressables code to learn more.

Multiple Content Directories Behavior

Here’s some extra information about the behavior of registering multiple content directories in 6.6:

  • Each content directory has a list of Loadables

  • When you load a Loadable, the ContentDirectories are iterated and the first match encountered (most recently registered directory) is chosen.

  • Direct references of the Object loaded via Loadable will always be from the Content Directory the Loadable is from and direct references will never cross into another Content Directory.

  • If the same asset is built into two content directories and directly referenced by a loaded Loadable in each, the referenced asset will be loaded into memory twice, once for each content directory.

You can think of the ContentDirectory as a mostly isolated unit where only the Loadable reference can cross between them. So generally we have recommended trying to have fewer, larger ContentDirectories instead of many, small ones. Addressables uses a single ContentDirectory for the entire build. This behavior could be changed in future releases to better enable sharing of assets between ContentDirectories.

2 Likes

Thanks JesOb for being an early adopter of the new build system and all the information and feedback you have posted here.

There is some detail of the expected naming convention here:
UnityDataTools contentdirectory-format

1 Like

Thanks for clarifications :slight_smile:

I will continue to investigate how to support

  • CDN packs
  • DLCs
  • Mods
  • Event Content Overrides

With this new system :slight_smile: and share my findings here :slight_smile:

1 Like

Please add CancellationToken support to LoadAsync and LoadSceneAsync, similar to Object.InstantiateAsync.

While the concept of ContentDirectory seems really great, without cancellation support, it is hard to manage loading efficiency and optimize large-scale games where asset/scene loading needs to be canceled frequently (which was one of the pain points with traditional AssetBundles / Addressables).

3 Likes

Thanks for this. I’d be interested what your use case is, what are the situations where you’d like to cancel a load operation which couldn’t be solved by not queuing up load operations?

@NikuPotato is on spot. So many usecases, the simplest coming in mind would be a clustered world / areas.
You pre-load neighbors clusters depending on player movement, and suddenly boom, it dies, or it teleports. So what, you wait until the loading finishes or you cut short ?

Internally the system would be loading every little bits and pieces, but as a user you just want to pre-load/cut load of your large chunks.

This is a spatial example but any usecase of the sort would want that. (Enemy waves, Runner game, multiplayer synchronisation, interaction, etc, …).

It’s a matter of fluidity and ease of use.

Real life example we had : On a turn base game loading “tactical data/path finding” is a lengthy operation, yet the player flickers the gamepad at will to browse areas/actions/toggle UIs. Everything thing needs to be lazy loaded/unloaded in the blink of an eye in sometimes non parallelizable contexts.

3 Likes

The simplest case of all is stability in the editor: we often exit play mode at “strange” times while running in the editor. Without cancellation tokens being threaded through all async work, this can leave async tasks running, leading to weird behavior and errors downstream from leaving play mode. We try to religiously thread cancellation tokens through all async work everywhere in a project to avoid this kind of thing. And this behavior in the editor is only a symptom of the underlying genuine need in the production build of the player. Users at runtime can “cancel” all kinds of things at “unexpected” times. For example, a user might enter a part of the UI where they browse a catalog of async loaded assets to equip a character with, and then quickly navigate “back” out of that UI while those assets are still loading. One would want to cancel those loads in many cases (this is game specific, of course). We wouldn’t want to have to choose between either async or the ability to cancel, right?

I think it’s pretty safe to say that if async is offered, it should expose and honor cancellation tokens. I can’t really think of any place where this would NOT be desirable, can you?

5 Likes

As @Whatever560 mentioned above, canceling pre-loaded assets is a major use case. We want to pre-load assets aggressively to eliminate loading screens, but those pre-loads often need to be canceled depending on player actions.

Currently, without native cancellation support, if we want to “cancel” a load operation, we are forced to wait for the load to complete fully before issuing an unload. This introduces significant CPU/IO overhead and memory pressure, which is precisely the performance bottleneck we want to avoid.

2 Likes

Cheers folks, I’ve made sure I’ve captured the above. Much appreciated <3

3 Likes

Great to hear !

EDIT : https://youtu.be/CYvLn-tYRag?t=899
The exact usecase I’m describing below can be seen in this video and game (other engin, but same idea).

As a technical note coming to my mind, there is a huge concurrency to handle there.
Client side :

  • I preload Room A & B
  • I teleport and load Room C, cutting Room A & B short.
    What happens to already loaded pieces in A & B that will be inside C ?

What I’d expect is for everything to work like a filesystem under the hood :

  • I have a memory budget (How it is calculated is irrelevant there)
  • When I unload Assets they are “invalidated” if they have no more referencers
  • When loading new assets they can scan the cached “invalidated” assets to see if they find something for them.
  • New loaded assets can move invalidated assets out of memory if budget is reached.

Content directory is geared toward that from what I saw.

I believe it’s an important matter as in many usecase we’d have :

  • Shared references (Like all rooms of the same dungeon) and this would highly reduce memory roundtrips.
  • Fast same package load/cut “flicker” depending on the gameplay.
2 Likes

Our case is a co-op multiplayer game with a large variety of weapons. Because of the sheer number of them, we load the meshes and animations on demand. We used Addressables for this before and recently switched to Content Directories.

Players have multiple weapon slots, each holds a single “what should be loaded” target (SO with all the asset references), and the latest request replaces the previous one. The trouble is that the demand can change at any time while a load is already in flight, and we can’t know that when the load starts:

  • Players can switch their weapon loadout freely at any time. Scrolling through slots, or trying weapons one after another, changes the target several times while the initial request is still loading.
  • The same happens for other players. Their weapon changes arrive over the network, so every client loads them on a schedule it doesn’t control.
  • A player can leave, or a weapon can be unequipped or swapped while its content is still loading.

Without cancellation, our loader has to do this for every slot:

while (target != loaded) {
    // Release what we have
    if (loaded.IsValid) { await Unload(); continue; }

    var requested = target;
    var loadable = new Loadable<T>(requested);

    // Cannot be abandoned
    await loadable.LoadAsync();
    handle = loadable; loaded = requested;

    // If stale, release it and go again
    if (target != requested) { continue; }

    Publish(loadable.Target);
}

So when the player has already moved on to weapon B, we still have to wait for weapon A to finish loading and then release it before B can even start. The weapon the player actually wants arrives later, and we’ve paid IO and memory for something that was stale before it finished.

Teardown has the same problem. If the owning object is destroyed mid-load, the loop has to keep running until the load completes, just so it can call Release() and keep reference counts correct.

The ability to cancel it would give up our claim on the content: the task completes right away as cancelled, and Unity releases whatever it had loaded on our behalf, whenever that finishes. We could then drop a stale request and start the new one immediately.

2 Likes

You can create a single list of abandoned requests.

  1. Load request comes
  2. If there’s existing unfinished load request - move it into abandoned list
  3. Start loading new request immediatelly

Then once per frame (or less frequently) go through list of abandoned requests and release the completed ones. You still pay for the I/O and memory, but at least you can start loading immediatelly.

Having Cancel would be nice. Having truly cancellable loading would be great (stop loading process as soon as possible).