On behalf of @kae :
The new incremental building of asset bundles and its deterministic build pipeline is a great leap forward for Unity and I hope to see similar improvements in the player pipeline going forward.
That said, there are a few things with the API I’d like to see changed/added. If you need more information on our use case to get some background context, just say so and I’ll write it out - I’ll try to start with a short version.
We pack each asset file into its own asset bundle. We have a workflow where we want quick incremental build times of asset bundles and we have a later stage where we pack these built bundles into large packfiles containing a large amount of bundles each for deployment with a player. We want both parts to be fast, but we want the asset bundles in the packfiles to be small. Currently we build the asset bundles uncompressed to achieve quick build times and in the packfile step we LZ4 compress the uncompressed asset bundles, optionally using LZ4HC for the release builds.
- The current asset bundle loading APIs are suboptimal for loading asset bundles from packfiles. CreateFromMemory requires allocating massive byte arrays on the C# heap and CreateFromFile requires creating a bunch of files.
I see a few options for extending the loading API here.
1a.
AssetBundle AssetBundle.CreateFromFileOffset(System.IntPtr fileHandle, long offset);
The simplest API. Will not work with custom decompression. I added a file handle parameter instead of the path since packfile implementations usually keep it around anyway.
1b.
class AssetBundleStreamContext
{
// Pushes data to the asset bundle loading system.
// Returns an assetbundle if it can be loaded from the buffered data or if it’s already loaded, otherwise false.
// This function should be called continuously to supply more data until the bundle can be created.
public AssetBundle Load(byte[ ] buffer, int offset, int count);
// Discards all internal state for the context, but does not affect a loaded AssetBundle.
// This allows the AssetBundleStreamContext heap object to be reused.
public void Clear();
}
AssetBundleStreamContext AssetBundle.CreateStreamContext();
An example of a streaming API where the application has full control of pushing data to the loading system.
This is the most flexible, supporting streaming from any media with any amount of latency. If the API is allowed to be used from other threads than the main thread, it also allows the user to be in control of what threads to load the asset bundles on.
For those who cannot use the IL2CPP implementation, this does not introduce any native → managed transitions, which can be a minor performance note.
1c.
// Returns false if there is no more data to be supplied, otherwise true.
delegate bool AssetBundleReadCallback(byte[ ] buffer, out int bytesRead, object userObject);
AssetBundle AssetBundle.CreateFromCallback(AssetBundleReadCallback readCallback, object userObject);
AssetBundleCreateRequest AssetBundle.CreateFromCallbackAsync(AssetBundleReadCallback readCallback, object userObject);
This type of API potentially allows Unity to be in charge of threading, but is not suitable for high latency media where all of the data might not be available synchronously and many loads are being done concurrently.
Personally I’d be OK with both 1b and 1c - we use custom LZ4 compression currently so 1a is not an option for us. This situation might change with changes to the asset bundle compression API, as detailed in 2 and 3.
- It’d be nice to be able to compress an already built asset bundle without rebuilding from the source asset.
bool BuildPipeline.RecompressAssetBundles(string[ ] assetBundlePaths, string[ ] outputPaths, BuildCompressionSettings newCompressionSettings);
The API would read the asset bundles at the locations supplied in the array of asset bundle paths, decompress them if necessary, then compress and write the output to the corresponding element in the outputPaths array.
- It’s important to be able to select which compressor and settings to use. The difference in compression speed between LZ4 default and LZ4 HC is quite significant.
To build on Alexey’s suggestion, a BuildCompressionSettings struct that can be passed into all the BuildPipeline Build* calls.
I would put streaming as a CompressionMethod value instead of a bool.
struct BuildCompressionSettings
{
public CompressionMethod method; // Compression method. Some share decompressors, like LZ4/LZ4HC
public int level; // compressor-specific integer that specifies compression ratio vs speed
public int dictSize; // memory to dedicate for dictionary
public int blockSize; // size of compression block
}
Some fields will mean nothing for some compressors. 0 means default for all compressors.
- This is a bit of a wishlist point really, but I’ll post it anyway because I think there can be real value in it.
I’d like all the BuildPipeline APIs to provide support for custom read and write callbacks instead of using the file system directly. In our pipeline each file is built into an asset bundle so we have tens of thousands of files cluttering up the file system. I partition them into directories using the first two characters of the hex of the hash, so it doesn’t slow down that bad, but I’d rather store them in a more optimal way for reading and writing. I know, I know, just buy SSDs right? Sigh.
Let’s make the asset pipeline amazing together! Tell me what you think.