I’m working on a project that is stressing the limits of the asset library, I think.
I need to import thousands upon thousands of meshes that I wrote an importer for. I put them into the AssetDatabase one by one and this is an astonishingly slow operation.
I understand that creating AssetBundles out of these is the best way to load them at runtime. I’m working on that.
What’s the best way to bring these into the Unity Editor? Right now it is taking more than 24 hours for a single import operation to complete on some of the larger collections of meshes.
I’ve tackled a similar problem, I had to make thousands of assetbundles out of 150 gb mesh and texture data.
I never actually tried to put the whole thing into Unity project, but tried with ‘smaller’ chunks of the data. I still noticed that the more you put in, the longer it would take to import per asset.
So I opted to automate process where I would import the required assets for one bundle, build the bundle, then remove all imported assets and import the next set.
So I never had more assets in unity than required to build a single bundle.
Still took several days, but suspect it would take a lot longer if I had pulled everything in together.
Maybe even splitting up in different projects can help, some teams do that. They mainly have one code project and content projects and then simply mirror the code assets to the content projects.
It depends on the scripts or other content you refer to but if it is just about the assets then you could
A. Import only a batch of them and then make one asset bundle (bundle_1, bundle_2 etc.)
B. Actually make one project for each bundle/ set of bundles You make one project first and copy everything over like project settings.
Either you know the location of your asset by using any way of cateogrization or you use he manifest to search for the right bundle for each asset on runtime.
I can’t guarantee that this will work but it should, asset bundles generate their own Object ID’s anyway right?
It would be nice if the AssetDatabase was an SQLite database or something. I see those everywhere, those can’t be hard to use.
I ran a benchmark earlier, I could parse 1000 files in Unity and create GameObjects of all of it, in far less than a minute. Importing 500 of those same assets, and saving them to the AssetDatabase is going to take at least half an hour.
I cannot fathom what it could be doing to make it that slow.
You know that assets are converted to custom formats? Like Images become DirectX files on windows or FBX an internal mesh file. Those are all stored in the Library folder.
You can disable Compress Assets On Import that might speed things up. But they have to be converted on a Build.
Unlike other engines you only have those converted files in the build. If you need to load raw assets on runtime you could do that but you have to write custom importers and converters. It is though possible to load Mesh or texture files on runtime from disk. This is used for modding for example.
Can you tell us what kid of files you have to handle?
I know they need to be converted to platform-specific assets on build, but I’m not talking about that.
I’m importing files into the AssetDatabase using a custom script. I’m not doing anything weird, I’m parsing ASCII text into geometry and materials and then adding those assets to the database. I save changes to the AssetDatabase once per file. I must do this at least this often so that I can reference existing materials when I import a new file. Turning compression on or off seems to have no effect on the speed at which this slug moves.
A very slightly modified version of that script can save to .obj and .mtl files, and it can do it for dozens of input files per second. The AssetDatabase API takes several seconds per input file. If I stop saving to the AssetDatabase entirely, and only spawn GameObjects, I’m back up to several dozen files per second.
I seem to remember that importing many assets at once speeds things up. Is it possible to import the materials in one go, and then the rest of the data?
The materials are defined at the top of the file, and I parse and create all of those before I even see the geometry that references it later in the file.
I parse the materials, create Unity materials out of them one by one, and once I’m out of the materials section, I start parsing geometry that references those materials. When I’m done with all of the meshes, I create prefabs for them all then save those to the AssetDatabase one by one.
Then I save the changes to the AssetDatabase and refresh it, and move on to the next file.
So I am doing materials first, but only per mesh. Not ALL materials before all meshes.
That sounds like you could do ALL materials first, then ALL meshes, then ALL prefabs, though, or at least merge the “sections” (materials, meshes, prefabs) of a few import files, to import stuff in larger chunks.
Grouping multiple ImportAsset() calls within StartAssetEditing()… StopAssetEditing() is quite a bit faster, e.g. importing 155 smallish textures takes about 5 seconds grouped, and upwards of 40 seconds one-by-one.
In neither case Unity is CPU- or I/O-bound, and it’s beyond me why importing is still done seemingly single-threaded…