If Streaming Assets folder is supposed to leave files as they are in the final build, then why does unity import them?
Unity’s asset importing system is very outdated and non-functional. Not only does it take multiple minutes to import mere MEGABYTES of data but it also has very frequent crashes, not to talk about all the trash .meta files that get generated and which get extremely annoying when working with many files.
That being said, is there any way to add files in StreamingAssets folder without the importer doing anything to them? They should also be availible in build versions, so no .hidden folders and ideally not using project/unity settings, since i need it to work in other machines too.
According to this:
StreamingAssets shouldn’t be getting imported anyway.
True, I just saw that i had “AssetDatabase.ImportAsset(filePath);” somewhere in my code! Sorry for the bad the question 
EDIT: I removed the line in question and its still importing, so i still have the same problem.
I have tried changing the extension to some non existing format (.unityimporterignore), but unity still imports it.
This only happens when I press Ctrl + r to reload domain, not when entering play mode.
if there is no way to remove the importing behaviour, is there at least some way to speed it up? It currently takes more than 5 minutes to import less than 400mb in files (many small files). I had thought about maybe creating the .meta files myself to “trick” the importer into thinking the files are already imported (note that i’m talking about the Default Importer, which does nothing apart from creating .meta), , is that possible?
The performance of import operations may depend heavily on (custom) importers and editor scripts that respond to asset import events. Be sure to profile! It could very well be an editor script in your project that slows everything down.
Also be sure to exclude your project from antivirus scans! These can needlessly wreck havok on file IO performance.
Lastly, many many many small files is always going to be far slower to walk through and read in compared to a single file with the same size. Easily by a hundred if not thousand-fold, and much more so on a spinning magnetic disk.
These are mandatory, not trash. They encode asset import settings as well as storing a unique GUID for every asset. If you were to ever delete a .meta of an asset file that is references elsewhere, you’d lose that reference.
StreamingAssets is designed not to touch the files inside. In fact, it is completely ignored by the AssetDatabase and no .meta files will be generated inside that folder.
If you see otherwise, you may not have used the correct name or location of that folder. It must be:
Assets/StreamingAssets/...
It won’t work in a subfolder of Assets. It won’t work if you call it “Streaming Assets”. It won’t work if you add a StreamingAssets folder to a package. Etc.
Sorry for the bad tone against Unity, I was pretty stressed at the moment, that being said:
-
Unity’s importer still seems absurdly slow to me, my script is reading data for all the files, transforming the data, and saving the data to Disk in mere seconds while the importer takes an order of magnitude more time to create .meta for that same data.
-
I agree that .meta files are necessary for Unity to work, but it could be interesting to explore or propose to Unity maybe a central .meta file for folders containing the data for all files or even the entire project.
Unity is generating .meta files for all my folders, subfolders and files in StreamingAssets
StreamingAssets works as intended in the final build, keeping the original data without the .meta files.
You can see that the default importer does indeed, import files in Streaming Assets.
I have also created a completely new Unity project with no plugins and no scripts and it is also importing to StreamingAssets and generating .meta files.
Since Unity seems to need .meta to work, the best solution appears to be to create the .meta files myself.
My question now is, how do I get the “guid” for an asset? The same that would be used by the importer to create the .meta files.
We used to be able to turn off “Visible .meta files” in settings. But I can’t find this anymore. Probably for the better.
It’s not just creating the 200 bytes .meta file. It runs importers which may transform the data to optimized versions (per platform), and then this process also raises events which means the editor and custom scripts will have their share in the process too.
Indeed, I stand corrected. It also runs the importer progress dialog for me if I put in enough files. I wonder if I recall it incorrectly or if this has changed in recent years. I tested with Unity 6 Preview. Last time I used StreamingAssets must have been with 2020 or even earlier.
This is not a solution, and its terrible! This won’t do the same thing as the importer does.
Use the profiler to see where time is going. Perhaps you do have an editor script that slows everything down.
And also consider your machine. Laptops are (IMO) generally too slow to be really productive. Virus scanners slow down the process unless you exclude the project. And generally, the faster the SSD and CPU the faster imports will be.
I forgot to mention, my files are raw byte data that i manage completely from code, this means that I need no compression or conversion from unity’s part.
Ideally I would store them completely outside of unity, but since I want to keep the build process simple I prefer to keep them in StreamingAssets folder.
These files are also accessed using System.IO, and not using any Unity built in loading method.
In this case, where I am directly loading raw data using System and i don’t expect any file modifications after creation, would it be acceptable to create the .meta files myself?
If so, can I just create a “Guid.NewGuid()” to add that as "guid: " in the meta files?
I am also doing some editor profiling, i will add my observations to this comment when i finish!
EDIT: I finished profiling, you were right, the actual Importing process only takes about 22% of the time for the frame, so about 1 minute, this still seems like a lot to me, since my script takes less than 10 seconds to write all original files to disk, but its a lot better than the almost 5 minutes it takes in total.
The remaining 78% goes to "EditorApplication.Internal_CallDelayFunctions()[Invoke], most of the time is taken by the method itself, not by the methods it invokes.
I am not very familiar with editor profiling and I’m not sure what that method is doing, but this seems to be either some plugin that I have or a unity bug causing this.
I suspect Unity needs to do a basic import of streaming assets to be able to show them in the project view. Otherwise, it would be more like a hidden folder (that ends with ~), which you don’t see at all inside the editor.
But it’s only the default importer and that should be very quick. I also suspect some other script in your project is trying to do something with these files (e.g. AssetPostprocessor.OnPostprocessAllAssets does get called for streaming assets).
You could try to verify this with an empty project, to see if it also takes so long when you change something in your streaming assets.
For the profiler, you probably need enable “Deep Profile” in the toolbar first, to see in which exact script/method the time is spent.
That may be your issue here. You gave them the extension .raw which is a known image file extension. It’s not widely used (typically found in digital cameras) and Unity may not even support it. But if it does, Unity may try to import the file as an image until it fails, which may be costly.
For custom binary data commonly used extensions are .bin or .dat(a). Can’t hurt to try and change the extension to see if this matters.
It is never acceptable or suitable nor necessary to do so. First, you don’t know what exactly goes in there. The format is not (officially) documented.
But most importantly, even if you create that .meta file, Unity will check if the asset is in the AssetDatabase, and since it can’t find an asset with that GUID, it will almost certainly try to import it “correctly” anyway.
Scan your codebase for uses of EditorApplication.delayCall. Often anonymous lambda functions are assigned to this event which the profiler may attribute entirely to delayCall - so the time may still be hiding in the event processing code.
So I have been testing the importing system using the Unity provided “AssetDatabase” interface and I have come to various conclusions:
- Importing individual files separately using “AssetDatabase.ImportAsset()” is much slower than using a single “AssetDatabase.Refresh()” call. At 16000 files the difference is about 16 minutes vs 90 seconds.
- File size doesn’t impact speed at kilobyte level (tested 32 kb vs 512 kb)
- For some reason after every import completes I’m getting this:
Editor Application.delayCall: System.Action.Invoke
Waiting for user code in mscorlib.dll to finish executing.
This generally takes longer than the import process itself. I’m not sure what this process is since the profiler doesn’t tell me much, this process also scales proportionally to file amount and it also seems to be affected by calling individual imports on files vs Refreshing all at once:
single AssetDatabase.Refresh() call:
90 seconds + 5 minutes extra time DelayCall: System Action Invoke
individual file importing:
15minutes + 17 minutes extra time DelayCall: System Action Invoke
I have started a new project with now plug-ins apart from “Burst”. I have also removed all non essential systems from my scripts, so after the files are created, nothing should happen.
When investigating further, there are three expensive methods:
1.UnityEditor.CoreModule.dll::‘FindEverywhere’<d__39’1.MoveNext() with 42.7% of ms time
and 29000 calls.
2.UnityEditor.EditorUtility.DisplayDialogImpl() with 20.9% and 1 Call, probably the unity dialogs showing “Importing x”
3.AssetDatabase.Refresh(): 1 call 21% performance
Out of these, the problematic one seems to be “UnityEditor.CoreModule.dll::‘FindEverywhere’<d__39’1.MoveNext()” since this happens after AssetDatabase.Refresh and after all the .meta files have already been generated. This one seems to be responsible for the:
Editor Application.delayCall: System.Action.Invoke
Waiting for user code in mscorlib.dll to finish executing.
For now, I will just tank the performance hit, since it doesn’t seem to be an easy way around it, this is pretty sad because i worked very hard on optimizing my own file creation system and now 90% of processing time is out of my control but well, that’s life.
Why are you using AssetDatabase.ImportAsset? I thought this was about StreamingAsset?
I would definitely expect ImportAsset on each individual asset to take much longer than Unity batch-importing them through Refresh.
Did you try with “Deep Profile”? This should show you where the time in EditorApplication.delayCall is actually spent. FindEverywhere is part of Unity’s internal FindAllAssets/EnumerateAllAssets, so something is processing all assets. That might be something unavoidable Unity does or there might be an issue that you can work around or report a bug for.
This is why we use batch importing via Start/StopAssetEditing:
try
{
AssetDatabase.StartAssetEditing();
// Import your assets here ...
}
finally
{
AssetDatabase.StopAssetEditing();
}
This is easily hundreds of times faster! Give it a shot …
Also note that Refresh() calls UnloadUnusedAssets() which means any in-editor cached resources are cleared from memory. For instance, every asset you now select will be loaded again from disk, further slowing down the editor.
I found an even better and simpler solution.
At the start I assumed that hidden folders wouldn’t get copied to StreamingAssets in final build, since they don’t get copied on normal Assets folder, but I was wrong, apparently StreamingAssets does indeed copy hidden folders.
This means that I can just create a hidden folder “.folder” and put everything in there, with the only downside that you won’t see the files in the Editor.
Even if the solution to my problem was that simple from the start, i don’t consider this a waste of time.
I learned a lot about Unity’s importing system, about Editor profiling, which i had never done and I learned that you should try every possible solution before delving deeper into a specific one.
Thanks for all the help Adrian and CodeSmile!