Could Unity apply it's new super fast LZ4 compression to the metadata folder?

News & General Discussion
1

Unity is adding LZ4 compression for WebGL builds.

This is a very fast modern streaming compression algorithms that still gives good compression ratios.

You like me might have noticed that the project folders, and in particular the metadata folders can grow massive. I have a couple of projects where they are over 1Gb.

In a test on a 1.64 Gb metadata folder zipped with fastest compression settings the files were compressed to 126 Mb.

So would it be possible to use LZ4 or similar compression on these folders content?

2

Aren’t those metadata files intended to be accessed by version control software?

2 Likes
3

Note I have tried applying the Windows Compression to the metadata folder but Unity crashes when I try to open the project.

4

But if you’re not using version control software they are just chewing up a lot of disk space, maybe compression could be optional?

5

While it may not be the focus of your thread, I feel like anyone not using version control is not a very serious developer. I can’t imagine not using it for any project as it is simply too beneficial not to be using it. Either way though the amount of space we’re talking about is not really that noticeable on a modern platform.

I don’t know why you’re encountering problems with Windows’ compression as that should be on the file system level and not affect Unity in any way. I may have to grab one of Unity’s sample projects and try it out on that as a simple project doesn’t seem to have any problems.

3 Likes
6

They aren’t just for version control, unity stores critical data in them. I would imagine that compression would slow things down.

7

I used to not bother with vcs on small/prototype projects, but after losing one, I ended up creating a repo just for multiple small projects. It’s really just become a way of life.

3 Likes
8

I don’t think Library/Metadata is version controlled I think Asset metadata files are, as Library/Metadata is regenerated on load.

9

Oh, sorry misread. Hmm… the metadata looks like cached or temp versions of the asset metas, maybe?

Yea, for version control, typically I ignore the library.

10

Doesn’t it store the “imported” assets, the thing that always takes forever when you open a project for the first time?

1 Like
11

Something like this. The library folder is a local cache used for optimising stuff. It doesn’t need to be VC’d as it can always be generated on the fly.

The .meta files on the other hand contain some critical data.

1 Like