Performance Improvements To Text Serialization In 2019.3

Unity Engine
,
1

Hey all!

Alongside many other exciting changes Unity 2019.3 also includes some large improvements to our text serialization backend.

What’s Changed?
Unity uses YAML as a text serialization format, up until 2019.3 we’ve been using an open source library called libYAML.
libYAML has served us well but for speed and maintainability reasons we’ve been working on a custom internal library called UnityYAML.

2019.3a1 brings about the full switch to this custom library.

Why does this mean for you?
In theory the switch should be completely transparent from a usability point of view.

From our internal testing the switch over to UnityYAML has resulted in a 30-40% speed up in scene loading times and a ~13% speed up when importing prefabs using text serialization.

You should see a speed up on the reading and writing of any text serialized documents, these are just some common examples.

We’d be very interested in hearing the improvements you see in your project from this change.

What should you look out for?
We’ve worked hard to test and QA this new improvement, but as always things may slip through the cracks.

Please file a bug report according to these instructions if you experience any of the following:

  • Version control noise:

For example, a text serialized asset changing in a way where the data is the same, but simply moved around a bit due to the new writer.
We have done thorough testing for this, but it’s worth keeping an eye out.

  • Data loss during serialization

We’ll be tracking new incoming bugs that we think could be related.

If you are creating your own YAML files
This has never been officially supported but If you’re currently externally creating YAML files then you might need to update your tooling to make sure the format is correct.

We also want to give you this forum thread as a place to voice any questions or ask about any related issues you’ve found.

19 Likes
2

Is it, or are there any plans to make it available for the runtime? Would love to have YAML available as an option to JSON for serialization.

2 Likes
3

Currently this is a C++ library integrated into the editor, we don’t have any plans to expose it via a C# api I’m afraid.

4

The scenes in my current project open so much faster in the editor now, it was immediately noticeable from the moment I opened the first scene.

Kudos to the team who worked on the serialization improvements!

13 Likes
5

Can support serialize dictionary?

5 Likes
6

It’s a very good news! Thanks for the hard work!

7

I also realized a speed-up on my powerful machine by opening a set of scenes. In fact, opening all 43 scenes of FAR: Lone Sails at once in the editor was 42% faster in Unity 2019.3 than in 2018.4. Thumbs up!

7 Likes
8

Would this speed up animation importing times too?
I know in the past when I was doing mocap work I made all projects binary because selecting any assets that contained the animations would hang the editor for ~30 seconds as they loaded

9

When I saw the title I thought this was improvements for serialization of string fields. Lol
Awesome work!
Was this done purely for improvement on loading scenes and prefabs?
Will this have effects on Unity collaborate?
Will there be an option for legacy text serialization if we are running in to issues?

10

Exciting! Though I can’t help but think this would’ve been a great time to address the gripes most people have with the current serialization system, like the inability to to serialize dictionaries. Still, looking forward to the changes!

11

If you have “Asset Serialization Mode = Force Text” turned on in the Project Editor Settings, many asset types are saved as text format and thus it affects more than just scenes and prefabs.

There is an option in the Project Editor Settings, it’s called “Asset Pipeline (experimental)”.

1 Like
12

This change was focused on optimizing the current system and as such doesn’t allow C# dictionaries to be serialized.

If the animation files are stored as YAML .assets then you should definitely see a speed up

This change should improve the performance of all systems that load and save text serialized data, scenes and prefabs are just very common use cases.

The only changes for collaborate would be improvements to importing times when pulling down new assets.

Currently there’s no option to revert back to the old system, as of 2019.3 UnityYAML is the default when using text serialization.

Yeah I can see that, as mentioned above this project was mainly focused on optimizations.
I’d personally love to see serializable dictionaries but it’s out of the scope of this project.

1 Like
13

Looks like they are, that’s awesome news then - text serialisation is so much nicer for source control :slight_smile:
I haven’t got a project to this version yet, but you’ve clearly done fantastic work here!
(Though I still imagine loading text-serialized mocap data will still be slow as the anim files are so damn verbose)

14

Few bugs I have noticed when working with version control on old system.

  • Sometimes serialized references was serialized in one line, sometimes in two making difference for source control:
m_SkyboxMaterial: {fileID: 10304, guid: 0000000000000000f000000000000000, type: 0}

m_SkyboxMaterial: {fileID: 10304, guid: 0000000000000000f000000000000000,
type: 0}``` suggest just always serialize it into exactly one line.

Order of serialized objects sometimes was changed.
Suggesting just always sort it by uid so every chunk of data will always be in same place no matter what happens.

Hope that this issues will be adressed with new serializer :)
2 Likes
15

This happens when the line passes over roughly 80 characters and is behavior we match with the old writer.
If we were to update the logic to only write one line we’d actually end up generating a bunch of version control noise fixing up existing two line references.

16

Can you please add Project option to make it serialize in 1 line and may be make it default for new projects.

Line length often changes from commit to commit and actually create noise in source control all the time :slight_smile:

8 Likes
17

We can definitely look into this, great idea! :slight_smile:

11 Likes
18

Very interesting, we have moved to flatbuffers for most of our data as unity wasn’t able to load scenes fast enough, maybe we can now move some of the smaller stuff back for easier usage ^^

1 Like
19

Wow! 0_o -What a fast this Unity 2019.3.a7 !!! :):):slight_smile:
-With beta 2019.2 - just can be no comparison!
This is just amazing! :slight_smile: I installed beta 2019.2 to be able to work with 2D animation and other preview packs and after working with it a little bit, I realized that most likely my notebook will simply not be able to work normally with any new versions of Unity… :frowning:

But with 2019.3.a7 - all again “Flies”! :))) It was completely unexpected … :slight_smile:

-Big Big BIG THANKS to the developers! :smile:

P.S. Sorry for Google-translate… :roll_eyes:

1 Like
20

Hi @harryr , this sounds like a really awesome improvement! I’m not sure this is the right place / scope, but it might be:

One thing that is bothering me a lot is that during Unity updates, on the actual update, only a few changes are applied in the actual project files (i.e. everything under Assets). However, there are a lot of changes that occur in the YAML files (I’m assuming it’s the YAML files, the two types of assets I see this most with are scenes and prefabs), and those changes seem to be then written into the files whenever the file is opened and a change is made, causing a lot of noise unrelated to the actual changes that we as developers do in those files.

While I can total see how this will usually improve the update experience by not having to iterate over all the files on every update, it really does mess up version control because after each Unity update (and that seems to also include minor releases), the changes end up in version control distributed over several days, whenever any file that has such changes is touched.

I believe this is a case where both approaches are needed because it’s like that some people will prefer the current approach, with quicker project updates. But other people, like me, would prefer to have every change caused by a Unity update to be in source control in my “Updated Unity to 2019.x.y” commit, separate from my actual development changes.

Such an option would most likely be best put into Preferences / General, next to “Compress Assets on Import” (because it’s a very similar decision to be made), and could be called something like “Immediately Update Asset Layout on Unity Update” or maybe “Update all YAML-Files on Unity Update”.

You might be doing this during the 2019.3 update already … or convert to Asset Database v2 “on-the-fly” which would also make finding serialization bugs much harder during the alpha. But from my experience, finding actual serialization bugs would be pretty difficult due to the many “Unity version update” changes we have when going from one version to the next (e.g. 2019.2 to 2019.3).