JSON-parsing and Steve?

Unity Engine
, , ,
1

Greetings,
I am doing a lot of the JSON-parsing in my 9-5, and am looking into ways of being able to efficiently use the JSON-data with the powerful parallel processing that comes from the Jobs.

Has anyone here done anything similar, tried, tested, decided for or against, either fully ECS, or quasi, or not? I would be very interested in hearing your findings on this. Thank you very much in advance for helping out.

2

I can imagine you can run multiple JSON parsing in parallel. Haven’t tested.
But each thread would be responsible, for each individual JSON data.
You parse back to class right?

But if you have big JSON file, and you want split process between threads, I don’t think there is easy way to do so.
For the first, you would need split data in right point. You don’t want to split data in mid of word.
Then writing to a class, would required writing from multiple threads.

Probably converting class to JSON string in parallel, is more feasible, as you could probably simply join job results strings, into one DATA, afer jobs are done.

You would need to know about the jobs order however.

1 Like
3

Are you considering rolling your own serializer? That’s almost never worth it. Parsing JSON sounds difficult to implement using Burst, especially if you need to support managed objects. And if you’re dropping Burst, you might as well use one of the existing solutions. I think most serializers should work fine in a parallel job as long as you don’t touch any (non-threadsafe) Unity APIs. Here’s a summary:

  • JsonUtility - fast, thread-safe, limited feature set (works just like the built-in Unity serialization, which is a plus if you’re using that extensively)
  • Newtonsoft.Json - versatile, popular, customizable, thread-safe, doesn’t seem to like certain Unity types but it’s mostly fixable
  • com.unity.serialization - pretty slow, internally uses burst+jobs for tokenization so I don’t think you can even use it in a job, but maybe it’s good for something I dunno

Performance-wise, here’s a random badly-written benchmark.
6641980--758131--upload_2020-12-20_10-56-34.png

And another for large objects. Ouch, that took a while to run.
6641980--758137--upload_2020-12-20_11-11-5.png

2 Likes
4

Not doing my own serialiser (I really hope, I hsd to make my own triangulator which took half a year), just wondering what is out there.

I am currently using the JSONUtility. I have looked into the code and wondered how it would look in burst, but since it’s all strings (“ew, gross”) it’s unlike the datatypes (bools/bytes/ints/floats) that I have so far used.

I tend to have extremely large objects to parse, so the example above is a realistic encounter for me. However I don’t intend parsing JSON at runtime, only in between play sessions or whenever there is no direct gameplay activity.

Also, you can use structs with JSON apparently.

5

Look at Project Tiny (com.unity.tiny.all). I just checked there now and found an implementation of an unmanaged JSON parser that looks like it is also Burst friendly. You should be able to simply copy that portion of the package out into whatever project you want to use it in without taking on all the other Tiny bagage.

2 Likes
6

Regarding the serializer take a look at GitHub - neuecc/Utf8Json: Definitely Fastest and Zero Allocation JSON Serializer for C#(NET, .NET Core, Unity, Xamarin). and messagepack as alternative to json GitHub - MessagePack-CSharp/MessagePack-CSharp: Extremely Fast MessagePack Serializer for C#(.NET, .NET Core, Unity, Xamarin). / msgpack.org[C#] (they both share quite some features and have quite some problems but if you want to understand how to be really fast they are a good start).

Also checkout these two benchmarks:

What do you want to achieve with Jobs in particular, that can’t be done with regular threads better?

1 Like