Using JSON for a database

Unity Engine
1

So if I have a JSON file that acts as a database, is there any way to use it without loading it all into RAM? Some way to “query” the JSON file instead?
If you take in the JSON file as a TextAsset, is the entire JSON file now stored in RAM anyways?
If it is, once you build your database class out of your JSON file, can you delete the TextAsset from memory by simply setting it to null and letting garbage collection do its thing?

My concern is RAM usage if the JSON file gets too large, for instance if you are storing mass amounts of quest dialog in it. In most cases you don’t actually need the entire database here and now, so storing the entire thing in RAM seems a bit pricey for what you get. Would something like SQLite be better for this case, or would that be overkill?

2

How big do you expect your json file to get?

3

I actually don’t have specific use case at the moment, just doing some research.
At what size would you start considering this a concern? Each object in JSON would translate to a class in Unity, which has some overhead of its own, as well as the data effectively being doubled in size during the time it is fully parsed but before the source file is deleted. Say you gave the player the ability to add new dialog or events, which are stored as JSON files to be shared with other players through something like Nexus, Steam Workshop, or even some in game interface. I could see this adding up if a player piles on the mods.

4

Rather than loading from the file using it as a TextAsset, you need to read it using a FileStream.

using Sytem.IO;

class FileReader
{
    public byte[] ReadFileFromPosition(string path, int position, int bytesToRead)
    {
        FileStream stream = new FileStream(name, FileMode.Open);
        byte[] readBytes = new byte[bytesToRead];
        stream.Read(readBytes, position, bytesToRead);
        return readBytes;
    }
}
5

I would rather suggest you to use a database instead. There are several good databases out there that you can use locally. So basically it saves the database as a file on your disk and reads the content from there.

This comes closest to the part saving everything in a file, but still be able to query stuff from it. The database will also be easier to be managed, than a very very big textfile with for example json parsed content that uses a lot of memory when loaded totally. Also the suggestion to read it from a file with position and count to read is not really maintainable as the data will move inside the file if you save more and more stuff in it. That way you would need a proper algorithm that stores the position of everything somewhere, which makes it even harder to use.

Beside of that I wouldn’t use JSON files for a whole database, it’s more of a use to send data from A to B. So I use it to send my data from some online database or something else to my game, because it’s easy to read and parse into objects. So it’s simply used to transfer stuff and drop it, but not use it constantly as a database source.

So my suggestion would be to use sqlite as you already mentioned. You can query the data with simple SQL commands and use a for example a provided sqlite library for easier access.

2 Likes
6

You could consider splitting your “database” into multiple json files. Then only load the ones needed. With your quest dialog example, only load the json file for the specific quest the player is running. You could have 100 quests, split into 100 json files, and you only load the json for the single quest.

As already stated, don’t use TextAsset, use FileStream or System.IO.File, and don’t make your json files a part of your unity assets, make them an outside file.

Doing it like this would also make it easy to modify quests, where you can make changes to your json files outside of Unity and don’t even need to generate a new build to implement the changes.

2 Likes
7

Don’t pre-optimize. (Admittedly this is a gray area between pre-optimization and planning for efficiency, but please see my rationale below.)

What’s the maximum amount of quest dialogue you expect to have in your game? An average novel has about 500,000 characters of text, or 0.5 MB. A single 2048x2048 texture is 4 MB. So even if your game has the equivalent of 8 novels of text, it only uses the same amount of memory as a single 2k texture.

You could spend days or weeks of development time adding (and maintaining and debugging) SQLite code, etc. In the big picture, you’ll probably get more bang for the buck by spending that development time elsewhere such as improving player controls, or tweaking mechanics, or even polishing the quest dialogue content itself.

4 Likes