Is there any way around storing my game's entire database in memory at runtime?

Unity Engine
1

I’m working on a simple RPG, and I need a way to add and edit NPCs and quests/story content by hand. Thus far I’ve been doing this by making ScriptableObjects with custom editors for each editable data type, and dumping them in a subfolder within Resources. The first time the player starts a new game, I manually iterate through every single file in Resources/, check if it’s a ScriptableObject containing stuff I want serialized, and add it to my game’s “database,” which is just a set of Dict<string,NPC> or <string,Quest> or whatever else I need to store. At runtime I access everything via its string ID, and saving/loading the game is simply a matter of packing or unpacking the dicts into lists and saving them in some arbitrary format (I’ve been using JSON).

This works fine, and it’s let me prototype the entire game out, but it has a big problem: I end up holding every active quest, character, and dialogue file in active memory 100% of the time. It’s not a performance concern, since this data is very simple (mostly just tons of strings and integers), but it feels like an absurdly sloppy workaround.

That having been said, I’m genuinely not sure there’s a better option. I reference both NPC and quest data a lot, which means that if I stored them in a binary or JSON file and simply kept a reference to the file in memory, I would be opening and parsing that data so frequently that it would cost more in performance than what it cleared up in memory.

So is there anything I can be doing differently? This feels like the definition of premature optimization, but holding your entire game’s database in active memory feels like such a humongously awkward thing to do that I keep thinking I’m missing something obvious.

2

I think it depends on how much data you’re talking about, and the constraints of the platform. On a PC/Mac standalone build you’re probably better off just loading everything into memory when the game starts unless you’re talking about several GB of data.

3

If you’re loading them from Resources anyway, you could modify the entries to only be loaded when needed. Something like:

public class LazyResource<T> where T : UnityEngine.Object
{
    private T _loaded;
    private string _path;

    public LazyResource(string path)
    {
        _path = path;
    }

    public T Value
    {
        get
        {
            _loaded = _loaded ?? Resources.Load<T>(_path);
            return _loaded;
        }
    }
}

But whether doing that will improve performance depends on how much data is involved and how much you’re likely to need loaded in a single play session.

4

This is a PC/Mac/Linux standalone game with relatively low performance requirements, so I think you’re absolutely right. It’s not a hard & fast indicator, but as a quick test I just added 15,000 character records and 15,000 randomly-generated quests to the database class (our projected final asset count is around 200 characters and 50-100 quests), and the profiler only jumped in size by about 10mb.

1 Like
5

This is a great thing I wish more engineers did. Imagine even if you had TEN TIMES that, it still would only be 100mb, so easily within reach of any PC environment.

Bravo to you for identifying that any attempt to optimize here would just be a waste of your time.

Now you can run ahead with your game!

2 Likes
6

Yeah, thank you for the quick feedback! I know it’s a terrible habit, but I am a master at getting fixated on premature optimization and over-engineering what should be simple systems. :slight_smile:

2 Likes