Any suggestions on my database system?

Unity Engine
, , ,
1

Just looking for a second pair of eyes on my “database” UPDATE. I’m currently using a local database for my game for more complex entities, to see if there’s a better way or if I may be missing something.

  • I load / create all my entities (anything that exists as a “thing” in my game) from their system: EntitySystem<T>
  • I flag a entity as dirty if any of the property data changes
  • In The Entity System I iterate over the collection of entities and check isDirty
    (what I’m looking for suggestions with the below code)
  • I spin up a bunch of tasks to call my Update function on the entitys with the flag

Tasks are almost perfect in this case, I can run them and forget about them. All of the data is just fired off and saved and I don’t need to worry about it. There’s also no real instance where I’m worried about storing data immediately, if there’s any important things I need to store I would call the update manually.

I

    //EntitySystem<T>
    public void UpdateData()
    {
        tasks.Clear();
        for (int i = 0; i < Datas.Items.Count; i++)
        {
            if (Datas.Items[i].isDirty)
            {
                var task = Task.Run(() => 
                {
                    databaseSystem.UpdateData<T>(Datas.Items[i].Id, Datas.Items[i]);

                    if (Datas.Items[i].isDirty)
                    {
                        Datas.Items[i].isDirty = false; 
                    }
                });

                tasks.Add(task);
            }
        }

        Task.WhenAll(tasks).Wait();
    }

    //DatabaseSystem
    public void UpdateData<T>(long id, T data) where T : class
    {
        if (data == null)
        {
            Debug.LogError("UpdateData failed: data is null.");
            return;
        }

        try
        {
            bool success = auto.Update(GetTableName<T>(), id, data);
            if (!success)
                Debug.LogError($"UpdateData failed for ID {id}.");
        }
        catch (Exception ex)
        {
            Debug.LogError($"UpdateData exception for ID {id}: {ex.Message}");
        }
    }

    //GameSystem (that constantly polls the systems)
    bool storeData = true;
    private void Update()
    {
        if(storeData){
            foo.UpdateData();
            bar.UpdateData();
        }
    }
2

Honestly, this is so generic I have no idea what it’s supposed to do. It flags systems as dirty and updates “datas”. :thinking:

Well, for what its worth, “data” is a word that has no “s” plural unless you refer to clones of Commander Data. :wink:

And if this runs per update, and you are using the Entities framework, it pains me to see you’re spawning a Task every frame and possibly do something that’s just like snap instantly updated if we’re talking about modifying a few in-memory database records (aka fields??).

And the obvious question being: why not use burst-compiled Jobs to do those updates?

Unless profiled otherwise, this looks overengineered to me.

2 Likes
3

It’s not using dots, an entity is a data object held by an identifier. Update is updating the db record so I would want to instantly update to keep my data synced, as this is acting as any save file for entities.

Burst is more efficient at heavy cpu loads, not I/O. I don’t see why I would use burst in this case it wouldn’t improve anything.

This is basic async programming, I don’t see how this is over engineered at all.

4

Is DatabaseSystem.UpdateData supposed to be thread-safe? Because Task.Run queues the task to be run on the ThreadPool. You can’t use almost any Unity APIs if you use Task.Run, and even beyond that have to be really careful to make everything thread-safe.

Also, avoid Task.Wait - you almost never want to use that method (I have never used it in my life). That causes the whole thread to be blocked entirely until the asynchronous task completes. So if it takes one second for that task to complete, then your entire game can visually freeze for one second.

You’d probably want your EntitySystem<T>.UpdateData method to look more like this, so that it’s non-blocking:

class EntitySystem<T>
{
	public Task UpdateData()
	{
		tasks.Clear();

		var items = Datas.Items;
		for (int i = 0; i < items.Count; i++)
		{
			var item = items[i];
			if (!item.isDirty)
			{
				continue;
			}
			
			item.isDirty = false;
			var updateDataTask = databaseSystem.UpdateData<T>(item.Id, item);
			tasks.Add(updateDataTask);
		}

		return Task.WhenAll(tasks);
	}
}

And then you’d probably want your update loop to wait until the previous asynchronous operations complete in a non-blocking manner, before executing them a second time:

async void UpdateLoop()
{
	while(!disposed)
	{
		await Awaitable.NextFrameAsync();

		if(storeData)
		{
			try
			{
				var fooTask = foo.UpdateData();
				var barTask bar.UpdateData();
				await Task.WhenAll(fooTask, barTask);
			}
			catch(Exception exception)
			{
				Debug.LogException(exception);
			}
		}
	}
}
2 Likes
5

It doesn’t matter on my side, this is just for DB UPDATE so theres no issues I can see happening.

Forgot my update wasnt async, you’re right.

This is great thanks!

    public Task UpdateData()
    {
        tasks.Clear();
        for (int i = 0; i < Datas.Items.Count; i++)
        {
            if (Datas.Items[i].isDirty)
            {
                var task = Task.Run(() => 
                {
                    databaseSystem.UpdateData<T>(Datas.Items[i].Id, Datas.Items[i]);
                    Datas.Items[i].isDirty = false; 
                });

                tasks.Add(task);
            }
        }

return Task.WhenAll(tasks);
    }
...
    bool storeData = true;
    async void Update()
    {
        if(storeData){
            await Awaitable.NextFrameAsync();
            await Task.WhenAll(for.UpdateData(),bar.UpdateData());
        }
    }
1 Like
6

If the issue doesn’t relate to Unity Entities please don’t tag it with Entities, it is misleading.

1 Like
7

Burst in isolation, I would agree, it’s just a compiler. What can it do really other than optimize? However, Mono is also a terribly slow compiler. And it’s not just about Burst, it’s also about unmanaged memory and this is the reason why your statement is incorrect.

SisusCo gave good advice so I’ll leave it at that.

8

I don’t know who set it, but it wasn’t my doing.

Whomever set it removed it, it seems.

9

primarily used to improve performance when working with Unity’s Job System, allowing for parallel processing and significantly faster execution, especially for computationally intensive tasks like physics calculations or large data manipulation

Please feel free to explain why my statement is incorrect.

10

Thanks. I think it’s possibly the doing of some moderators or the forum’s auto-mod.

11

Well says it right there: large data manipulation
When it’s good for large data manipulation it’s also good for small ones. And databases are all about large data manipulations, right?

Anyway, the real reason here is that unlike a typical C# heap allocated class that resides at random points in your memory, entities bundles them together in unmanaged memory. That not only means the data is faster for read/write access but Burst can also skip the additional overhead that come with C# classes and sanity checks which would slow things down.

I’m not sure you need that but since you made a statement and asked, here’s your answer. If you are really interested, take a deep dive into some of the concepts and a good primer on this page: Introduction to the Data-Oriented Technology Stack | Unity

12

Data manipulation refers to the process of modifying, transforming, or processing data. “data manipulation” usually refers to the logical modification of data, while “I/O” refers to the physical operations of reading/writing data (all I’m doing is writing) and even if the DB is manipulating anything for an update I don’t have any control over that so it wouldn’t matter.

But either way I don’t see how or why you’re saying I’m incorrect. Bust or DOTs won’t improve this process, and I already understand the stack and ECS, maybe me using entities with identifiers should have been a clue I’m rolling my own data oriented design?

So far in my question I’ve gotten “your code is over optimized” and now I’m getting “refactor your entire code base, all your systems and handlers, because it potentially could be slightly faster to save your data”.

A quarter of what makes good developers is knowing when to use or not use tools.

13

I think I changed that. The fact that you mentioned “complex entities” and EntitySystem<T> and “data” repeatedly throughout gave the impression that it was about “data-oriented” Entities Systems’ code.

Well no, you mentioned “database updates” first and foremost. Nothing really is clear from your original post as you leave out important details such as this, or what the database is all about in the first place and what data and types it stores, how many records there may be and how big they are, for what purpose that data is used and most importantly, what exactly an “update” entails.

The code you posted is basically just the high-level for loop that updates the systems of which we know nothing of and yes, that seems like over-engineering because you have to have a good reason for designing your own Task-based Entities system rather than using bursted Jobs - that rationale is also absent.

14

The code has all the info, it has entities with identifiers, what information “isn’t clear“. Why would I explain the DB what exactly would any of that information give you? It’s a generic function that updates the generic data to the DB. There’s no reason for me to add any of that info as it doesn’t add anything.
Read the code, it’s very clear what it’s doing.

It’s not my job to explain to you what a DB operation is, if you don’t know then you don’t need to try and answer questions when you don’t have answers. Especially if it’s just “your code is over optimized”.

“You should use burst jobs”
“Why, when I have a great system working?”
“Because, burst jobs”

You’ve already assumed wrong on the entities, twice now. Why are you bringing them up again for a third time? Nowhere in my code am I using dots. So you talking about the entity system with burst again is adding nothing to the conversation, it also shows you don’t seem to understand what this conversation is about.

You understand ecs is a design pattern? You don’t have to use dots to make a data oriented design. async tasks is what I need for I/O. Burst wouldn’t optimize anything or help or fix anything the boxing, unmanaged, casting, and refactor wouldn’t yield anything, it would just increase development cause headaches and mess the team up. Performance isn’t an issue and I don’t ever see it being one, so I’m not going to refactor my code when it’s doing exactly what it needs to do for minimal gains.

For someone who’s saying the “rational is absent” you haven’t shown a single ounce of understanding, you keep getting confused over ecs and dots (to the point where other users are complaining at me about discussion tags) and you seem to not understand the basic code when it’s self explanatory in what it’s doing.

15

It’s clear that it loops over and makes Update calls. But whether that is a good approach or not is largely dependent on what those Update methods actually do.

Exactly, because to understand what you’re trying to accomplish with your code it matters a lot what that Update() method is actually doing. And how many items we’re talking about, how common the “dirty” flag gets set, among other things.

I’ll try to explain based on your code:

  • for every record in the table
  • check if the record is marked dirty
  • if so, run a Task
  • Task calls UpdateData() on the databaseSystem with a single record
  • databaseSystem does nothing but error check and call auto.Update, passing the data along, where “auto” presumably is the gameSystem but it doesn’t match the signature of gameSystem’s Update method that you posted
  • gameSystem then, if it should store data, calls UpdateData() on some foo and bar we know nothing of
  • Wait for completion of all task

So you’re having an EntitySystem calling a DatabaseSystem calling a GameSystem (or not?) and you pass the record along through each system. None of the why you’re doing that is clear, and at the end you stop short of telling what UpdateData is actually doing.

EntitySystem and DatabaseSystem share responsibility of “updating” data, and the whole process is stretched over several classes at least and works off of a single record each.

The single record updates makes it highly inefficient compared to running over the data in sequence as you did in the EntitySystem update that generates tasks. This tells me that that whole method ought to be moved entirely into DatabaseSystem, especially given how little the “database system” actually does.

Overall, this system is strangely complex, works off single records, and the purpose of why its set up like this remains a mystery.

It would make more sense if you worked off a table, iterate over the table records to gather the id of each “dirty” record, then spawn a single Task that does await WriteRecordsToDiskAsync(listOfDirtyRecords). That’s what I would expect from a clean and simple “database system”.

Even if “update” is not actually writing to disk, I would question why there has to be a Task running for each and every dirty record. Here it matters what the worst case scenario will be. If there is ever going to be a case where you would spawn thousands of Tasks performing invididual “updates” then this could be disastrous to performance and memory usage, more so depending on the actual record which may be just an int or it may be hundreds of bytes and possibly even contain object references.

Hence why I’m saying this code is over-engineered. Perhaps there is a purpose to separating the systems you did, but without knowing any of these details, it remains a mystery.

2 Likes
16

You don’t understand what it’s even doing and you’re trying to say it’s over engineered? All the details are in the code, you don’t seem to understand what an UPDATE operation is or that I’m not working with tables or records as

for (int i = 0; i < Datas.Items.Count; i++)

Is clearly a collection of objects, and I even wrote

In The Entity System I iterate over the collection of entities and check isDirty

Nowhere in my code am I referencing the Entities namespace but you continued to assume this is the ecs framework. Now you’re continuing to assume the codes doing xyz without understanding it.

It’s 50 lines of code, why are you saying my database system is “calling a GameSystem”. I wrote a comment //GameSystem (that constantly polls the systems).
And in the database system
bool success = auto.Update(GetTableName<T>(), id, data);
Doesn’t have the same signature or return type as void Update() a unity built in method, so I can’t understand why you would think that’s calling that same update(). I really can’t help you if that basic of a line is confusing to you. None of the code is a mystery if you actually read it.

I would highly suggest in the future if you don’t understand don’t comment.

17

So what does auto.Update do? As everything is funneled into this call it would have been great to get an explanation.

18

I specifically pointed out many of the important details that are NOT in the code you posted.

And yes, I call it over-engineered for as long as it loops over some data to generate potentially hundreds of Tasks that each call an update method that may or may not call another update method on some “database system” passing in the data that it operates on which, as is typical for databases, is just a container type for primitive fields (bool, int, float, string, datetime, byte array, …).

In my last post I have not even implied that you use the Entities package.

Yeah I noticed this and I tried to point out that this is confusing, and the code is likely unrelated due to the signature differences, yet even after 17 posts it remains a complete mystery what your update methods actually do.

Ultimately it boils down to this:

What the heck is this method doing? Give us an example! Because in all likelihood, it matters what this does.

For instance, you still haven’t said whether this “update” is writing to disk.
Nor how many “dirty” updates there can be at most in a single frame.
Nor how many “dirty” updates there will be on average per frame.

This is essential information for us to provide the feedback you asked for.

19

I truly wonder what an Update is doing that takes a

TABLE NAME
ID
DATA

On a question about updating a database…

I find it wild you can’t figure that conundrum out, then go on to tell me it’s too complex.

I highly suggest taking some time to learn db operations since this seems to elude you.

No you finally figured it out after 3 tries.

It’s not “likely” as it’s not related at all, so why do you keep saying it’s calling an update on the GameSystem?

So if it doesn’t match why would it be calling it? If it doesn’t match then clearly it’s calling a Update function that takes a TABLE NAME any competent developer should be able to figure that one out instead of trying to bring up game system.

From this entire post it’s clear to me that you guys have issues with simple blocks of code, and that you either need to stop assuming you know what’s happening or either take the proper amount of time to actually read without assuming or immediately trying to answer questions that you don’t understand.

All of my information was presented and easily understood from the code, you trying to make excuses that I didn’t explain the code is just trying to blame me for you not being able to read 50 lines by telling me that the game system is being called when arts and signatures don’t match.

20

In a SQL database the UPDATE command is well defined.

You however are building your own database system, we have absolutely zero idea what “Update” means in the context of your self-built database.

For all we know it could literally convert the record into a byte stream, encode it as UTF32 string, send it to a webservice, translate it to Japanese, Brotli compress the result, and then save it to Json files each guaranteed to be exactly 64 bytes long, and padding the last file with ‘bell’ characters.

Or it could literally just do:

isDirty = false;

So how about you step up and post an example of what your “data record update method” is actually doing?

No, absolutely not.

And you keep repeating your own misconception of what was being said or implied while refusing to answer questions. It tells me you’re the one not willing to open your mind and be receptive about feedback.

1 Like