Storing arrays: best practices

I have a 2D tilemap. What is the best way to store that data?
Let’s assume the map has layers which some are pure data like i.e. height, type id, etc. and some are references to objects like any foliage or structure present on a tile. In OOP I would have an array of Tiles (class) with all the data or multiple arrays with just one of the data types.
Keeping Tiles as separate entities seem easiest. But then searching would be a nightmare. Sometimes I would need to recreate the array i.e. for pathfinding so it seems it’s the wrong approach.
Looks like DynamicBuffer doesn’t be any better.
Keeping an array in the system seems wrong also as the system shouldn’t have any data.

How should I approach this type of structure in DOTS?

This might help a bit. It helped me understand NativeMultiHashMaps better:

1 Like

Keeping Tiles as Entities should be the way to go. You can maintain a sorted data structure somewhere else.

So keeping local data in a system is fine. It’ll be weird when you need to share that data with other systems. In that case, you can make use of the Singleton helper methods (i.e. GetSingleton, GetSingtleEntity, etc…) to keep some global data around.

1 Like

Yeah, that’s the problem. The map will be used by virtually everything. As a reference look at the games like Rimworld. Everything happens on the map: units moving, items spawning, plants growing.

Thanks, but he keeps data in the system. Anyway, that gave me an idea. Quadrant system from the video but as an ISharedComponentData. Then I should be able to look up quickly everything without persistent data. The problem that the solution has is a high fragmentation of data that depends on quadrant size.

Thanks. If anyone has any other idea please share!

There’s been quite a few threads about sharing containers between systems, usually as they relate to JobHandle management. If you don’t like storing containers on systems, you can instead store them on class components. Or you can create a more advanced mechanism that handles the JobHandles more implicitly. I provide such a mechanism in my framework if that interests you: Latios-Framework/Documentation~/Core/Collection and Managed Struct Components.md at v0.2.1 · Dreaming381/Latios-Framework · GitHub

If the array won’t change after created (or seldom change), I think just flatten your array and store it as blob array in a blob asset reference, and store that blob asset on a singleton entity. This way you can access it in jobs safely.

Seems like a reasonable solution, but I’m seeking a minimal dependency solution right now.

Sadly my data will change a lot. I’m still wondering if rebuilding blob asset on changes wouldn’t be fast enough.

Then I think your best bet it just make tiles Entities, have a HashMap somewhere where you can access these entities by coordinates. For pathfinding you’ll need a different data anyway (for marking nodes etc…) so you will build a temp array for that using the HashMap.

In my game I represent my tilemap in a dynamic buffer and it works great. Why did you discount it in your op?

1 Like

As far as I understand I can’t use Burst with dynamic buffers. Can I?

I have another idea. What if I’ll make one system for map handling? Pathfinding, searching, and other stuff. My concern is that it will be a nightmare to read and maintain but in theory, there will be no downsides if I’ll make a few jobs, each with a different query, in one system.

You definitely can use dynamic buffer with jobs. You can add to Entities.ForEach parameter or use BufferFromEntity

1 Like

Of course you can, they’re Burst compatible

1 Like

As others said yes you can use it in Burst, and it’s very convenient since you can include it directly in your queries for your map entity. Any time you need to associate an array with an entity Dynamic Buffers should be your first stop.

1 Like

Are there any disadvantages while using DynamicBuffers? I want to have maps 1024x1024 and bigger.

Main problem with DynamicBuffers is that there can only be one dynamic buffer per IBufferElementData type per entity unless you add a redirection array / list where it maps to multiple entities, each with their own DynamicBuffers.

Depending on the size of this list, if it’s small or contains small sized data, you can use the various value type FixedLists. However with a map size of 1024 or larger, that’ll be much larger than what a fixed list can support.

In that case, the final option is UnsafeLists. You can do truly magical things with unsafe containers like nest them 10 deep within each other but it’s now your job to keep track of them and they’re just pointers. Accessing them breaks the cache structure. It’s not recommended to use them in a nested format (like inside an entity or a NativeArray<UnsafeList<>>) in performance critical code.

I’ve used a NativeArray<UnsafeHashSet>() personally as a sort of “unique only” NativeMultiHashMap successfully in a IJobParallelFor but the performance wasn’t so good. It was only for loading so I didn’t care.

Personally, I would create a blobasset array, 2 dimensional, and populate it with entities of the tile data. If one or a few specific entities must change structrually, then remove all component datas and introduce a single redirection component data with the only value being the new tile data entity.

I have no clue how the performance of that will be though.

1 Like

Sorry but I don’t get your point? Or is miss worded.
Yes you can have only one type of IBufferElement per entity, but with many elements inside as you like.
You can not duplicate same BufferElement however. But same true is for IComponentsData.

But then entity can have multiple dynamic buffers of different types.
So for example one buffer is TilePlantsBuffer, with entity data.
A other buffer is TilePlayerUnitsBuffer, with entity data.
Etc.
So as you see, you can have many buffers, storing same data type.

I think OP struggles in general concept, how to store relevant data.

If each tile is an entity, it can have buffers of contained instances entities, like player units, buildings, plants etc. Consider map like Civilisation game series. Tile entity containing different type data, to store relevant info about tile.

Each entity tile can have reference to its neighbour entity. Or store index to flat array, of whole tile map. Then can access data, of flat array. For example needed for path finding.
Whatever is needed.

Then can process each entity tile in parallel, as required.

4 Likes

Is there any limitation to how long these buffers can get?

If stored on the heap, not in the chunks, they can be as big as memory permits.
I had entities in past with many thousands buffer size, no problem.

You just need to be aware, if you want to iterate often large buffers, for many entities.

How to store them on the heap instead of chunks?

I was wondering if I should just do a buffer of all tiles and have only one entity of map instead of millions of tile entities.

For example set bufferElement capacity to 0. Then after entity is created, resize entity DynamicBuffer to required size. I normally use uninitialized resizing, as I don’t need clearing memory of the buffer.

When you can use entities, use them. Jobs filtering is superior. However, is nothing stopping from having also references to tiles in components or buffers.
Or maybe store them in hash map. All depends, what you need to do with them.

Stress test and profile it.

1 Like