Dealing in the Billions.

I don’t know whether I mean UK or American billion so I’ll specify: 1x10^9 or 1,000,000,000

This is the kind of number I need to be able to handle. Imagine we have a 1000x1000x1000 3D grid. This will have 1x10^9 individual points, I need each of these points to contain 2 or 3 pieces of data, one is likely to be an integer, another a float and the last one is undecided.

The problem here is that my current method of building this grid is to assign an index, using an algorithm, to every point. The purpose of this is so that Unity can simply run through an array (of length 1x10^9 (which is obviously too big)) and assign the data to every point as it goes.

Now, my main problem here is not speed, but memory. Is there any way I can deal with that many values? Can I split the 1000x1000x1000 grid down in some way and manage each segment at a time so as not to overload memory?

I’m simply looking for suggestions as how to best deal with extremely large scale arrays etc.


It would help if you detailed a bit what kind of project you’re doing, and why you need 1000x1000x1000 grids. Are you working with voxels, or a tile-based game like Civilization, for example.

The thing is, do you need ALL of the 1000x1000x1000 structs all the time? What I would suggest to you is paging. You would keep that huge structure in a filesystem repository (or a database if it is more in tune with your project), and then I would load in memory pages of the data while you’re working with it. If you really need all of them, I would separate this structure in a octree-like structure and load/unload each cube in memory, do what is needed, persist it, and then load the next one.

I’ll wait for you to detail your project a bit more to extend this answer based on your needs.

How about dealing in the Trillions? Really. See the paper:

If you have a lot of empty space in the grid, I’d recommend implementing it as a level set. Basically you’d have a grid that contains grids - and if an entry on the root grid is completely empty, it’s just a null value instead of an actual grid.