# Efficient massive heightmap storage

I am working on a project that creates procedural solar systems with planets with a radius of about 12720 units. I use a Noise function to generate the terrain + Quad Tree for the LOD.

My goal is to store the generated heightmap for the player to edit as he sees fit. A big problem with that is the amount of storage it takes up to have 11 levels of detail of meshes 16 vertex wide.
To achieve a diecent detail on a player level it’s about 4^1116 vertexes i need to store just for one side of a cubed sphere. For the entire planet it will be about 4^1116*6 vertexes.

I did some testing and writing the float heightmap , as a 32bit color to a texture, the whole map took about 384 MB. I can live with that for just one planet, but i have over a 100 planets.

What can i do to bring down storage to a more realistic level ?

You are using floats for the heightmap? That’s 4 bytes per height point, which is quite a lot.
If you just use two bytes (a ushort), you can store 32768 height values (from 0 to 32,767).
So, if each unit was a meter, you could have mountains 32,767 meters high. That’s VERY high. If you don’t need things that high, count them as 1/10th of a meter instead…that gives you mountains that are 3,276 meters high, in increments of 1/10th of a meter. this will give you a much smoother landscape also.
3,276 meters may not sound that high…but it’s actually not that bad. May be worth a shot.
Anyway…at that point…you cut your storage in half.
Another options…and this is up to you really. You may want to save it as a compressed height map, zip it or something. That could save a little space, but in my experiences height maps do NOT compress very well because of the huge variety in data. There aren’t many patterns.

You mentioned 11 levels of detail meshes, 32 vertices wide. This means your total heightmap size is…
32 * 11 = 352, so each side of the cubed sphere is a 352x352 heightmap
You need 6, so for an entire planet you need 2112 across. Why not just round down to 2048x2048?
That gives you 4,194,304 heights per planet. At 2 bytes a pop…8mb per planet.
Is that right, or am I completely missing something about the total heightmap sizes needed for a planet?

Fractals, baby, stored as math formulas. Another option is storing only the relative difference between vertices, not absolute heights, but for that you’ll need kind of `keyframe` vertices that are still absolute.

You can use a random seed, and just store the seed so the terrain is regenerated the same way every time, then store the player edits on top of that. That way the unedited planets will take essentially zero space.

–Eric

Yeah, considering how small a proportion of the space users are likely to actually edit, that’s a great idea.

In and of itself this won’t save space, will it? However, considering that the height of a vertex is unlikely to be radically different to that of adjacent vertices you could use that to your advantage in combination with jc’s suggestion and store only 8 bits for each.

Additionally, I’m sure you could find some form of data compression algorithm that would help.

Also, considering that you’ll only be looking at up to maybe 4 of the highest detail maps at a time (if you’re at a corner where 4 meet together, and assuming you can’t look that closely at multiple planets at once) consider loading and unloading them as needed. Load as the player gets near, unload as the player moves away, saving updated data back to disk if relevant.

Then, also have some on the fly (de)compression going on. With the above in mind, you can probably move the whole loading/unloading task to another thread and then just use off-the-shelf libraries to do the (de)compression.

Coming from a completely different direction, you could also store less verts for less detailed terrain. Instead of a grid you could store an arbitrary mesh, and add/remove verts to areas as the user edits it. Also, space will be used where and as detail requires it, essentially giving you a kind of natural adaptive compression. Also also, you probably wouldn’t need to store distinct LOD meshes - I’m sure you could run a filter over it and generate them on the fly (I think this would be a high-pass filter?). This would require a fair bit of engineering, though!

So let me make the calculations more extensive.

one Chunk of a Cube’s side is 16 pixels wide,
each Chunk when you go a level down spits in 4 chunks also with 16 pixels wide,
that makes a total of 4 ^ 11 = 419 430 4 chunks a side,
that makes the width of all the chunks to sqrt(419 430 4) = 2048.

each chunk has is a 16 wide so that makes the with of all the vertex in a cube size to 32 768.
This is outsource the most detailed level the 11-th.

that makes an area of 32 768 * 32 768 = 1 073 741 824 vertex per cube side.
If I were to store that with just 2 bytes per vertex it goes like this:
2 147 483 648 bytes => 2048 MB or raw data.

if I were to take it to 10 levels of detail not 11
it would take up 4 ^ 10 = 1048576 chunks => 1024 chunks wide
1024 * 16 => 16384 vertex wide => 16384 * 16384 => 268 435 456 vertex area
268 435 456 * 2 bytes => 536 870 912 bytes => 512 MB raw data.

My goal is to save the heightmap so I can achieve more realistic terrain with multiple added noises and decals, witch adding at runtime may not be as fast as I want.
Like Spore, but with much larger planets

Why not try image storage formats PNG, JPEG ect, could your heightmap be converted to grayscale images?

You could also store a low resolution version of the planet, so you have some control over its general shape, but then fill in the fine-details with some fractal algorithm as you load it . thus it’ll look almost like it did last time. You can then edit parts, to some degree, and keep the edits, provided the user doesn’t mind things not being 100% accurate.

In terms of trying to implement compression of any kind, it is generally always better to have knowledge of how to generate the result, e.g. stored math formulas, or seeds, or whatever, rather than not having that knowledge after the fact and having to try to figure out how to compress the result of the generation. It usually takes a loss less space to store information about how to generate, than how to compress what was generated.

how to you save to a JPG Image ? I only know of EncodeToPNG and save the bytes to a file.

From a byte array. You need a jpeg saver script. There is one floating around on the forums or maybe on the web somewhere. I have one in my TextureHelper asset. Lets you save a jpeg.

While I hope you find your answer, and a mighty efficient format you’ll have, i think the random seed is the most tried and trusted for generation of all this data, your method for storing data could get elaborate and impressive, and that plus the lighter expectations on the hardware/possible storage space could translate into the edited sections having more features, or an ability to subdivide when you can edit (like player edited planets could get elaborate while built on the procedural base and your storage needs would start from a very low amount and, depending on how you figure to store your data, be able to plan for) - more out of less?

I should have read more beyond eric’s post, i just looked up to see i’ve smooshed the thread into a redundant block of text. Well good luck anyway!