How to save and load an array of floats

Questions & Answers
1

Context:
I am working on a project where I have an array of floats (assigned to a tilemap)
What I’ve been trying to do is to easely (and efficiently in performance) to save that array into a file then loading it back again.

So far I managed to make it work by saving the floats into a .txt file separated by “/” and then loading it back again ignoring those slashes. The problem is that when I save a relatively small amount of floats (250000 floats) it takes a couple of minutes to save it, (strangely enough, it loads instantaniously) and I am planning to expand it to millions of numbers. You can see where the problem is.

So what I need is either a way to directly save the array into a .txt and then load it back again
or a way to efficiently save an array of floats with slashes between them (to separate them)

Here is my code:

private void SaveTilemap()
{
    string saveFile = "";
    saveFile += regionMapSize.x + "/" + regionMapSize.y + "/" + regionsize + "/" + "

";
for (int regionX = 0; regionX < regionMapSize.x; regionX++)
{
for (int regionY = 0; regionY < regionMapSize.y; regionY++)
{
for (int tilemapX = 0; tilemapX < regionsize; tilemapX++)
{
for (int tilemapY = 0; tilemapY < regionsize; tilemapY++)
{
saveFile += tilemapRegionValues[regionX, regionY, tilemapX, tilemapY].ToString() + “/”;
}
}
}
}
}
private void LoadTilemap()
{
print(“Loading tilemap”);
int index1 = 0;
int index2 = 0;
string save = File.ReadAllText(Application.dataPath + “/Tilemaps/save.txt”);

    index1 = save.IndexOf("/");
    regionMapSize.x = int.Parse(save.Substring(0, index1));
    index2 = save.IndexOf("/", index1 + 1);
    regionMapSize.y = int.Parse(save.Substring(index1 + 1, index2 - index1 - 1));
    index1 = save.IndexOf("/", index2 + 1);
    regionsize = int.Parse(save.Substring(index2 + 1, index1 - index2 - 1));
    float[,,,] loadTilemap = new float[regionMapSize.x, regionMapSize.y, regionsize, regionsize];
    int lastIndex = index2 + 5;
    int index = 0;
    for (int regionX = 0; regionX < regionMapSize.x; regionX++)
    {
        for (int regionY = 0; regionY < regionMapSize.y; regionY++)
        {
            for (int tilemapX = 0; tilemapX < regionsize; tilemapX++)
            {
                for (int tilemapY = 0; tilemapY < regionsize; tilemapY++)
                {
                    index = save.IndexOf("/", lastIndex + 1);
                    loadTilemap[regionX, regionY, tilemapX, tilemapY] = float.Parse(save.Substring(lastIndex + 1, index - lastIndex - 1));
                    lastIndex = index;
                }
            }

        }

    }
}

And by the way I separated my tilemap into “regions” to optimise it.

Don’t hesitate to ask any questions.

2

My assumption here would be that the bottleneck is actually the memory allocation you have to do to create all the strings.
Afaik C# creates a new string each time you perform a string concatenation → each time you have a new allocation.

I suggest you start the Unity Profiler to see which action takes up all the performance here. (you can do this with a lot less numbers for tests)

In case the string concat is the slowest thing check this link. There someone describes the performance implications of string concat and that using string builders can save ~40% time here.

Apart from that my suggestion would be: if you do not need the values in a human readable format save it in a binary file. This should be way faster than doing this complete string build thing you are currently doing.

For this you simply have to put your data into a Serializableobject (class or struct) which you then can save to a binary file with a binary formatter.

Why should this be faster?

Assume you have a float which is 4 bytes and the number is “0.123456789”
To write this in binary you need 4 digits. To write that in string is 11+1 seperator digits which is at least 12 bytes and that only if we assume that you have ascii encoding.

Let me know if anything was unclear or you have further questions regarding this.

3

Like Captain_Pineapple said, the way you create your string is horrible inefficient. You said you have 2.5M values. So if each value has a string representation of about 10 characters on average, it means the final string has a size of 25MB. Though you concat one value at a time. So each time you create a new string that is longer than the previous one. That means you create tons of garbage, in fact you create 2.5 million strings and each one is getting longer. So the total amount of memory allocated would be a geometric series. Sum(N*10) for N from 1 to 2.5M. That’s in essence the 2500000th triangle number multiplied by 10. That’S about 31 terra byte that you’re allocating. Luckily you don’t need all that memory at the same time. However the GC has to kick in thousands of times to clean up all those string instances you don’t need anymore. So the allocation of those 2.5 million strings and the garbage cleanup takes those minutes you’re seeing.

IF you really want / need to use such a custom text format, you should use a StringBuilder to create your final string. An alternative which is also not great but would still be way faster is to open your file stream and write value by value directly into the file. This is also not fast and causes a lot of file IO operations, but would be orders of magnitudes faster than your current approach ^^.

If you don’t need the data to be human readable (and I guess you don’t, nobody is reading through a flattened 4d array with 2.5 million values) I would suggest to just use the BinaryWriter / BinaryReader. This would be much simpler than your current appraoch anyways. I would not recommend using the binary formatter as it is not meant for such usage and microsoft itself does not recommend using it because it has many security issues.

You can roll your own binary format just like that:

private void SaveTilemap(string aFileName)
{
    int fileVersion = 1;
    using (var fileStream = System.IO.File.OpenWrite(aFileName))
    using (var writer = new System.IO.BinaryWriter(fileStream))
    {
        writer.Write("YourFileMagic");
        writer.Write(fileVersion);
        writer.Write(regionMapSize.x);
        writer.Write(regionMapSize.y);
        writer.Write(regionsize);
        for (int regionX = 0; regionX < regionMapSize.x; regionX++)
        {
            for (int regionY = 0; regionY < regionMapSize.y; regionY++)
            {
                for (int tilemapX = 0; tilemapX < regionsize; tilemapX++)
                {
                    for (int tilemapY = 0; tilemapY < regionsize; tilemapY++)
                    {
                        float value = tilemapRegionValues[regionX, regionY, tilemapX, tilemapY];
                        writer.Write(value);
                    }
                }
            }
        }
    }
}
private void LoadTilemap(string aFileName)
{
    try
    {
        using (var fileStream = System.IO.File.OpenRead(aFileName))
        using (var reader = new System.IO.BinaryReader(fileStream))
        {
            var magic = reader.ReadString();
            // check your file magic to identify your file, so you can be sure
            // you access the right file
            if (magic != "YourFileMagic")
                throw new System.Exception("Wrong file format");
            // check your file version in order to be future proof
            var version = reader.ReadInt32();
            if (version != 1)
                throw new System.Exception("Not supported file version");
            // read our own 
            regionMapSize.x = reader.ReadInt32();
            regionMapSize.y = reader.ReadInt32();
            regionsize = reader.ReadInt32();

            tilemapRegionValues = new float[regionMapSize.x, regionMapSize.y, regionsize, regionsize];
            for (int regionX = 0; regionX < regionMapSize.x; regionX++)
            {
                for (int regionY = 0; regionY < regionMapSize.y; regionY++)
                {
                    for (int tilemapX = 0; tilemapX < regionsize; tilemapX++)
                    {
                        for (int tilemapY = 0; tilemapY < regionsize; tilemapY++)
                        {
                            float value = reader.ReadSingle();
                            tilemapRegionValues[regionX, regionY, tilemapX, tilemapY] = value;
                        }
                    }
                }
            }
        }
    }
    catch(System.Exception e)
    {
        // handle errors here.
    }
}

Note: In the past I always forgot to include a proper version code in the file header. However this is really important. It’s not really to make your current code future proof but to make your current data future proof. When you decide to change your format in the future, there may be people out there upgrading their game / application to a newer version. So they may have data stored in the “old” format. The new game / application of course could support loading both formats or at least read the old format and convert it into the new one. However to do this you need to be able to distiguish the different formats.

ps: Note that multidimensional arrays are not really great for performance reasons. C# .NET implements them internally as a flattend array and all the index calculations happens under the hood. This is what makes the array quite slow. For each and every read or write operation the flattened index has to be recalculated. The framework will also check all indices for all dimensions against the set bounds. When you manually flatten the array by simply using a one dimensional array and calculate the flattened index yourself, the overall performance would be better. Also you can omit certain range checks since you have a better insight into the actual structure. I almost never use multidimensional arrays, though it’s a decision you have to make for yourself. Run some tests and decide for yourself.

In some cases using a jagged array (an array of arrays) can be a better approach, especially when memory is low and the array is large. This seems counter intuitive since a jagged array does require a bit more memory than a flattened array. However a flattened array requires a single continuous memory space while a jagged array consists of several smaller arrays. Sometimes a hybrid solution may be the best approach.