What is the best way to store and load big chunks of data, like a large array? Or is it even possible?
One way I can think of is to store the data in a text asset, then convert the strings to floats to fill an array at runtime. But that seems pretty inefficient. And I don’t want to spend a lot of time trying to set that up if it definitely won’t work.
The only other alternative I can think of is to just enter the data in a script like:
myArray = [1.0, 2.0, 3.0 …];
But that will get pretty cumbersome with 100s or 1000s of array elements.
Is there any other way in Unity to store and access external data?
That looks like way overkill just for storing an array or two. Plus I don’t know anything about databases.
Also, wouldn’t the data have to be stored on a server somewhere? In my case, I’d like the data stored as part of the project, as it will never change.
I’m converting an old Director project in which I stored the array data in text fields, which were loaded and converted to numeric arrays at runtime. If I were doing it in Director now, I’d store the data in binary files that would be loaded at runtime.
You could store them as Comma Separate Values to a file.
You would have to use the system filestream, and add the comma in manually, but extracting them would be a breeze.
Writing:
float[] values;
string toFile = "";
for(int i = 0; i < values.Length; i++)
{
toFile += values[i] + ",";
}
//Removes the last comma
toFile = toFile.Substring(0, toFile.Length - 1);
//Write toFile using Filestream here
to Write to file
Reading:
//Read code in as one big string
string fromFile;
string[] data = fromFile.Split(',');
float[] values = new float[data.Length];
for(int i = 0; i < values.Length; i++)
{
values[i] = data[i].ToString();
}
for file writing examples
It’s only one or two lines to read and write to files(although checking whether the files exist or making it asynchronous can take a little more).
Edit:
In the file the data would read:
“1.0,2.0,2.2,5.5,2.000,3.66,4.00” etc…
And you can use a second separator, like ‘|’ to store multiple arrays in one file, by calling Split(‘|’) and then processing the two pieces of hte array as listed before.
Thanks. That looks like exactly what I need. I don’t even need to worry about writing to a file, as I can create the file ahead of time.
One question, not sure if this is a typo. The data come in as a string, right? So why use ToString? I don’t understand this: values[i] = data[i].ToString();
Shouldn’t it be something like this:
Actually, I wasn’t thinking. I need to store an array of Vector3 values. Doesn’t look like there’s an equivalent to parseFloat for vector data (is there?). I’ll need to figure out away to delimit the data file. Maybe something like:
1,2,3|4,5,6|7,8,9|…
As suggested with multiple arrays, I could then split with “|”, then with “,”.
It looks like this will work, except that I’m not so good at translating the C# examples to JS. What’s the syntax for Split? It says it take a Char array as an argument, but there is no such type in JS. And it won’t take a String.
Also, will the file path I specify (in my Assets folder) be the same in the app that gets built (for iPhone) as in the editor?
Yeah the ToString was a typo, I wrote that in the thread and didn’t actually test it. It should be some sort of Convert.toSingle or float.parse if I recall.
If you are creating it ahead of time, you can place it into a Resources Folder (inside Assets so Assets/Resources/yourFile.txt) and then use Resources.Load(“yourFile.txt”). This file gets compressed and is not edittable once on the device however. Good for initial data, not so hot for save games or data that may be modified.
It will probably come in as a TextAsset, although I’m unsure.
For writing to an edittable location you need to use the Documents folder. I currently track it with:
It’s currently clunky and based on my project name, but it essentially removes my ‘[projectname]/data/’ from Application.dataPath and replaces it with Documents. There should be some posts on the forums to access it easier than my method.
It does come in as a TextAsset, which means a two-step process: Put the file into a TextAsset, then put the text of that asset into a string. Does that mean there are two copies of the data in memory? Or is the string just a pointer to the TextAsset text?
If .text is a copy, you can store it and TextAsset will be taken away at the next Garbage Collection.
If .text is a reference it will not add any extra memory for the two step process.
So long as you don’t duplicate it per frame or have each object store it you should be ok, and less than the actual Vector3 array you are creating with it(provided you use generally 1 or 2 digit precision).
A little off topic for this thread, but since this worked so well for getting data from a file, I thought I’d try it for saving image data.
I have an app that has 20 256 x 256 pixel images that I need to get pixel data from (GetPixels). Enabling Get/SetPixels on all these images makes the app too big. I thought I could get the data ahead of time (these images never change), save the data in text files, then get the data from the text files rather than the images at runtime. And it works. Problem is, the text files are way bigger than the images – a 1.2 MB text file for a 300k image.
Is there a way to save/read a non-text file? The data themselves are not that big. It’s converting them to strings that is the problem. StreamReader only works with text files. Is there a way to save and read the data in binary format, without converting back and forth to strings?
When you say you are converting to text format, are you just treating the raw bytes as if they were text or are you writing lines of real text with the pixel values? For example:-
0,0,0
128, 128, 0
...
I would imagine you are using the former approach. If so, the increase in file size is probably just down to the lack of compression on the raw pixel data.
I’m doing the latter. This is how I tried saving the pixel data:
function SaveImageData () {
var sw: System.IO.StreamWriter;
var theData: String;
var imageData: Color[];
theData = "";
imageData = theImage.GetPixels();
for (i = 0; i < imageData.length; i ++) {
theData += imageData[i].r.ToString("F3") + "," + imageData[i].g.ToString("F3") + "," + imageData[i].b.ToString("F3") + "|";
}
sw = new System.IO.StreamWriter(Application.dataPath + "/Resources/Image Data.txt", true);
sw.Write(theData);
sw.Close();
}
I know this is not the best way, but I haven’t been able to come up with a better solution. The increase in file size is the result of converting the data to strings.
I figured out a way to save image data without using strings or increasing file size. I saved all the data from the images, then was able to delete the images. The images were 300k, but the data files are 197k, so I saved about a third of the space. Plus, I don’t have to have all those images with Get/SetPixels enabled any more, just one to draw the image data into.
To save space, I converted the Color data (floats) to 1 byte integers. No problem since I’m used to dealing with color values that are 0-255 rather than Unity’s 0.0-1.0. And I don’t need alpha data for these images.
Here’s how I saved the image data (used only once to store the non-changing image data):
function SaveImageData () {
var i: int;
var xx: int;
var yy: int;
var FILE_NAME: String;
var fs: System.IO.FileStream;
var ww: System.IO.BinaryWriter;
var tt: System.Byte;
FILE_NAME = Application.dataPath + "/Resources/" + imageName + " Data.dat";
// Create the new, empty data file.
if (System.IO.File.Exists(FILE_NAME)) {
Debug.Log("{0} already exists!" + FILE_NAME);
return;
}
fs = new System.IO.FileStream(FILE_NAME, System.IO.FileMode.CreateNew);
ww = new System.IO.BinaryWriter(fs);
imageData = new Color[imageSize[0] * imageSize[1]];
i = 0;
for (xx = 0; xx < imageSize[0]; xx++) {
for (yy = 0; yy < imageSize[1]; yy++) {
imageData[i] = theImage.GetPixel(xx, yy);
i += 1;
}
}
// Write data
for (i = 0; i < imageData.length; i ++) {
tt = Mathf.RoundToInt(imageData[i].r * 255);
ww.Write(tt);
tt = Mathf.RoundToInt(imageData[i].g * 255);
ww.Write(tt);
tt = Mathf.RoundToInt(imageData[i].b * 255);
ww.Write(tt);
}
ww.Close();
}
And here’s now I retrieve the image data and draw the picture:
function DrawOriginalImage (destImage: Texture2D, sourceFile: String, theWidth: int, theHeight: int) {
var xx: int;
var yy: int;
var tempR: float;
var tempG: float;
var tempB: float;
var FILE_NAME: String;
var fs: System.IO.FileStream;
var rr: System.IO.BinaryReader;
FILE_NAME = Application.dataPath + "/Resources/" + sourceFile + " Data.dat";
fs = new System.IO.FileStream(FILE_NAME, System.IO.FileMode.Open, System.IO.FileAccess.Read);
rr = new System.IO.BinaryReader(fs);
originalImageColors = new Color[theWidth * theHeight];
i = 0;
for (xx = 0; xx < theWidth; xx++) {
for (yy = 0; yy < theHeight; yy++) {
tempR = rr.ReadByte() / 255.0;
tempG = rr.ReadByte() / 255.0;
tempB = rr.ReadByte() / 255.0;
originalImageColors[i] = Color(tempR, tempG, tempB);
destImage.SetPixel(xx, yy, Color(tempR, tempG, tempB));
i += 1;
}
}
rr.Close();
destImage.Apply();
}
The code could still be cleaned up a bit, but at least it works.