C# How do I read an external text file asynchronously in Unity?

Hi, I’m currently working on a visualization which required text file reading during run-time, and there will be a progress bar showing how much of the reading is done. Here’s the problem; the text file is relatively large and takes roughly 20~ seconds to finish reading. During the reading Unity froze and you can’t actually see the progress bar updating.

I’ve thought of a few ways to read the file async but I’ve encountered a bit of problem.

First I’ve tried using BackgroundWorker which will run the file-reading function in a background thread, the code goes something like this;

BackgroundWorker bgWorker = new BackgroundWorker ();
    
    	private void bgWorker_DoWork(object sender, DoWorkEventArgs e)
    	{
    		readFile ();
    	}

void Start () {

		maxSize = new Vector3 (150, 30, 150);
		curSize = maxSize / 2;
		progressPercentage = 0;

		renderCubes = new List<TemperatureCubes> ();


		bgWorker.DoWork += new DoWorkEventHandler (bgWorker_DoWork);
		bgWorker.RunWorkerAsync ();
	}

readFile()

List<RowData> data;

    void readFile ()
    	{
    		data = new List<RowData>();
    
    		var reader = new StreamReader (File.OpenRead (Application.dataPath + "/heatmap_output_high_res_run.csv")); //heatmap_output_all_one_cm
    		noOfLines = File.ReadAllLines (Application.dataPath + "/heatmap_output_high_res_run.csv").Count ();
    		int lineCount = 0;
    
    		while (!reader.EndOfStream) 
    		{
    
    			RowData tempData = new RowData();
    
    			var line = reader.ReadLine ();
    			lineCount++;
    			var values = line.Split (',');
    
    			if (values.Length == 6) 
    			{
    				if (int.TryParse (values [0], out tempData.timeStep)
    				    && int.TryParse (values [1], out tempData.level)
    					&& float.TryParse (values [2], NumberStyles.Float, CultureInfo.InvariantCulture, out tempData.pos.x)
    					&& float.TryParse (values [3], NumberStyles.Float, CultureInfo.InvariantCulture, out tempData.pos.y)
    					&& float.TryParse (values [4], NumberStyles.Float, CultureInfo.InvariantCulture, out tempData.pos.z)
    					&& float.TryParse (values [5], NumberStyles.Float, CultureInfo.InvariantCulture, out tempData.temperature)
    					)
    				{
    					tempData.timeStep += 1; //somedatafix
    
    					tempData.pos.x /= 100;
    					tempData.pos.y /= 100;
    					tempData.pos.z /= 100;
    
    					data.Add(tempData);
    
    				} 
    				else 
    				{
    					continue;	
    				}
    			}
    
    			if(lineCount == Mathf.Floor(noOfLines*0.1f))
    			{
    				progressPercentage = 10;
    			}
    		}
    
    		
    	}

RowData

public class RowData
	{
		public int timeStep;
		public int level;
		public Vector3 pos;
		public float temperature;

		public RowData()
		{
			timeStep = -1;
			level = -1;
			pos =  new Vector3();
			temperature = -1;
		}
	}

Edit: File size is roughly 150mb~, 5.7M+ lines.

However this gives an error saying:

get_dataPath can only be called from the main thread.
Constructors and field initializers will be executed from the loading thread when loading a scene.
Don’t use this function in the constructor or field initializers, instead move initialization code to the Awake or Start function.

I’m not sure if there’s any way around it, if there is please enlighten me.

Then I researched google a bit on reading text asynchronously and found out that C# has this. I tried it but turns out that Unity doesn’t have Threading.Task framework(?). I’m not 100% familiar with it myself.

Basically I’ve tried all that I can think of but still can’t figure a way around it. Is there other approach in reading text file asynchronously in Unity or fix for the methods I’ve tried? Thanks in advance!

Edit 2: Tried coroutine as suggested by @Bonfire Boy, by changing “void readFile()” to “IEnumerator readFile()”, and in also update, if press space “StartCoroutine(“readFile”);”

It’s notably faster, took roughly 3~ seconds to read the file (twice because of counting lines), however during that time Unity still freezes and progress on that bar can’t be seen. Am I doing it wrong?

Edit 3: TL;DR Main issue: I needed to use an Unity API to read the text file;

var reader = new StreamReader (File.OpenRead (Application.dataPath + "/heatmap_output_high_res_run.csv"));

However it can only be run in the main thread, and that causes a 20~ seconds (3~ seconds using Coroutine) spike. I needed a progress bar to show the progress of the reading but during the spike Unity freezes can no progress can be seen on the bar, which defeats the purpose of the progress bar. Is there any way around it if I really do need the progress bar?

Edit 4:

I’ve tried moving Application.dataPath outside of readData()

string filePath = Application.dataPath + "/heatmap_output_high_res_run.csv";

and in readData,

var reader = new StreamReader (File.OpenRead (filePath)); 
noOfLines = File.ReadAllLines (filePath).Count ();

however this is still giving me the same error.

Just move the call to Application.dataPath out to the main thread.

This error should go away then. Note I haven’t read the rest of the code to see if it will throw any other errors.

I’m pretty sure you read the error the wrong way. StreamReader should work fine in a seperate thread. What you can’t use in a seperate thread are things from the UnityAPI. Since the only thing you seem to use from Unity is “Application.dataPath” i suspect that the getter is actually throwing that error. Maybe post your actual error (with stacktrace) from the console.

Btw: File.ReadAllLines reads already the whole file into a string array- You basically open and read the file two times. However you just use the Linq extension Count() to get the length and then you throw the array away.

You might want to read in the file once as a string array in the main thread and then just pass the array to your method and not using the stream reader at all. Also you did not show where you initialize your “data” variable. I guess it’s a generic List? Did you specify a capacity? If not this will create a huge amount of garbage as the internal array has to be resized several times.

About how many lines do we talk about? 10k? 100? 1M? 10M? 100+M?

Furthermore what’s RowData? a class or a struct? A class would be bad for performance.

A bit more details on your setup and i can suggest a better approach.