Maintaining responsiveness while reading from large file

Hello,

I’m working on a project that at this stage basically boils down to the following steps:

  1. Start the application
  2. Specify a file path in the UI’s input field
  3. Hit “Import data”
  4. Data is now represented in the virtual environment in some manner.

The file is almost 100MB, which I know is indeed a large file. Therefore, I thought that since the UI will feature some options to filter, map, and render this data, you’d probably want the application to be responsive while the data is being read and represented in the environment, at the very least update the user of the progress of reading the file.

Loading this data without doing anything with it in the environment takes a few seconds, which is tolerable, but also generating objects from prefabs causes it to take uptowards a minute, which is unacceptable. Therefore, I thought of using coroutines.
A naïve implementation of coroutines allows the data import to work “in the background” while the rest goes on as usual, though it is prohibitively slow. 800000 rows of data that was read in and represented in about 1 minutes is now ticking about 120 rows/second, using ‘yield return null’. I also tried some very small WaitForSeconds, but to no discernible effect.

Adding a for loop of arbitrary length before the yield certainly speeds up the process, but the loop length is, as mentioned, arbitrary. It might be good for my system in editor mode, but too slow as a finished application, and anything goes for other people’s systems.

My question is the following:

Is it possible to implement some methods, using coroutines or not, to basically do all computation needed to produce a frame normally according to one’s set framerate, and then spend the rest of the frame’s residual computation time to the things that pertains to reading the file?

Here’s an abstracted version of the coroutine:

IEnumerator ReadFileCoRoutine(string FileName) {
        if (File.Exists(FileName)) {
            int counter = 1;
            using(StreamReader file = new StreamReader(FileName)) {
                string ln;
                while ((ln = file.ReadLine()) != null) {
                    for (int _ = 0; _ < 600; _++) { //FOR LOOP TO SPECIFY HOW MANY LINES BEFORE NEXT YIELD
                        if (_ > 0) {
                            if ((ln = file.ReadLine()) == null)
                                break;
                        }
                        //USE DATA

                        counter++;
                    }
                    yield return null;
                }
                file.Close();
            }
        }
    }