Get list of all files from directory in WebGL

I have a directory inside StreamingAssets where 10s-20s of files are located. I am using DirectoryInfo to load all the files. It works in my editor but in WebGL I need to use UnityWebRequest. How do I convert my code to support UnityWebRequest?

void Start () {
        string path = System.IO.Path.Combine (Application.streamingAssetsPath, "Files/");
       
        DirectoryInfo dir = new DirectoryInfo (path);
        FileInfo[] info = dir.GetFiles ("*.*");
        foreach (FileInfo file in info) {
            string[] lines = File.ReadAllLines (file.FullName);
            foreach (string line in lines) {

                if (line.Contains ("001")) {

                    Debug.Log ("Link: " + file.FullName);
                    break;
                }
            }

        }
    }

you could have a server side script to return list of files from certain folder.
(for example: php script that you call from unitywebrequest, and php script scans the folder)

But why would i need a php script?
If I could load any file using Unity’s UnityWebRequest, I would like to implement from it itself.

yes, you can load files with webrequest,
but cannot get list of unknown files on the server folder…

unless you brute force and try to guess if file exists in given url,
which could work, if you know the base file name and increment counter: “myfile1.png”, “myfile2.png”…

or i guess, if you enable directory browsing on the server side,
then you get that html page with files list, and could parse filenames: (then no need server side script)

php script would use this,

ah so i cannot get a whole directory and add all files to a list, is it?

yeah, no direct file access in webgl like that.

other alternative could be using postbuild script in editor,
that generates StreamingAssets files list automatically. (and saves the list.txt into StreamingAssets)

then you could load that list.txt using unitywebrequest.

Thank you. Can you please explain the process of php sending files to Unity? I did not follow it.

some examples, outputs to json

*can remove those file sizes or other extras

so webrequest would call that php url, http://yourserver.asdf/getfiles.php, and then you have json array of files.

php doesn’t sent file data, just list of files (although it could send binary data too),
but easier to load with webrequest, using filename from that list, and base server url:
http://yourserver.asdf/myfiles/somefile.png

Thanks so that means in Unity, what I have to do is this:

UnityWebRequest uwr = UnityWebRequest.Get(https://myserver.com/GetFiles.php);
yield return uwr.SendWebRequest();

string jsonFile = uwr.downloadHandler.text;

Once this is done, I want to go through each file that is present inside the directory so as to find a specific string. Currently I am doing that using FileInfo. I am not sure how can I do it through jsonFile.

foreach (FileInfo file in jsonFile) {
            Debug.Log (file.Name);

            string[] lines = File.ReadAllLines (file.FullName);
            foreach (string line in lines) {

                if (line.Contains ("001")) {
                    fileName = file.Name;

                    Debug.Log ("Broken Link: " + file.FullName);
                    break;
                }
            }

        }

json can be converted into array/list/class,

but confirm that json is valid (manually paste that downloaded json string here)

and can use this to create class

*and unity json has some issues with certain arrays, so might have to modify php, if have those issues
http://answers.unity.com/answers/1796846/view.html

Thanks alot. I have implemented it and it works successfully.
I will explain the working. I first create a php script to output all the files in json format:

<?php

$dir          = "./myFolder";
$return_array = array();

if(is_dir($dir)){

    if($dh = opendir($dir)){
        while(($file = readdir($dh)) != false){

            if($file == "." or $file == ".."){

            } else {
                $return_array[] = $file; // Add the file to the array
            }
        }
    }

    echo json_encode($return_array);
}

?>

and then in Unity, i use WebRequest call to php to get all the files and then add them to a list.
I then go through each file again using WebRequest to find a string.

using System;
using System.Collections;
using System.Collections.Generic;
using System.IO;
using SimpleJSON;
using UnityEngine;
using UnityEngine.Networking;


public List<string> httpFiles = new List<string>();
public string fileName;

IEnumerator Start () {

        UnityWebRequest uwr = UnityWebRequest.Get("https://MyServer/GetFiles.php");
        yield return uwr.SendWebRequest();

        var jsonFile = uwr.downloadHandler.text;
        var parseJSON = JSON.Parse (jsonFile);

        for(int i = 0; i < parseJSON.Count; i++) {
            httpFiles.Add("https://MyServer/myFolder/"+parseJSON[i]);
        }

        for(int i = 0; i < httpFiles.Count; i++) {
            UnityWebRequest webReq = UnityWebRequest.Get(httpFiles[i]);
            yield return webReq.SendWebRequest();
            var json = webReq.downloadHandler.text;

            if(json.Contains("TEXT TO FIND")){
                fileName = httpFiles[i];
                Debug.Log(fileName);
                break;
            }
        }

    }

FYI, I use SimpleJSON to parse the json files.

@mgear it just takes some time to load all the files though. Is there any way I can faster the process?

what kind of files those are, how many in the list and file sizes?

if those are txt files, you could do work on the server already (return list of only those files that you want)

they are json files and there are 1200 of them.

that sounds too much to load using webrequests…

what do you need to json files for?

CORS, if you have access to server, can allow it

If you design your own json web interface I would always recommend you have an object as top level element. It’s also some sort of standard. It’s common to have a success / error field to include metadata about the success or failure. The outer object would then have a “data” field that contains the actual “payload”.

Also keep in mind that a webserver does not necessarily have folders / directories at all. HTTP works by using URLs (Uniform Resource Locators) which don’t need to map to actual files. Yes, in most trivial cases they do. Though most front ends nowadays work on completely virtual routes. So that’s why there is no way to “see” the content of a folder as HTTP itself does not have a concept of folders. A client sends a request to the server and the server responds with the requested resource, or an error if that resource does not exist.

Well, at the moment you load one file after the other. You can usually load several files in parallel. How many completely depends on the implementation of your webserver and if it has a limit of parallel downloads. You could start a separate coroutine for each file, so each coroutine can wait for its own webrequest. Another solution is have a list of UnityWebRequests, start them all at once and then have a single coroutine checking each element in the list so see when all are done.

In many cases a hybrid approach is the better solution. You usually don’t want too many parallel downloads at the same time to keep the memory hit low, but enough to better use the bandwidth. For this you usually would create a list or Queue of tasks that contains all things you want to download. Now you can start one or multiple “consumer” coroutines which simply pick an element out of the queue, download it and when finished, take out the next element until the queue is empty. That way you can control how many parallel downloads you want (by starting that coroutine x times)

Ohh, just updated as I was writing:

If they are all json files, you could download them all at once in a single webrequest. Your php script can simply load them all and simply include them all in a single json response. We don’t know what the content looks like, but you could load them, decode them into arrays, combine them into a single object and finally encode it again. So as a result you would have a single json object that would have an array of more json objects.

That is a good idea to download all at once. Currently I am looping through each which I understand is a huge performance issue.

yeah not all JSONs are structured the same but I hope I can have a single json object and it would still work the same.
Just out of curiosity if we both are on the same page, could you please improve my code to your logic please?