WebGL with streaming option - like WebPlayer

Unity Engine
1

Hey,

While we worked on a WebGL port for our game, we needed an option to stream levels like in the WebPlayer.
@jonas-echterhoff_1 wrote a basic script that did that on the beta version of Unity5.
We updated it, and added the option to check download progress.

Instead of Application.CanStreamedLevelBeLoaded() and Application.GetStreamProgressForLevel()
We now have
WebGLLevelStreaming.CanStreamedLevelBeLoaded()
WebGLLevelStreaming.GetStreamProgressForLevel()

Use “firstStreamedLevelWithResources” like in the “First Streamed Level” option of the WebPlayer
http://docs.unity3d.com/Manual/class-PlayerSettingsWeb.html

So, the code for WebGLLevelStreaming.cs

using UnityEngine;
#if UNITY_EDITOR
using UnityEditor;
using UnityEditor.Callbacks;
#endif
using System.Collections;
using System.Collections.Generic;
using System.IO;
using System.Diagnostics;
using System.Linq;
using System.Runtime.InteropServices;


public class WebGLLevelStreaming {

#if UNITY_EDITOR && UNITY_WEBGL
    private const string kOutputDataExtension = ".data";
    private const string kOutputFileLoaderFileName = "fileloader.js";
    private const string kResourcesDirName = "Resources";
    private const string kResourcesFileName = "unity_default_resources";
    private const string kResourcesExtraFileName = "unity_builtin_extra";

    public static string buildToolsDir
    {
        get {
            // Find WebGL build tools
            var webGLBuildTools = Path.Combine("WebGLSupport", "BuildTools");
            var playbackEngines = Path.Combine(EditorApplication.applicationContentsPath, "PlaybackEngines");
            var path = Path.Combine(playbackEngines, webGLBuildTools);
            if (Directory.Exists(path))
                return path;
            else
                return Path.Combine(Path.Combine(EditorApplication.applicationPath, "../../"), webGLBuildTools);
        }
    }

    public static string emscriptenDir
    {
        get { return buildToolsDir + "/Emscripten"; }
    }

    static string packager
    {
        get { return emscriptenDir + "/" + "tools/file_packager.py"; }
    }

    public static string pythonExecutable
    {
        get
        {
            if (Application.platform == RuntimePlatform.WindowsEditor)
                return "\"" +  buildToolsDir + "/" + "Emscripten_Win/python/2.7.5.3_64bit/python.exe\"";
            return "python";
        }
    }

    public static int firstStreamedLevelWithResources = 1;

    // Run emscripten file packager to pack a .data file
    private static bool RunPackager (string filename, string stagingAreaData, IEnumerable<string> filesToShip)
    {
        var argumentsForPacker = "\"" + packager + "\"" ;
        argumentsForPacker += string.Format(" \"{0}\"", filename + kOutputDataExtension);
        argumentsForPacker += " --no-heap-copy";
        argumentsForPacker += " --js-output=\"" + filename + ".loader.js" + "\"";

        if (PlayerSettings.GetPropertyBool("dataCaching", BuildTargetGroup.WebGL))
            argumentsForPacker += " --use-preload-cache";

        argumentsForPacker += " --preload";
        argumentsForPacker += filesToShip.Aggregate("", (current, file) => current + (" \"" + file + "\""));

        var processStartInfo = new ProcessStartInfo(pythonExecutable)
        {
            Arguments = argumentsForPacker,
            WorkingDirectory = stagingAreaData,
            UseShellExecute = false,
        };

        var p = Process.Start(processStartInfo);
        p.WaitForExit();
        if (p.ExitCode == 0)
            return true;

        throw new System.Exception("Failed running " + processStartInfo.FileName + " " + processStartInfo.Arguments);
    }
   
    private static bool PackageData (string filename, string stagingAreaData, IEnumerable<string> _filesToShip)
    {
        var filesToShip = new HashSet<string>(_filesToShip.Select(o => Path.GetFileName(o)));
        filesToShip.Add(Path.Combine(kResourcesDirName, kResourcesFileName));
        filesToShip.Add(Path.Combine(kResourcesDirName, kResourcesExtraFileName));

        if (firstStreamedLevelWithResources < 0)
            firstStreamedLevelWithResources = 0;
        if (firstStreamedLevelWithResources >= Application.levelCount)
            firstStreamedLevelWithResources = Application.levelCount-1;
        var loaderJSStart = "var StreamProgressForLevelArray = [];\nStreamProgressForLevelArray[0]=1;\n";
        var loaderJS = "";
        var loaderJS0 = "";

        int packageIndex = 0;
        // Generate a data file for each streaming level
        while (true)
        {
            // find all the files for this level
            var fileNames = new List<string>();
            int levelIndex = packageIndex;
            fileNames.Add("sharedassets" + levelIndex + ".resource");
            fileNames.Add("sharedassets" + levelIndex + ".assets");
            if(packageIndex>0)
            {
                fileNames.Add("level" + (levelIndex-1) + "");
            }else{
                fileNames.Add(Path.Combine(kResourcesDirName, kResourcesFileName));
                fileNames.Add(Path.Combine(kResourcesDirName, kResourcesExtraFileName));
                fileNames.Add("mainData");
            }

            if(levelIndex==firstStreamedLevelWithResources)
            {
                fileNames.Add("resources.assets");
                fileNames.Add("resources.resource");
            }

            var files = new List<string>();
            foreach (var f in fileNames)
            {
                if (filesToShip.Contains(f))
                {
                    filesToShip.Remove(f);
                    files.Add(f);
                }
            }
            if (files.Count == 0)
                break;

            var packageFile = filename + "." + packageIndex;
            if (!RunPackager(packageFile, stagingAreaData, files))
                return false;

            // Extract loader script for this data file, and put it's content into a JavaScript function
            if(packageIndex>0)
            {
                var loaderText = File.ReadAllText(Path.Combine(stagingAreaData, packageFile + ".loader.js"));
                loaderText = loaderText.Substring(loaderText.IndexOf("\n(function() {"));
                loaderText = loaderText.Replace("if (Module['setStatus']) Module['setStatus']('Downloading data... (' + loaded + '/' + total + ')');",
                                                "StreamProgressForLevelArray["+packageIndex+"]=event.loaded/size;");
                loaderJS += "function DownloadDataForPackage" + packageIndex + "(){\n" + loaderText + "}\n";
                loaderJSStart +="StreamProgressForLevelArray[" + packageIndex + "]=0;\n";
            }else{
                loaderJS0 = File.ReadAllText(Path.Combine(stagingAreaData, filename + ".0.loader.js"));
                loaderJS0 = loaderJS0.Replace("if (Module['setStatus']) Module['setStatus']('Downloading data... (' + loaded + '/' + total + ')');",
                                              "if (Module['setStatus']) Module['setStatus']('Downloading data... (' + event.loaded + '/' + size + ')');");
            }
            packageIndex++;
        }

        if(filesToShip.Count>0)
        {
            var packageFile = filename + "." + packageIndex;
            if (!RunPackager(packageFile, stagingAreaData, filesToShip))
                return false;
            var loaderText2 = File.ReadAllText(Path.Combine(stagingAreaData, packageFile + ".loader.js"));
            loaderText2 = loaderText2.Substring(loaderText2.IndexOf("\n(function() {"));
            loaderJS += "function DownloadDataForPackage" + packageIndex + "(){\n" + loaderText2 + "}\n";
            packageIndex++;
        }

        loaderJS = loaderJSStart + loaderJS + loaderJS0;

        for (int j=0; j < packageIndex-1; j++)
        {
            // Patch loader scripts to invoke the function to start loading the next data file when it's done
            var line = "Module['removeRunDependency']('datafile_" + filename + "." + j + ".data');";
            // Call function to load next data file in a callback to avoid calling it while clearing pre-run dependencies.
            loaderJS = loaderJS.Replace (line, line + "\nStreamProgressForLevelArray[" + j + "]=1;\nwindow.setTimeout(DownloadDataForPackage" + (j+1) + @",1);");
        }

        // write out loader script for all data files
        File.WriteAllText(Path.Combine(stagingAreaData, kOutputFileLoaderFileName), loaderJS);
        return true;
    }

    // Generate gzip compressed versions of files
    private static void CompressFilesInOutputDirectory (string dir, string outputDir)
    {
        var filesToCompress = Directory.GetFiles(dir).Where(f => f.EndsWith(kOutputFileLoaderFileName) || f.EndsWith(".data"));
        foreach (var file in filesToCompress)
        {
            var processName = "7za";
            if (Application.platform == RuntimePlatform.WindowsEditor)
                processName = "7z.exe";
            var processStartInfo = new ProcessStartInfo(EditorApplication.applicationContentsPath+"/Tools/"+processName)
            {
                Arguments = "a -tgzip \""+Path.Combine(outputDir, Path.GetFileName(file))+"gz\" \""+file+"\"",
                UseShellExecute = false,
                CreateNoWindow = true
            };

            var p = Process.Start(processStartInfo);
            p.WaitForExit();
            if (p.ExitCode != 0)
                throw new System.Exception("Failed running " + processStartInfo.FileName + " " + processStartInfo.Arguments);
        }
    }

    [PostProcessBuild]
    public static void OnPostprocessBuild(BuildTarget target, string pathToBuiltProject) {
        if (target != BuildTarget.WebGL)
            return;
        var stagingAreaData = Path.Combine(Path.Combine("Temp", "StagingArea"), "Data");
        var filename = Path.GetFileName(pathToBuiltProject);

        // Find all files to package
        IEnumerable<string> filesToShip = Directory.GetFiles(stagingAreaData, "*.resource");
        filesToShip = filesToShip.Concat(Directory.GetFiles(stagingAreaData, "*.assets"));
        filesToShip = filesToShip.Concat(Directory.GetFiles(stagingAreaData, "mainData"));
        filesToShip = filesToShip.Concat(Directory.GetFiles(stagingAreaData, "level*"));

        // put files into packages
        PackageData (filename, stagingAreaData, filesToShip);

        // delete old data file
        File.Delete (Path.Combine(pathToBuiltProject, Path.Combine("Release", filename+".data")));

        // copy new data files into build
        File.Copy(Path.Combine(stagingAreaData, kOutputFileLoaderFileName), Path.Combine(Path.Combine(pathToBuiltProject, "Release"),kOutputFileLoaderFileName), true);
        foreach (var f in Directory.GetFiles(stagingAreaData, "*.*.data"))
            File.Copy(f, Path.Combine(Path.Combine(pathToBuiltProject, "Release"),Path.GetFileName(f)), true);

        // compress new data files
        CompressFilesInOutputDirectory (Path.Combine(pathToBuiltProject, "Release"), Path.Combine(pathToBuiltProject, "Compressed"));

        // delete old compressed data file
        File.Delete (Path.Combine(pathToBuiltProject, Path.Combine("Compressed", filename+".datagz")));
    }
#endif

#if UNITY_WEBGL
    [DllImport("__Internal")]
    private static extern float GetStreamProgressForLevelFromWeb(int levelIndex);
#endif

    static public bool CanStreamedLevelBeLoaded (int levelIndex)
    {
#if UNITY_WEBGL && !UNITY_EDITOR
        if (levelIndex == 0) // file = mainData
            return true;
        return File.Exists("level"+(levelIndex-1));
#else
        return Application.CanStreamedLevelBeLoaded(levelIndex);
#endif   
    }

    static public float GetStreamProgressForLevel (int levelIndex)
    {
        #if UNITY_WEBGL && !UNITY_EDITOR
        if (levelIndex == 0) // file = mainData
            return 1;
        return GetStreamProgressForLevelFromWeb(levelIndex);
        #else
        return Application.GetStreamProgressForLevel(levelIndex);
        #endif   
    }
}

And code for the JS file, WebGLLevelStreamingJS.jslib

var WebGLLevelStreamingJS = {
    GetStreamProgressForLevelFromWeb: function(levelIndex)
    {
        return StreamProgressForLevelArray[levelIndex];
    }
};

mergeInto(LibraryManager.library, WebGLLevelStreamingJS);

IMPORTANT
Tested on Unity5.0.0f4.
Unity still working on the WebGL build process, so this script may not work on versions newer than 5.0.0f4.

Known Issues:
GetStreamProgressForLevel() return 0 if loading from cache.
Some other issues with the progress that we need to check, but weren’t critical for us.

You all are welcome to improve the script :slight_smile:

2 Likes
2

srry for being off-topic on your problem, De-Panther, but could you mention what is your game about? Type, genre, online features, etc?

3

Hey,

The script is very useful, thanks!
For those who is not aware where is is coming from here is the link to the presentation.

De-Panther, the presentation also contains very interesting slide #25, that says how you dealt with textures and resources. Could you please share details?

Is that possible to share build script?

4

Can any1 tell me how to make this work??
I Used the 1st script and it did split files.
But what to do with 2nd script.

When i tested not all objects in level where loaded.

5

@slmoloch I can’t share those, as they are specific for the client’s project.
But maybe @liortal can explain how it works - he worked on the build script.
I just remember that it’s something about searching on the scene files for specific textures and replace them with URL to download.
And about removing resources and scenes - we move them to the “Editor” folder while the build proccess is running, and move them back when it finished.

@skeleton_king , just put the “WebGLLevelStreamingJS.jslib” file under Assets/Plugins/WebGL/

6

We have actually created a build infrastructure (see the menu item in the screenshot) that builds the game.

This takes into account a customized build properties file that has a few pieces of data:

  • Excluded scenes - this is a collection of scenes (dragged from the project window) that will not be included in the final output. The build process calls BuildPipeline.BuildPlayer; this method accepts an array of scenes, so we can easily pass in the scenes we want to build.
  • Excluded resources - this is a collection of resources (files / folders under a Resources folder) that also will be excluded from the build. These can be stuff that is unused in the game, so there’s no point of including them in the final game build. Since there’s no way to “exclude” resources, we do a little trick - we just move these into a folder that is not under “Resources” and then move them back when the build is done.
  • Remove texture usage - for some textures that were too large, we actually incorporated a small trick - we don’t include them in the game, but at runtime we download them from a server. This is done automatically by the build process - we have a data file that describes which textures we’d like to “patch” in this way. The build process looks for all of these textures, and removes them (replacing them with a special component that knows how to fetch them at runtime).

At this point, the build process was created for this project in our company (Moon Active), i don’t think we can share it.

I have taken the concepts from the design we used for this project and i plan to create a new project (open source?) of build tools for Unity. Watch this space in the future: https://github.com/liortal53/UnityBuildTools

7

Nice, thanks for the prompt reply! I totally understand that you can not share the script, but could you point me to the direction on where to find the sample of component that downloads the texture?

8

Looking forward for updates to that repository. When do you expect to add something there?

9

The component that downloads textures is a super basic one.

It has a public URL field that is set from the editor, and in its Awake method it downloads the texture at the given URL, creates a Sprite object and assigns it to the SpriteRenderer.

10

Very Cool! I need this for my project, but I’m currently using Unity 5.0.2. Any idea if this will work there?

Also has the “GetStreamProgressForLevel() return 0 if loading from cache.” error been resolved? If not does it actually cause a problem in the WebGL build for the player to move forward with loading the game or level even if it was cached?

I think this should be an officially added feature as with all the Memory Allocation issues WebGL has currently, something like this is critical for us to build and deploy WebGL games in the current state.

11

Should be fine, although we haven’t tested it with that particular version yet.

1 Like
12

Thanks @jonkuze
GetStreamProgressForLevel() should return 1 now if loading from cache, but I suggest you should also check it yourself.

Please reply if it work or didn’t work

13

FYI, my original version of this script is now also available on the AssetStore:

(Though you may still prefer to use De-Panther’s improved version, of course.)

2 Likes
14

Cool Thanks! Good to know!

15

Hi De-Panther,
When i compiled your WebGLLevelStreaming.cs script from unity 5.1,i get huge python errors.

Failed running python “/Applications/Unity5.1/Unity.app/Contents/PlaybackEngines/WebGLSupport/BuildTools/Emscripten/emcc” @“/Users/apple/Downloads/GameFolder/Game/Assets/…/Temp/emcc_arguments.resp”

stdout:
stderr:
warning: unresolved symbol: glFlushMappedBufferRange
warning: unresolved symbol: glGetInternalformativ
warning: unresolved symbol: tcflush
warning: unresolved symbol: pthread_create
warning: unresolved symbol: _ZN4FMOD13DSPConnection6setMixEf
warning: unresolved symbol: glUnmapBuffer
warning: unresolved symbol: glGetStringi
warning: unresolved symbol: glProgramBinary
warning: unresolved symbol: glMapBufferRange
warning: unresolved symbol: glGetProgramBinary
warning: unresolved symbol: glCopyBufferSubData
Traceback (most recent call last):
File “/Applications/Unity5.1/Unity.app/Contents/PlaybackEngines/WebGLSupport/BuildTools/Emscripten/emscripten.py”, line 1675, in
_main(environ=os.environ)
File “/Applications/Unity5.1/Unity.app/Contents/PlaybackEngines/WebGLSupport/BuildTools/Emscripten/emscripten.py”, line 1663, in _main
temp_files.run_and_clean(lambda: main(
File “/Applications/Unity5.1/Unity.app/Contents/PlaybackEngines/WebGLSupport/BuildTools/Emscripten/tools/tempfiles.py”, line 39, in run_and_clean
return func()
File “/Applications/Unity5.1/Unity.app/Contents/PlaybackEngines/WebGLSupport/BuildTools/Emscripten/emscripten.py”, line 1671, in
DEBUG_CACHE=DEBUG_CACHE,
File “/Applications/Unity5.1/Unity.app/Contents/PlaybackEngines/WebGLSupport/BuildTools/Emscripten/emscripten.py”, line 1558, in main
jcache=jcache, temp_files=temp_files, DEBUG=DEBUG, DEBUG_CACHE=DEBUG_CACHE)
File “/Applications/Unity5.1/Unity.app/Contents/PlaybackEngines/WebGLSupport/BuildTools/Emscripten/emscripten.py”, line 924, in emscript_fast
%s’‘’ % (staticbump, global_initializers, mem_init)) # XXX wrong size calculation!
UnicodeDecodeError: ‘ascii’ codec can’t decode byte 0xc2 in position 308695: ordinal not in range(128)
Traceback (most recent call last):
File “/Applications/Unity5.1/Unity.app/Contents/PlaybackEngines/WebGLSupport/BuildTools/Emscripten/emcc”, line 1323, in
final = shared.Building.emscripten(final, append_ext=False, extra_args=extra_args)
File “/Applications/Unity5.1/Unity.app/Contents/PlaybackEngines/WebGLSupport/BuildTools/Emscripten/tools/shared.py”, line 1535, in emscripten
assert os.path.exists(filename + ‘.o.js’), ‘Emscripten failed to generate .js’
AssertionError: Emscripten failed to generate .js

UnityEditor.HostView:OnGUI()

Can you help me on this?

16

@Gaurav-Gulati I didn’t test it on 5.1, so I’m not familiar with this problem.
try @jonas-echterhoff_1 's version from the asset store

and if you want to add the feature of GetStreamProgressForLevel compare both of the versions for changes…

17

The asset store script is not working with Unity 5.1.2 - it’s working with 5.1.1 - but with the new version I get the error:
Uncaught RangeError: Maximum call stack size exceded

@jonas-echterhoff_1 It would be cool if you could check that for me.

18

Very helpfull.Thanks!

19

Starting from 5.1.2 there is a another file that you need to add to the first scene - global-metadata.dat. Just add two lines of code:

filesToShip.Add(Path.Combine(@“Il2CppData\Metadata”, “global-metadata.dat”));

and

fileNames.Add(Path.Combine(@“Il2CppData\Metadata”, “global-metadata.dat”));

1 Like
20

Hi, i’m having a bit of an issue, i copied paste the de-panther script and added the 2 lines above, set the "firstStreamedLevelWithResources " to 0 which is my preloader scene, but when i launch the game, only the “webGL.data” file is being downloaded.

The “webGL.0.data”, “webGL.1.data” etc… files aren’t downloaded so i guess i must have missed something :s. Is there anything else to add, or maybe call a function somewhere ?

I tried to use the version of @jonas-echterhoff_1 and added the 2 lines above, but i get the following errors:

  • run() called, but dependencies remain, so not running
  • Not implemented: Class::GetClassBitmap (mutilple times)
  • Not implemented: Il2CppTypeCompare::compare

I noticed that using his version correctly download the “webGL.0.data”, “webGL.1.data” files, but generate errors when loading is complete :s.

Any idea how to fix this ? thx for reading :slight_smile: