What I want to do is create a WebGL client that performs the photo segmentation and all of that image analysis magic and then stores the user avatar and data in a database. My plan is then to create a client package other developers can use to import the finished user avatar into their apps/games.
I just can’t figure out how to actually load an image from the harddrive in WebGL, given the security limitations of the platform. I tried to use dropbox/onedrive/gdrive publicly shared file urls but I run into issues with CORS and since this was a workaround in the first place I think this is the wrong way to go other than as a last resort.
I suspect one could create a popup window using jslib and upload an image to it, create a texture and send backa texture pointer to the webgl app but my javascript skills are simply not up to the task.
If anyone could help me get this to work or point me to the correct resources/tutorials I would be very grateful!
It is very much possible to load an image (or any file) from the user hard drive into your WebGL application.
You can not do this directly from your application, but you can implement a JavaScript plugin that will do the job. Normally I would suggest to transfer data between JavaScript and WebGL using module heap, but for people not familiar with the asm.js infrastructure it might be more convenient to use blobs. The whole process can be split into two separate tasks:
1) Uploading an image using html button
First, you need to implement a JavaScript plugin that will handle the browser file dialog, create a blob for the selected file and transfer the blob address to the WebGL application. You can create the following Assets/Plugins/ImageUploader.jslib file for that:
var ImageUploaderPlugin = {
ImageUploaderInit: function() {
var fileInput = document.createElement('input');
fileInput.setAttribute('type', 'file');
fileInput.onclick = function (event) {
this.value = null;
};
fileInput.onchange = function (event) {
SendMessage('Canvas', 'FileSelected', URL.createObjectURL(event.target.files[0]));
}
document.body.appendChild(fileInput);
}
};
mergeInto(LibraryManager.library, ImageUploaderPlugin);
Now you can attach the following script to your Canvas, that will initialize the plugin and load the texture from the generated blob address:
using UnityEngine;
using System.Collections;
using System.Runtime.InteropServices;
public class CanvasScript : MonoBehaviour {
[DllImport("__Internal")]
private static extern void ImageUploaderInit();
IEnumerator LoadTexture (string url) {
WWW image = new WWW (url);
yield return image;
Texture2D texture = new Texture2D (1, 1);
image.LoadImageIntoTexture (texture);
Debug.Log ("Loaded image size: " + texture.width + "x" + texture.height);
}
void FileSelected (string url) {
StartCoroutine(LoadTexture (url));
}
void Start () {
ImageUploaderInit ();
}
}
2) Uploading an image using UI button
This one is quite tricky. The idea would be of course to make the html browse button hidden, and simulate its click programmatically. We are however facing a security restriction here, specifically, it is not possible to launch the onclick event handler programmatically, unless you make the click() call from another event handler initiated by the user. And as in most complex systems, in Unity WebGL input events are not processed directly but instead are passed through an intermediate queue. This means that the initial JavaScript event handler initiated by the user has been already processed by the time when you receive the OnClick event in the managed code.
Nevertheless, there is a trick that we can perform to make it work: as soon as the user pushes the UI button down, we can register onclick JavaScript event for the whole WebGL canvas, so that when the user releases the button, we will get a user-initiated onclick JavaScript event, and perform the input click simulation in the handler.
You will need the following Assets/Plugins/ImageUploader.jslib plugin:
var ImageUploaderPlugin = {
ImageUploaderCaptureClick: function() {
if (!document.getElementById('ImageUploaderInput')) {
var fileInput = document.createElement('input');
fileInput.setAttribute('type', 'file');
fileInput.setAttribute('id', 'ImageUploaderInput');
fileInput.style.visibility = 'hidden';
fileInput.onclick = function (event) {
this.value = null;
};
fileInput.onchange = function (event) {
SendMessage('Canvas', 'FileSelected', URL.createObjectURL(event.target.files[0]));
}
document.body.appendChild(fileInput);
}
var OpenFileDialog = function() {
document.getElementById('ImageUploaderInput').click();
document.getElementById('canvas').removeEventListener('click', OpenFileDialog);
};
document.getElementById('canvas').addEventListener('click', OpenFileDialog, false);
}
};
mergeInto(LibraryManager.library, ImageUploaderPlugin);
And the following script to interact with it:
using UnityEngine;
using System.Collections;
using System.Runtime.InteropServices;
public class CanvasScript : MonoBehaviour {
[DllImport("__Internal")]
private static extern void ImageUploaderCaptureClick();
IEnumerator LoadTexture (string url) {
WWW image = new WWW (url);
yield return image;
Texture2D texture = new Texture2D (1, 1);
image.LoadImageIntoTexture (texture);
Debug.Log ("Loaded image size: " + texture.width + "x" + texture.height);
}
void FileSelected (string url) {
StartCoroutine(LoadTexture (url));
}
public void OnButtonPointerDown () {
ImageUploaderCaptureClick ();
}
}
The OnButtonPointerDown should be triggered by the Pointer Down event (available through the Event Trigger component, that you can attach to your UI button)
The second method is of course not 100% reliable (for example, user can push the button, move the mouse outside the canvas, then release the button, so that the attached onclick event handler remains unprocessed), but for most use cases it should be quite appropriate.
Great news!
You may also want to try method 2. Not only you will get a native Unity button for opening a file, but you will also be able to make it cross-platform, so that your open file dialog will work both in the editor and in the browser:
Ha you read my mind, I was just scratching my head, thinking: Damn how do I make this work cross-platform Thanks!
So if I understand this correctly, if WWW-class just has a url (or filepath in this case) then it CAN load from the user’s HD even with WebGL? I was under the impression that it would not work due to security reasons but obviously I was wrong because it DOES load
Yes, it does load, but only if the requesting page has also been loaded from the local hard drive. You may think of local filesystem (i.e. file://) as of a separate domain (while in fact it is a different protocol). As soon as you upload the requesting WebGL page to the server, the file:// WWW request becomes cross-origin and therefore can not be performed (otherwise you would be able to scan local file system of any user visiting your website :))
So with all the help from you @alexsuvorov and the info provided in those links I think even I can produce a system with the ability to load images from anywhere. Nice!
I would just like to mention that setting up a simple proxy server will give you even more freedom, as you will be able to upload an image from almost any location on the web (with or without proper CORS). So that the user will be able to both paste an url of an image from the web, or upload an image from the local hard drive (like in google image search for example), all implemented using native Unity UI.
So, thanks a lot for the help earlier with this. My project is progressing. I am experiencing huge memory problems though,
Loading any image that is too large (I am talking about maybe 3MB here) crashes the web gl app. This seems very strange to me, I have tried increasing the memory by very large amounts and still this issue occurs.
I mean regardless of how the image is stored, DXT or whatever, it should never take up that much memory?? Am I missing something here?
Yes, this is indeed weird. Do you think it would be possible to reproduce the same issue using an empty project with just an image loader and nothing else? If this is the case, could you share the exact code you are using to load the texture, so that the issue can be analyzed?
So… I have been trying to do some experiments… I have constructed a simple webgl app where all you do is upload a set of images. I then tried to crash it. It required loading a handful of images, <10. Since I didn’t set memory size to more than 256 MB this isn’t terrible, but not great either. I decided to do some profiling to measure how much memory is consumed by loading an image.
When loading one image of a moderately ugly face (~1MB jpg file) the total memory usage goes from 16.5MB to 61.7MB. Looking at the individual memories it seems that the memory is changed for both unity memory (RAM I guess?) and gfx memory. As the memory increases by this much for a single image it isn’t vere surprising I run out pretty fast.
Is this working as intended or may there be a bug somewhere? Or maybe there’s something I could do to reduce memory load? I guess I could resize the image to a smaller size if needed as long as I know what limits I have to live with.
I tried to attach the zip of the project but the 3.05MB size is apparently too large for the max 4MB limit on the site
Instead I attached an image of the moderately ugly face causing this whole mess and supply a download link: Microsoft OneDrive
Actually, the size is not that big, considering that even 24-bit uncompressed is 2271 * 3452 * 3 = 23518476. The real question here, I suppose, is why the memory does not get deallocated, right?
Good point… I would’ve thought it should get compressed to DXT on the gpu and take take no room in RAM once it was loaded though. One stupid question… Does texture memory count towards the memory allocated for the web gl app?
I just tried running in editor and I get the same results. That should indicate it is all my fault I suppose I think I should probably resize the image after loading it at least and save some memory that way. I don’t think my algorithm for face segmentation or final texture building needs much more than 1024x1024 anyway…
Also, in my main project, I search the image structure of several scale spaces to find faces, so an image will take up about twice as much memory due to that. So the memory allocation for the webgl app starts filling up fast I guess.
I believe dynamically created Texture2D would retain a shadow copy of the texture data on the heap (I can double check on that). Nevertheless, lets try one experiment. Could you crash the following image uploader and let me know which image sizes / number of images can crash it (it has default 256 MB heap and updates the texture in its original resolution, so the size of the image does matter): http://files.unity3d.com/alexsuvorov/test/uploadimage/
Umh it crashes when uploading just one image if I use one that is approx. 3MB. If I use one that is approx. 1 MB then I can upload it >10 times.
I actually managed to solve the problem with my own app by resizing any uploaded image to be at the most 1k pixels on either side. I can then segment both images and create a resulting texture wihtout breaking the max memory limit. So I’m pretty happy for now
I need to do a lot of work on user interaction and stuff but at least it is possible to do segmentation and start storing avatars! If anyone want to check it out here’s a link (warning, very very very early alpha!) http://vikingcrew.net/Selfie_to_Avatar/index.html
Yes, resizing the image should reduce the memory usage, however it does not resolve the actual problem. The reason why you run out of memory is because the texture in the following code does not get destroyed between uploads:
IEnumerator LoadTexture(string url) {
WWW image = new WWW(url);
//yields until loading is done
yield return image;
Texture2D texture = new Texture2D(1, 1);
image.LoadImageIntoTexture(texture);
Debug.Log("Loaded image size: " + texture.width + "x" + texture.height + " from " + url);
OnFinishedLoadingTexture.Invoke(texture);
}
Assets are not garbage collected, therefore you should take care of this yourself using one of the following ways:
Call Destroy() on the created texture when it is no longer used.
Call Resources.UnloadUnusedAssets() to unload assets that are no longer referenced, including your texture.
Reuse the same Texture2D object (probably the most convenient way to deal with this), consider the following code:
using UnityEngine;
using System.Collections;
using System.Runtime.InteropServices;
public class upload : MonoBehaviour {
[DllImport("__Internal")]
private static extern void ImageUploaderCaptureClick();
private Texture2D displayedTexture;
IEnumerator LoadTexture (string url) {
WWW image = new WWW (url);
yield return image;
image.LoadImageIntoTexture (displayedTexture);
}
void FileSelected (string url) {
StartCoroutine(LoadTexture (url));
}
public void OnButtonPointerDown () {
#if UNITY_EDITOR
string path = UnityEditor.EditorUtility.OpenFilePanel("Open image","","jpg,png,bmp");
if (!System.String.IsNullOrEmpty (path))
FileSelected ("file:///" + path);
#else
ImageUploaderCaptureClick ();
#endif
}
void Start () {
displayedTexture = new Texture2D (1, 1);
GameObject.Find ("Quad").GetComponent<MeshRenderer> ().material.mainTexture = displayedTexture;
}
}
Note that once associated with the mesh material, the texture will be automatically uploaded to the GPU each time you modify it (i.e. using LoadImageIntoTexture), and as it is the very same object being reused, the memory should no longer grow.
I should have taken care of this in my initial code example, of course.
P.S. A small note. When using blobs to transfer data from JavaScript to WebGL, make sure to call URL.revokeObjectURL() on no longer used blob url to get it garbage collected from the JavaScript memory (for example, you can do this by calling an additional jslib callback from the LoadTexture coroutine after the blob has been downloaded). You can monitor currently allocated blobs in the Firefox memory profiler. If you don’t explicitly revoke them, they will remain allocated for the whole lifetime of the page.
but it doesnt seem to work all the time, it sets the texture for some but not all, and the ones that gets the texture ends up setting the same texture for all the gameObjects using that script to the same texture when they should all be different.
Im stuck because it runs out of memory when loading the scene in the browser. It first loads the scene, then goes around loading all the textures from different urls then it runs out of memory.
Any ideas?
I will repost this in a new thread to see if anybody has any ideas
