Texture2D.LoadImage is slow. Here's a workaround. @Alexey

Hey guys, particularly Alexey. ( bug report case # 605471)
Texture2D.LoadImage is slow. I’ve created a method to asynchronously load textures from files on disk on iOS. This is fast as hell and runs circles around Texture2D.LoadImage. Basically, you provide an URL of the image on disk you want to load, and it’ll load the file asynchronously. I haven’t heavily tested my method that resizes the texture if you ask it to, but resize == false version I tested a bunch.

usage:
mediaTexture = FastTexture2D.CreateFastTexture2D(media.url, false, media.w, media.h, GotFastTexture);
(GotFastTexture is a delegate of Action OnFastTexture2DLoaded; )

Here’s a C# class:

using UnityEngine;
using System.Collections;
using System.Collections.Generic;
using System;
using System.IO;
using System.Runtime.InteropServices;




public class FastTexture2D : ScriptableObject {
//(c) Brian Chasalow 2014 - brian@chasalow.com
    [AttributeUsage (AttributeTargets.Method)]
    public sealed class MonoPInvokeCallbackAttribute : Attribute {
        public MonoPInvokeCallbackAttribute (Type t) {}
    }

    [DllImport ("__Internal")]
    private static extern void DeleteFastTexture2DAtTextureID(int id);
    [DllImport ("__Internal")]
    private static extern void CreateFastTexture2DFromAssetPath(string assetPath, int uuid, bool resize, int resizeW, int resizeH);
   

    [DllImport ("__Internal")]
    private static extern void RegisterFastTexture2DCallbacks(TextureLoadedCallback callback);

    public static void CreateFastTexture2D(string path, int uuid, bool resize, int resizeW, int resizeH){
        #if UNITY_EDITOR
        #elif UNITY_IOS
        CreateFastTexture2DFromAssetPath(path, uuid, resize, resizeW, resizeH);
        #endif
    }
   
    public static void CleanupFastTexture2D(int texID){
        #if UNITY_EDITOR
        #elif UNITY_IOS
        DeleteFastTexture2DAtTextureID(texID);
        #endif
    }


    private static int tex2DCount = 0;
    private static Dictionary<int, FastTexture2D> instances;
    public static Dictionary<int, FastTexture2D> Instances{
        get{
            if(instances == null){
                instances = new Dictionary<int, FastTexture2D>();
            }
            return instances;
        }
    }

    [SerializeField]
    public string url;
    [SerializeField]
    public int uuid;
    [SerializeField]
    public bool resize;
    [SerializeField]
    public int w;
    [SerializeField]
    public int h;
    [SerializeField]
    public int glTextureID;
    [SerializeField]
    private Texture2D nativeTexture;
    public Texture2D NativeTexture{ get{ return nativeTexture; }}

    [SerializeField]
    public bool isLoaded = false;

    public delegate void TextureLoadedCallback(int nativeTexID, int original_uuid, int w, int h);

    [MonoPInvokeCallback (typeof (TextureLoadedCallback))]
    public static void TextureLoaded(int nativeTexID, int original_uuid, int w, int h){
        if(Instances.ContainsKey(original_uuid)  nativeTexID > -1){
            FastTexture2D tex = Instances[original_uuid];
            tex.glTextureID = nativeTexID;
            tex.nativeTexture = Texture2D.CreateExternalTexture(w, h, TextureFormat.ARGB32, false, true, (System.IntPtr)nativeTexID);
            tex.nativeTexture.UpdateExternalTexture( (System.IntPtr)nativeTexID);
            tex.isLoaded = true;
            tex.OnFastTexture2DLoaded(tex);
        }
    }


    private Action<FastTexture2D> OnFastTexture2DLoaded;

    protected void InitFastTexture2D(string _url, int _uuid, bool _resize, int _w, int _h, Action<FastTexture2D> callback){
        this.url = _url;
        this.uuid = _uuid;
        this.resize = _resize;
        this.w = _w;
        this.h = _h;
        this.glTextureID = -1;
        this.OnFastTexture2DLoaded = callback;
        this.isLoaded = false;
    }

    private static bool registeredCallbacks = false;
    private static void RegisterTheCallbacks(){
        if(!registeredCallbacks){
            registeredCallbacks = true;
            #if UNITY_IOS
            if(Application.platform == RuntimePlatform.IPhonePlayer)
                RegisterFastTexture2DCallbacks(TextureLoaded);
            #endif

        }
    }


    //dimensions options: if resize is false, w/h are not used. if true, it will downsample to provided dimensions.
    //to create a new texture, call this with the file path of the texture, resize parameters,
    //and a callback to be notified when the texture is loaded.
    public static FastTexture2D CreateFastTexture2D(string url,bool resize, int assetW, int assetH, Action<FastTexture2D> callback){
        //register that you want a callback when it's been created.
        RegisterTheCallbacks();
        //the uuid is the instance count at time of creation. you pass this into the method to grab the gl texture, and it returns the gl texture with this uuid
        int uuid = tex2DCount;
        tex2DCount = (tex2DCount +1 ) % int.MaxValue;

        FastTexture2D tex2D = ScriptableObject.CreateInstance<FastTexture2D>();
        tex2D.InitFastTexture2D(url, uuid, resize, assetW, assetH, callback);
        //call into the plugin to create the thing
        CreateFastTexture2D(tex2D.url, tex2D.uuid, tex2D.resize, tex2D.w, tex2D.h);

        //add the instance to the list
        Instances.Add(uuid, tex2D);

        //return the instance, someone might want it (but they'll get it with the callback soon anyway)
        return tex2D;
    }
   

    private void CleanupTexture(){
        isLoaded = false;

        //delete the gl texture
        if(glTextureID != -1)
            CleanupFastTexture2D(glTextureID);
        glTextureID = -1;

        //destroy the wrapper object
        if(nativeTexture)
            Destroy(nativeTexture);

        //remove it from the list so further callbacks dont try to find it
        if(Instances.ContainsKey(this.uuid))
            Instances.Remove(this.uuid);
    }

    //to destroy a FastTexture2D object, you call Destroy() on it.
    public void OnDestroy(){
        CleanupTexture();
    }

}

and you’ll need this code in a .mm (i use it in an ARC-enabled compiled .a plugin, so no [whatever release] is necessary. )
and you will need this dependency: https://github.com/coryalder/UIImage_Resize

    extern "C" EAGLContext* ivcp_UnityGetContext();
    typedef void (*TextureLoadedCallback)(int texID, int originalUUID, int w, int h);
    static TextureLoadedCallback textureLoaded;
    static GLKTextureLoader* asyncLoader = nil;
    //(c) Brian Chasalow 2014 - brian@chasalow.com
    void RegisterFastTexture2DCallbacks(void (*cb)(int texID, int originalUUID, int w, int h)){
        textureLoaded = *cb;
    }
   
    void CreateFastTexture2DFromAssetPath(const char* assetPath, int uuid, bool resize, int resizeW, int resizeH){
        @autoreleasepool {
            NSDictionary* options = [NSDictionary dictionaryWithObjectsAndKeys:
                                     [NSNumber numberWithBool:YES],
                                     GLKTextureLoaderOriginBottomLeft,
                                     nil];
            //maybe look here?
//            http://stackoverflow.com/questions/16043204/handle-large-images-in-ios
            NSString* assetPathString = [NSString stringWithCString: assetPath encoding:NSUTF8StringEncoding];
           
            if(asyncLoader == nil)
            asyncLoader = [[GLKTextureLoader alloc] initWithSharegroup:[ivcp_UnityGetContext() sharegroup]];
           
            if(resize){
                UIImage* img = [UIImage imageWithContentsOfFile:assetPathString];
                __block UIImage* smallerImg = [img resizedImage:CGSizeMake(resizeW, resizeH) interpolationQuality:kCGInterpolationDefault ];
                               
                [asyncLoader textureWithCGImage:[smallerImg CGImage]
                                        options:options
                                          queue:NULL
                              completionHandler:^(GLKTextureInfo *textureInfo, NSError *outError) {
                                  if(outError){
                                    smallerImg = nil;
                                    NSLog(@"got error creating texture at path: %@. error: %@ ", assetPathString,[outError localizedDescription] );
                                      textureLoaded(-1, uuid, 0, 0);
                                  }
                                  else{
                                      textureLoaded(textureInfo.name, uuid, resizeW, resizeH);
                                  }
                              }];
               
            }
            else{
                [asyncLoader textureWithContentsOfFile:assetPathString
                                        options:options
                                          queue:NULL
                              completionHandler:^(GLKTextureInfo *textureInfo, NSError *outError) {
                      if(outError){
                          NSLog(@"got error creating texture at path: %@. error: %@ ", assetPathString,[outError localizedDescription] );
                              NSLog(@"returning texID -1 ");
                          textureLoaded(-1, uuid, 0, 0);
                      }
                      else
                      {
                          //this will get returned on the main thread cuz the queue above is NULL
                        textureLoaded(textureInfo.name, uuid, textureInfo.width, textureInfo.height);
                      }
                  }];
            }
        }
    }
   
    void DeleteFastTexture2DAtTextureID(int texID){
        @autoreleasepool {
            GLuint texIDGL = (GLuint)texID;
            if(texIDGL > 0){
                if(glIsTexture(texIDGL)){
//                    NSLog(@"deleting a texture because it's a texture. %i", texIDGL);
                    glDeleteTextures(1, &texIDGL);
                }
            }
        }
    }

and you’ll need to inject ivcp_UnityGetContext() into the UnityAppController to get the surface.context, as in

extern "C" EAGLContext* ivcp_UnityGetContext()
{
    return GetAppController().mainDisplay->surface.context;
}

or use any other means of getting the render context that you desire.

I hope you guys can integrate something along these lines into Unity directly.
Maybe one day I’ll make a plugin and release it on the asset store, but I’d rather you guys just fix it.
edit: fixed a bug that i inadvertently added as i cleaned up code to post it :wink:

1 Like

if i am allowed to whine: you could cut the code quite considerably and get rid of GLKit dependency if you used CVTextureCache we provide (in /Classes/Unity :wink:
also, we kinda provide more and more ways to make stuff behind unity’s back, e.g. we’ve added (sorry, i dont remember exact version, so it if is not yet out, sorry) Texture2D.LoadRawTextureData (though it wont do png conversion, sure).
so for me it is more about giving you tools not implementing everything :wink:

You’re allowed! :smiley: So if I’m following correctly, you mean:

  1. loading a UIImage from disk
  2. grabbing its CVImageBufferRef, via iphone - Convert UIImage to CVImageBufferRef - Stack Overflow
  3. passing it to CVOpenGLESTextureCacheCreateTextureFromImage, and the rest is the same?

That’s not a bad idea, although it’s not asynchronous, but would cut out the GLKit dependency which is a big plus. I’ll stick with the asynchronous method for now because background image loading is a huge win for this app i’m working on.

Yeah I hadn’t really poked at LoadRawTextureData, and no idea what that’s doing behind the scenes.

@brianchasalow Thanks so much for the code. It needed some fixes before I got it to work, but if anyone else is looking for it, here it is:

The C# class above works as-is.

You need to put one file in Plugins/iOS:

FastTex2D.mm
Note: I commented out the resize part, as i didn’t need it, if you want to use it un-comment it, put the includes provided above in the same folder and add the imports at the top of the file.

//(c) Brian Chasalow 2014 - brian@chasalow.com
// Edits by Miha Krajnc
#import <GLKit/GLKit.h>

extern "C" {

  typedef void (*TextureLoadedCallback)(int texID, int originalUUID, int w, int h);
  static TextureLoadedCallback textureLoaded;
  static GLKTextureLoader* asyncLoader = nil;

  void RegisterFastTexture2DCallbacks(void (*cb)(int texID, int originalUUID, int w, int h)){
      textureLoaded = *cb;
  }

  void CreateFastTexture2DFromAssetPath(const char* assetPath, int uuid, bool resize, int resizeW, int resizeH){
      @autoreleasepool {
          NSDictionary* options = [NSDictionary dictionaryWithObjectsAndKeys:
                                   [NSNumber numberWithBool:YES],
                                   GLKTextureLoaderOriginBottomLeft,
                                   nil];

          NSString* assetPathString = [NSString stringWithCString: assetPath encoding:NSUTF8StringEncoding];

          if(asyncLoader == nil) {
              asyncLoader = [[GLKTextureLoader alloc] initWithSharegroup:[[EAGLContext currentContext] sharegroup]];
          }

          if(resize){
              // UIImage* img = [UIImage imageWithContentsOfFile:assetPathString];
              // __block UIImage* smallerImg = [img resizedImage:CGSizeMake(resizeW, resizeH) interpolationQuality:kCGInterpolationDefault ];
              //
              // [asyncLoader textureWithCGImage:[smallerImg CGImage]
              //                         options:options
              //                           queue:NULL
              //               completionHandler:^(GLKTextureInfo *textureInfo, NSError *outError) {
              //                   if(outError){
              //                     smallerImg = nil;
              //                     NSLog(@"got error creating texture at path: %@. error: %@ ", assetPathString,[outError localizedDescription] );
              //                       textureLoaded(-1, uuid, 0, 0);
              //                   }
              //                   else{
              //                       textureLoaded(textureInfo.name, uuid, resizeW, resizeH);
              //                   }
              //               }];

          } else {
              [asyncLoader textureWithContentsOfFile:assetPathString
                           options:options
                           queue:NULL
                           completionHandler:^(GLKTextureInfo *textureInfo, NSError *outError) {
                    if(outError){
                        NSLog(@"got error creating texture at path: %@. error: %@ ", assetPathString,[outError localizedDescription] );
                        NSLog(@"returning texID -1 ");
                        textureLoaded(-1, uuid, 0, 0);
                    }
                    else
                    {
                      //this will get returned on the main thread cuz the queue above is NULL
                      textureLoaded(textureInfo.name, uuid, textureInfo.width, textureInfo.height);
                    }
                }];
          }
      }
  }

  void DeleteFastTexture2DAtTextureID(int texID){
      @autoreleasepool {
          GLuint texIDGL = (GLuint)texID;
          if(texIDGL > 0){
              if(glIsTexture(texIDGL)){
                  NSLog(@"deleting a texture because it's a texture. %i", texIDGL);
                  glDeleteTextures(1, &texIDGL);
              }
          }
      }
  }
}

After this, select the FastTex2D.mm file in unity, go to “Rarely used frameworks” in the inspector and select GLKit.

Now you must manually disable Metal as the Graphics API. Go to player settings and uncheck Auto Graphics API and remove metal from the list. Make sure you only have OpenGLES2 in the list.

That’s all the setup you need, then to use the async loader you need to put your image files in a StreamingAssets folder, with the .bytes extension (I used png files), and in your code, you would run something like:

public class TexLoad : MonoBehaviour {
    void Start () {
        FastTexture2D.CreateFastTexture2D (Application.streamingAssetsPath + "/tex.bytes", false, 0, 0, Callback);
    }

    private void Callback(FastTexture2D t){
        GetComponent<SpriteRenderer> ().sprite = Sprite.Create (t.NativeTexture, new Rect (0, 0, t.NativeTexture.width, t.NativeTexture.height), new Vector2 (.5f, .5f));
    }
}

And voila, async texture loading. I’m stumped why Unity doesn’t natively support this yet, but never give up hope!

2 Likes

Great stuff- I’m glad it worked for you.

So what’s the reason Unity doesn’t natively support it? Is it the classic “there’s nobody among our 500+ staff that has time” ? :wink:

We’ve optimized slow texture loading around a year ago. After improvement, it’s possible to achieve comparable performance using entirely Unity’s APIs.

Do you mean with Texture2D.LoadImage? I have a feeling that this method still works slightly faster, but it might not be the case.
In any way, my problem was that there is no way to asynchronously load a texture with the Unity API. I have a 2D game that uses around 20 2700x1700 images as backgrounds and using unity’s Textuer2D.LoadImage has a visible lag of around half a second to a second. As I want to display an animated loading screen, this will not work.

Using unity’s build in system doesn’t work for me either. If I set the image quality of the sprite to TrueColor all images end up at around 16MB which adds up to a lot of space on the disk of a mobile device. Setting it to compressed, and putting them in a POT texture / sprite sheet degrades the quality too much.

Is there another way that you could do this that I am missing?

EDIT: Another thing: Does any iOS guru know why this isn’t working on iPhones. iPads work fine, but when running this code on an iPhone it crashes, because the the EAGLContext from MyAppController is null…

mihakinova: you might be able to get away with [EAGLContext currentContext] instead of getting it from MyAppController.

and re: Povilas: the question was about Asynchronous texture loading, not just texture loading performance, and Unity’s LoadImage is still a synchronous process as far as I am aware (or at least is on the UI thread) and causes visible jitter. Even if its per-second-loading-time performance has improved, that’s not good enough for many use cases.

@brianchasalow Sadly it doesn’t work. The weird thing though, is that it does work on iPads… and I could have sworn that it didn’t the first time I tried it… But it returns nil on iPhones. Any ideas?

Heureka! The fix is a simple yet stupid one. The reason it wasn’t working on my iPhone is that it has iOS 9. And my iPad has iOS 7.
And so, Unity automatically used Metal as the Graphics API, so of course unity didn’t find the context, because there wasn’t one. And when i created it and sent the pointer to Texture2D.CreateExternalTexture it broke.

The solution is simple: Go to your player settings and uncheck Auto Graphics API, then remove Metal. This fix was good enough for my game, as I don’t really need Metal.

In iOS 9.0 there’s something called MetalKit which effectively does the same thing as GLKit’s Texture loader. Unfortunately you might not want to restrict your users to iOS 9-only.
See MTKTextureLoader :

texture2d.loadimage is still terribly slow, and an async method is needed too. Happy freezing while loading textures form my skin editor…

2 Likes