Working Multi-Resolution Sprite Sheets

Does anybody have strategies for how to use different sprite sheets for different resolutions? This is a very common workflow in just about every other app authoring environment I’ve worked in, and I find it very surprising there aren’t more people talking about this?

Before you say just use Reference Resolution, that is only half the solution. Inherently this means you are either upscaling low-res assets which looks poor on hi-res devices, or you are downscaling hi-res assets which looks great but incurs a huge memory hit on older devices. The official word is that we should roll our own solution:

Okay so I get that we’re expected to roll our own solution, but that is very non-trivial:

The sprite versions you’re not using should never be loaded into memory. This means:

  • You can’t reference multiple versions of sprites in the inspector, as they will consume application memory when the object is created on level load or instantiated (correct me if I’m wrong. I think you will at least avoid texture memory until it is set as the active sprite)
  • You instead need some kind of scheme involving Resources.Load. This gets tricky because the APIs for working with sprites sheets aren’t great. E.g. it’s difficult to enumerate over the sprites in a sprite sheet or to say which spritesheet asset a sprite came from.
  • You will still incur the memory hit of whatever sprite is used to set up your scene. Eg if you set up your scene using 1x assets then you will still pay the memory cost of those when switching to a 2x environment

Furthermore, you need to ensure sprites are correctly referenced at any point in it’s life cycle, such as:

  • Sprite present in the scene on Load
  • Sprites dynamically instantiated from prefabs
  • Sprites referenced in animations.

Quite frankly this sounds like a world of pain. If anybody has strategies for dealing with this I’d love to hear it. Are most people just using 2x assets, scaling down and saying to hell with older devices?

And to the devs, ideally for unity to be a competent 2D app development platform this should be automagical. There are numerous ways to go about this, but the most straightforward I can think of is specifying different scaling versions of sprites in the property inspector or asset importer.

I’m not sure how memory management would work, but I imagine there would be a global flag you could set to choose your desired resolution. This means loading the correct assets could be built into the scene loading scheme. As a dev that would be wonderful. :wink:

4 Likes

Hi!

The “official word” is a bit different now since beta 20 introduced some differences. Can you let me know where the post is you quoted so I can update it?

Basically, it doesn’t matter now if you use SD or HD as reference, text will be rendered according to the current resolution regardless. Also, sprite resolution can now be controlled by their pixelsPerUnit settings in the importer rather than having to set it from script on the Image components. There is also proper mip-map support now without having to go to the advanced settings of the texture importer.

As for how to switching between different versions of sprites, that’s still not well solved. I believe this talk by Veli from Unity has suggestions for best practices (it’s about the 2D feature but that part should apply to UI as well):

1 Like

Okay sure, was quoted by Tim C in this thread:

And wow, I really wish I’d seen that a few months ago - great resource. Anyway, I’ve arrived at pretty much the same solution as the one described in the talk:

  • An Editor script to pre-process all sprites/images/etc and replace sprite references with string references to assets in the Resources folder
  • A SpriteResolver component which fulfils these string references and loads in the correct sprite using Resources.Load() when the object is created.

This means there are no inspector references to any sprites (and thus the scene looks empty). But crucially because textures are loaded on demand from the Resources folder, it means we always have the most efficient memory profile depending on the resolution of the device.

Some points that help make this workable: by sticking to a strict naming convention and folder structure it’s actually quite feasible to find the source path of a sprite and determine if it has a matching 2x version.

Secondly, in order to layout the scene properly Sprite Resolver uses [ExecuteInMode] to load the textures in the editor. This works but is far from perfect as it creates those crucial-to-avoid inspector references which may get saved with the scene. It’s possible to customise the build process to guarantee sprites are never referenced, but I’ll leave that for a more advanced version.

I’ll post my scripts as soon as I have something stable.

For the sake of completeness I should say that another strategy seems to be mipmap level management. By generating mipmaps on your highest res textures and using QualitySettings.masterTextureLimit you can define which resolution level is drawn on device.

This has caveats: Mipmaps can’t be manually defined, so you’re relying on scaling algorithms to downscale your assets. Secondly when you’re using the base texture level (ie your highest resolution) all the mipmaps in the chain are loaded. So this means at 2x you are paying the memory price of 1x as well (plus a little bit of interest for the lower mipmap levels).

I couldn’t actually get this working right (testing on Android) but at least know the option exists.

To the devs: From an API point of view this is quite an elegant system to emulate. All of my textures are defined as a single asset and using a single API call I can change the resolution. Memory management would have to be quite magical behind the scenes, but I see no reason why it isn’t feasible. In fact a less manual version of the asset bundle approach (as highlighted in the video) could work quite well in that regard.

It’s true that older devices with SD resolutions are gradually being phased out, But phones are already coming out in ultra HD and regular pc displays are going Retina, so I don’t exactly see the problem as going away anytime soon.

I use the mipmap approach and did some performance tests to confirm that on iOS at least setting this speeds up loading and reduces the memory footprint as expected.

Note that the Unity Profiler does not report the amount of memory used by the textures correctly it still reports the full size image even though it’s never loaded. Confirmed by using large textures and watching the memory via xCode instruments.

I’d be interested if anyone has done similar tests on Android.

The main downside to this solution is that you’re wasting memory with the below the minimum size you want since as far as I’m aware all of the mipmaps down to 1x1 or some small size have to be loaded. Although on the flip side it allows you to scale your assets in transitions and get better looking results.

On the plus, these days with so many device resolutions I’ve given up on pixel perfect and expect some scaling across devices (We design so that our main device has no scaling), mip maps and trilinear should give a visual improvement when you’re between two sizes.

Ah, so if the profiler reports the wrong figure that would explain things. Does anybody know if there is an equivalent of xCode instruments for Android?

Also it’s worth mentioning the extra memory for mipmaps will always be bound to 33%.
Mip level base: 2048x2048 tex is ~16mb + 33% mip maps = ~ 22mb total
Mip level 1 : 1024x1024 tex is ~4mb + 33% mip maps = ~5.3mb total

So from a pure memory point of view this can be very effective.

As promised the scripts I’m using. They aren’t prefect but it seems to work. Biggest point to remember is that this should always be run before you execute a build to make sure there are no lingering inspector references to sprites.

Also I’m not doing anything about animations or prefabs that aren’t present in your scene:

SpriteTools.cs:

using UnityEditor;
using UnityEngine;
using UnityEngine.UI;

public class SpriteTools : Editor
{
    public const string SUFFIX_SD = "";
    public const string SUFFIX_HD = "@2x";


    /// <summary>
    /// Can be used as part of a build tool chain to make sure all sprites are mem friendly string references
    /// to textures in the Resources folder.
    /// </summary>
    [MenuItem("Sprites/Preprocess All Scenes")]
    public static void PreprocessAllScenes()
    {
        if (!EditorApplication.SaveCurrentSceneIfUserWantsTo())
        {
            return;
        }

        var currentScene = EditorApplication.currentScene;

        foreach (var scene in EditorBuildSettings.scenes)
        {
            EditorApplication.OpenScene(scene.path);
            PreprocessSprites();
        }

        EditorApplication.OpenScene(currentScene);
    }

    /// <summary>
    /// This removes all scene references and make sure they are being loaded from Resources.
    /// </summary>
    [MenuItem("Sprites/Preprocess Current Scene")]
    public static void PreprocessSprites()
    {
        if (!EditorApplication.SaveCurrentSceneIfUserWantsTo())
        {
            return;
        }
       
        //Resources.FindObjectsOfTypeAll is the same as FindObjectsOfType, except if finds disabled objects as well
        var spriteRenderers = Resources.FindObjectsOfTypeAll<SpriteRenderer>();
        for (int i = 0; i < spriteRenderers.Length; i++)
        {
            var renderer = spriteRenderers[i];
            if(MapSprite(renderer.sprite, renderer))
            {
                //A valid mapping has been created, safe to remove scene reference
                renderer.sprite = null;
            }
        }

        var imageRenderers = Resources.FindObjectsOfTypeAll<Image>();
        for (int i = 0; i < imageRenderers.Length; i++)
        {
            var image = imageRenderers[i];
            if(MapSprite(image.sprite, image))
            {
                //A valid mapping has been created, safe to remove scene reference
                image.sprite = null;
            }
        }

        EditorApplication.SaveScene();
    }

    private static bool MapSprite(Sprite sprite, Component rendererComponent)
    {
        if(sprite == null) return false;

        var path = AssetDatabase.GetAssetPath(sprite.texture);

        if(path.IndexOf("Resources/unity_builtin_extra") > -1)
        {
            //Ignore default unity assets
            return false;
        }

        if(path.IndexOf("Assets/Resources") < 0)
        {
            Debug.LogError(string.Format("The sprite {0} doesn't exist in the resources folder and will be ignored. Scene: {1} Current path: {2}",
                                    sprite.name,
                                    EditorApplication.currentScene,
                                    path)
                                );
            return false;
        }

        var resolver = rendererComponent.GetComponent<SpriteResolver>();
        if (resolver == null)
        {
            resolver = rendererComponent.gameObject.AddComponent<SpriteResolver>();
            resolver.Preinitialize();
        }

        BuildDefinitions(resolver, sprite);

        return true;
    }

    private static void BuildDefinitions(SpriteResolver resolver, Sprite sprite)
    {

        resolver.spriteName = StripName(sprite.name);
        resolver.assetPath = StripName(StripPath(AssetDatabase.GetAssetPath(sprite.texture)));

        BuildDefinitonAtScale(resolver, 1, SUFFIX_SD);
        BuildDefinitonAtScale(resolver, 2, SUFFIX_HD);
    }

    private static void BuildDefinitonAtScale(SpriteResolver resolver, float scale, string suffix)
    {
        var resolution = resolver.GetResolution(scale, false);
       
        if(resolution == null)
        {
            resolution = new SpriteResolution() {suffix = suffix, scale = scale};
            resolver.resolutions.Add(resolution);
        }

        if (!CheckResourceExists(resolver.spriteName, suffix, resolver.assetPath))
        {
            resolver.resolutions.Remove(resolution);
            Debug.LogError(string.Format("{0} has a missing resolution at scale {1}", resolver.spriteName, scale));
        }
    }

    private static bool CheckResourceExists(string spriteName, string suffix, string assetPath)
    {
        //Sadly it seems to only way to verify the existence of sprites is to load them.
        //Though this only happens in the editor, so should have no gameplay memory consequences

        var sprites = Resources.LoadAll<Sprite>(assetPath + suffix);

        if (sprites == null || sprites.Length < 1) return false;

        for (int i = 0; i < sprites.Length; i++)
        {
            if (spriteName + suffix == sprites[i].name) return true;
        }
        return false;
    }

    [MenuItem("Sprites/Test SD")]
    private static void TestSD()
    {
        var spriteResolvers = FindObjectsOfType<SpriteResolver>();
        foreach (var sr in spriteResolvers)
        {
            sr.ForceSD();
        }
    }

    [MenuItem("Sprites/Test HD")]
    private static void TestHD()
    {
        var spriteResolvers = FindObjectsOfType<SpriteResolver>();
        foreach (var sr in spriteResolvers)
        {
            sr.ForceHD();
        }
    }


    #region Utils
    private static bool IsHD(string name)
    {
        if(name.IndexOf(SUFFIX_HD) > -1)
        {
            return true;
        }
        return false;
    }

    private static string StripName(string name)
    {
        var stripped = name;
        if(SUFFIX_SD.Length> 0)
        {
            stripped = stripped.Replace(SUFFIX_SD, string.Empty);
        }
        if(SUFFIX_HD.Length > 0)
        {
            stripped = stripped.Replace(SUFFIX_HD, string.Empty);
        }
        return stripped;
    }

    private static string StripPath(string path)
    {
        var stripped = path.Replace("Assets/Resources/", string.Empty);
        var ind = stripped.LastIndexOf(".");
        stripped = stripped.Substring(0, ind);
        return stripped;
    }

    private static string GetExtension(string filepath)
    {
        var result = filepath.Split(".".ToCharArray());
        return "." + result[result.Length - 1];
    }
    #endregion

}

SpriteResolver.cs:

using System.Collections.Generic;
using UnityEngine;
using System.Collections;
using UnityEngine.UI;

[ExecuteInEditMode]
public class SpriteResolver : MonoBehaviour
{
    //Set this on Applicaiton start up to determine which sprites get loaded.
    public static float scaleFactor = 1;

    public string spriteName;
    public string assetPath;

    public List<SpriteResolution> resolutions;

    private SpriteRenderer spriteRenderer;
    private Image imageRenderer;

    //Called from editor script
    public void Preinitialize()
    {
        if(resolutions == null)
        {
            resolutions = new List<SpriteResolution>();
        }
    }

    void Awake()
    {
        spriteRenderer = GetComponent<SpriteRenderer>();
        imageRenderer = GetComponent<Image>();

        //No point loading if the component has just been created by the editor
        if (resolutions != null && resolutions.Count > 0)
        {
            Load(GetResolution());
        }
    }

    private void Load(SpriteResolution resolution)
    {
        var sprite = GetSprite(resolution);

        if(sprite == null)
        {
            Debug.LogError(string.Format("The sprite {0} doesn't exist", spriteName));
        }

        if (spriteRenderer != null)
        {
            spriteRenderer.sprite = sprite;
        }
        if (imageRenderer != null)
        {
            imageRenderer.sprite = sprite;
        }
    }

    private SpriteResolution GetResolution()
    {
        //No point seeing high res on pc //Can override for testing
        if (Application.isEditor || !Application.isMobilePlatform)
        {
            return GetResolution(1);
        }

        //Normal behavior
        return GetResolution(scaleFactor);
    }

    public SpriteResolution GetResolution(float scale, bool fallback = true)
    {
        for (int i = 0; i < resolutions.Count; i++)
        {
            if(resolutions[i].scale == scale)
            {
                return resolutions[i];
            }
        }

        //Try accomodate missing assets
        if(fallback && resolutions.Count > 0)
        {   
            Debug.LogError(string.Format("The sprite {0} doesn't exist at scale {1}, falling back to scale {2}", spriteName,scale, resolutions[0].scale));
            return resolutions[0];
        }

        return null;
    }

    private Sprite GetSprite(SpriteResolution resolution)
    {
        if(resolution == null)
        {
            return null;
        }

        var fullAssetPath = assetPath + resolution.suffix;
        var sprites = Resources.LoadAll<Sprite>(fullAssetPath);

        if (sprites == null || sprites.Length < 1)
        {
            Debug.LogError("No sprite assets can be found at the path " + fullAssetPath + ". Please be sure to reprocess sprites");
            return null;
        }

        Sprite sprite = null;

        for (int i = 0; i < sprites.Length; i++)
        {
            string fullname = spriteName + resolution.suffix;
            if (fullname == sprites[i].name)
            {
                sprite = sprites[i];
                break;
            }
        }

        return sprite;

    }

    //Use this for testing form the editor to check which sprites are present at each resolution
    public void ForceHD()
    {
        Load(GetResolution(2, false));
    }

    //Use this for testing form the editor to check which sprites are present at each resolution
    public void ForceSD()
    {
        Load(GetResolution(1, false));
    }
}


[System.Serializable]
public class SpriteResolution
{
    public float scale = 1;
    public string suffix;
}
7 Likes

Great work, looks good. Been wanting a solution like this in Unity since we started playing with the new UI. 2D Toolkit had a really nice way of handling it, but as this isn’t available in Unity, we have used 4x assets and hoping for the best. Its not been tested on low end devices yet, but this will definitely ensure we don’t blow memory.

Thanks for sharing your solution!

Unfortunately, I don’t think sprites located in “Resources” folders are getting packed by the sprite packer: Am I missing the plot with Sprite Packer Atlas' - Unity Engine - Unity Discussions

I did a quick test and confirmed those results.

I’m still looking for a solution that works like Reference Atlases in NGUI…

Yep you’re correct, take a look at http://forum.unity3d.com/threads/unity-4-5-sprite-packer-does-not-pack-images-inside-resources-folder.248349/ It appears to be the current thinking of ‘correct’ functionality but I think it’s really an oversight as packing them would not introduce double storage and would instead save memory and provide the functionality we are after.

I’m in research also for get a solution like this. but i’m doing it for my interfaces with unity 4.6

All the “tileable” elemets are not problems, they are not resized thanks to the sliced sprites, but for some backgrounds and avatar images that i have, its a mess.

And i need all of these images in the view on editor for the GUI guy (The ones who makes the interfaces visually) needs to see them :frowning:

Interesting, I was actually mostly unaware of Unity’s default packing options.

I can happily report though that sprite sheets imported from TexturePacker work as expected in the Resources folder.

I was looking for a solution to this, also possibly 2dtk based like the kujo mentioned above. I’ll give the script a run later on, if I make any improvements i’ll report back.

I’m surprised there’s no solution to this yet. 2D Toolkit handles it perfectly with the sprite collections, but would be great to see a “native” solution.

I’m disappointed this is still an issue in Unity 5. Could we get an update from Unity on this? Can we expect a solution soon?

The funny thing is unity UI has the canvas scaler, so really its clear that it would take a engine dev a few minutes to whipup a built-in, editor based(no coding on our side), all encompassing, multi device/resolutin solution… yet it doesnt happen.

That point of view is naïve, Kiori.

Due to the huge number of devices, aspect ratios and resolutions, it is not possible to automate UI design for every possible situation - and more importantly - every possible game.

These decisions need to be made by the team developing the project.

Each situation, each device and each game or app has it’s own requirements.

Control schemes can vary immensely between devices, and even more between platforms.

When developing your project in Unity, these are decisions you need to make.

There are many tools, including anchors, the canvas scaler and more, at the disposal of the project developers such as yourself… but whether a UI is scaled, anchored or completely re-created depending on the target device must be a decision by the people building the project based on their project and the targeted devices.

We have a session on Resolution Independence in the Live Training Archive.

If, for some reason, I don’t understand your point of view, and you do find it easy to do: please create this package and release it on the Asset Store! I’m sure people will find it a useful resource. Just be aware that this will probably not be trivial.

Now, these people claim to have a solution: https://www.assetstore.unity3d.com/en/#!/content/2116 They’ve been working hard at developing this solution. I’ve not tested it.

@Adam-Buckner_1

It sounds like you’ve misunderstood the problem.

This isn’t an issue about anchors, placement, sizes or aspect ratios. It’s about loading low or high resolution images depending on the pixel density (or perhaps other criteria) of the running device. It’s a concept that is built right into Android and iOS because it’s essential for mobile development. Unity doesn’t have built-in support for this like native iOS/Android development does and it doesn’t seem like there is a good way to work around that.

Please take the time to read through this thread. So far it seems impossible to do things “the Unity way” without wasting memory or cpu/graphics power. This issue does come up during the question period of your Resolution & Device Independence video at around this point www.youtube.com/watch?v=ezeoYnLBpnE&t=35m30s. The issue is eventually left off with “put the images in your resources folder and load from there” which is what TheFuntastic’s script above does. But then read along further in this thread and it’s noted that sprites in the Resources folder don’t get packed by the sprite packer.

If I’m wrong about it not being possible, then please do explain how to accomplish this.

Also, FYI, it rubs customers the wrong way when a Unity employee’s response to a basic problem in Unity is to spend an extra $150 on the asset store. And judging by the reviews, it doesn’t even support Unity 5. I’d say there’s a good chance it’s just another abandoned asset.

So could @Adam-Buckner_1 , @runevision or any other Unity employee let us know if anything is in the works to address this and if so, what priority is it? If there is no plan to address this, please let us know in that case as well so we can make decisions accordingly.

Sprite swap (using different sprites depending on resolution or other criteria) affects all usage of sprites and is under the domain of 2D Team. UI Team is not actively involved. I’ve notified 2D Team about this thread.

5 Likes
  • 1 on this subject. It would really be nice if unity would provide a built in solution for displaying assets of different resolutions. We are rebuilding one of our UIs really soon. For now we might be using a solution involving asset bundles and some scripting for ui itself. Basically we will try to achieve result where we can see all assets in stage (so that we can work in editor with all assets visible) but save stage with no assets assigned and reassign them during runtime after selecting proper assetbundle or folder in resources. Assetbundle sounds little better due to async loading.