Texture2D.GetPixels without alloc?

Graphics
1

Hi,

I’m currently trying to have a “trail” effect with a simple system of RenderTexture and Texture2D. Here is what I have:

  • I have an Image with the current trails, which is a Texture2D
  • I capture the elements I want a trail on on a renderTexture
  • I render this on a Temp Texture2D
  • I fade every pixels on my Image Texture2D
  • I write above this Image the pixels of the Temp Texture2D which are not clear

It works as intended, but the performance is not too great, mainly because of the last two steps.
I tried to do it in two ways:

  • A double for loop with GetPixel and SetPixel calls > 0 alloc, but far too slow.
  • A double GetPixels on my temp and image Texture2D to have 2 Color I iterate on, then one final SetPixels > way faster, but allocates some MB each frame, which is clearly not acceptable (it triggers the GC.Collect very frequently, obviously).

I have spent a lot of time already looking for solutions, so I’m not sure there is one, but maybe I missed something, so here I am for help :slight_smile:
I don’t see a good reason why the GetPixels function couldn’t have an override with a Color parameter, in order to not generate alloc (I create the array once, then the same one is always used), the same way some Physics functions work.

Here is the code if you’re curious (or if you can spot some dumb error I made):

public class RemanentCamera : MonoBehaviour
{

    public UnityEngine.UI.RawImage destImage;


    Camera c;


    Texture2D tempTexture;

    Texture2D imageTexture;



    int width;

    int height;



    void Start ()
    {
        height = 256;
        width = Screen.width * height / Screen.height;

        c = GetComponent<Camera>();
        c.targetTexture = new RenderTexture(width, height, 16);
        tempTexture = new Texture2D(width, height, TextureFormat.ARGB32, false);
        imageTexture = new Texture2D(width, height, TextureFormat.ARGB32, false);
        destImage.texture = imageTexture;
    }



    void OnDestroy()
    {
        c.targetTexture.Release();
    }

    void Update()
    {
        RenderTexture.active = c.targetTexture;

        tempTexture.ReadPixels(new Rect(0, 0, width, height), 0, 0);

        RenderTexture.active = null;

        //FIRST SOLUTION
        {
            for(int i = 0; i < imageTexture.width; i++)
            {
                for(int j = 0; j < imageTexture.height; j++)
                {
                    Color tempColor = tempTexture.GetPixel(i, j);
                    if(tempColor.a == 1)
                    {
                        imageTexture.SetPixel(i, j, tempColor);
                    }
                    else
                    {
                        tempColor = imageTexture.GetPixel(i, j);
                        if(tempColor.a > 0)
                        {
                            tempColor.a = Mathf.Max(0f, tempColor.a - Time.deltaTime * 2f);
                            imageTexture.SetPixel(i, j, tempColor);
                        }
                    }
                }
            }
        }

        //SECOND SOLUTION
        {
            Color[] tempColor = tempTexture.GetPixels();
            Color[] currentColor = imageTexture.GetPixels();
            for(int i = 0; i < currentColor.Length; i++)
            {
                if(tempColor[i].a == 1)
                {
                    currentColor[i] = tempColor[i];
                    fullPixelAmount++;
                }
                else
                {
                    if(currentColor[i].a > 0)
                    {
                        currentColor[i].a = Mathf.Max(0f, currentColor[i].a - Time.deltaTime * 2f);
                        fadePixelAmount++;
                    }
                    if(tempColor[i].a > 0)
                    {
                        currentColor[i].r = tempColor[i].a * tempColor[i].r + (1f - tempColor[i].a) * currentColor[i].r;
                        currentColor[i].g = tempColor[i].a * tempColor[i].g + (1f - tempColor[i].a) * currentColor[i].g;
                        currentColor[i].b = tempColor[i].a * tempColor[i].b + (1f - tempColor[i].a) * currentColor[i].b;
                        currentColor[i].a = Mathf.Min(1f, currentColor[i].a + tempColor[i].a);
                        blendPixelAmount++;
                    }
                }
            }
            imageTexture.SetPixels(currentColor);
        }

        imageTexture.Apply();
    }

}

I’m also open to other ways of getting the same result, it’s the only way I found.

2

The alloc isn’t the slow part you should be concerned about (though that is slow). The slow part is you’re using any of these kinds of texture manipulation functions to begin with to do something that should be done as an image effect. I.e.: all shaders & render textures. Generally, apart from some very special cases, if you’re using ReadPixels, or SetPixels for an effect that you want to be real time, you’ve probably done something wrong.

This is very similar to how motion blur was done during the PS2 era. Check out the Motion Blur script in standard assets.
https://docs.unity3d.com/Manual/script-MotionBlur.html

3

That’s a valid point. I looked a bit around about motion blur shaders, but I couldn’t find anything to handle a per-object blur, nor a way to do this with shader forge (from what I’ve seen, it’s done using an accumulation buffer, and I didn’t find that in Shader forge - I couldn’t write manually a shader even if my life was at stake) :frowning: It’s good to read you saying that though, I guess I’ll keep looking a bit more into this motion blur thing.
Just for the record, if you put aside the gc.collect, the performance was almost ok with a 512pixels texture, running at a bit more than 60fps (with nothing else beside).

Edit: Started to check the effects in the standard assets (motion blur and camera motion blur), and for now I couldn’t find a way to do the same kind of effect you can see in the gif of my first post :confused: My goal is not really to blur the object when it’s moving, but to keep a trail of its rendering on the screen. (But I’ve just started, I’m gonna keep investigating!)

4

He is right though, I have a long history with the Texture2D class, and its Apply() and SetPixel/SetPixels() methods…

I hate to break it to you, but for realtime “stuff” to do with editing stuff at a per pixel level… unless your gonna be working with teeny tiny little images (or like, 10k of them to represent a larger image) then the Apply() call is tremendously resource intensive, as it has to push data to the GPU from the CPU side of things, and this chokes the memory bridge between them, on desktops, and severely on mobiles.

So your best bet is to avoid the crap out of those methods unless your planning on doing a boatload of invisible-to-the-end-user work, like making a huge grid of tiny images that eventually will bottleneck you with draw calls or just plain too many gameobjects.

If you HAVE to do per pixel editing of textures, the best option I have found is a compute shader written specifically for the task, that keeps the texture on the GPU end of things, and makes changes on that end as well. This avoids the bottleneck of the memory bridge between the GPU and CPU, and also the GPU is really good at doing just that, stuff with pixels.

Alternatively, and probably better in your case, and more compatible with current available mobile hardware (if you ever target mobile idk…) is to use a normal shader, and create a blurring/trail effect the old fashioned way, in a normal shader. I don’t know how to do that, and only know very little about compute shaders so far, but if your interested in pursuing a compute shader example, you might check out this thread:

5

Well I don’t know why I didn’t try that yet, since it was my initial intention when I first tried to do that a few weeks ago, but I think I can see how to do it now, with what I’ve learned since that! Gonna give it a new chance, with a shader fed with the “current state” and the “trail state” captured by an additional camera! (That’s a lot of cameras for one simple thing, but no pixel manipulation on the CPU side, so it should be better!)

Thank you both for your help, gonna keep this thread updated.

Edit: I am gonna keep looking at the blur shader option as well, if I can figure out a way to use a buffer in my shader, maybe the color buffer?

6

Soo I’ve spent a lot of time continuing experimentations and it’s slowly starting to work, but there are still some limitations I’m not sure I understand…

This time all the calculations are done inside shaders. There may be a simplier way, but here is how it’s done:

  • 1 Background camera
  • 1 “Trailed objects” camera than renders the sphere on a “CurrentTrailedObjects” render texture.
  • 1 “Trail mix” camera, that renders a blend of this “CurrentTrailedObjects” texture with a “CurrentTrail” texture with a slightly raised alpha, on a “Mix” render texture.
  • 1 “Back to trail” camera, that overrides the “CurrentTrail” texture with the content of the “Mix” texture (since I cannot render a texture over itself).
  • FInally, a last camera to render this “Mix” texture on the screen.

That’s a lot of camera, but I couldn’t find a way to use less.
I had a lot of trouble setting up the cameras properly, because my initial instinct made me setup them with “Depth only” clear flags, and for some reason I’m not sure I totally understand, it won’t work (the trail doesn’t fade). So I finally tried with “Solid Color” (with a 0 alpha), and you can see the result above.

Yet it’s still not perfect, I wanted the trail to fade wayy slower, but if I reduce the fade speed, it just doesn’t fade anymore. At first I thought it may be due to the precision used in the shader, and indeed Shader Forge returned a fixed4 value in the frag part. However, I tried to replace it with a float4, and it did not change anything.

I’m updating the DeltaTime variable each frame to have a smooth and constant fade, but somehow it doesn’t work as I think it should. Even with a value where the trail still fades, it fades a lot faster than my “_FadeDuration” variable.
If I set “_FadeDuration” to 10, the trail doesn’t fade anymore, until an update is a bit longer (that is, _DeltaTime is bigger), and then the whole trail instantly disappear (or sometimes quickly fades).

I’m not sure there’s a solution to this, maybe what I want to do just can’t be achieved this way, but if anybody has an idea, I’d glady try it!

7

I have tried something a little bit different, and now I’m definitely sure there’s something weird going on I don’t understand; rather than setting the _DeltaTime in my shader, I just give it once in a while a _FrameFadeAmount (equal to 0 the rest of the time) in order to dispatch the fade.

In code:

    public float fadeDuration = 2f;
    float didntFadeSince;
    const float minAlphaDiff = 0.002f;

    void Update()
    {
        float fadeAmount = (Time.realtimeSinceStartup - didntFadeSince) / fadeDuration;
        if(fadeAmount >= minAlphaDiff)
        {
            remanenceGroupPlane.material.SetFloat("_FrameFadeAmount", fadeAmount);
            didntFadeSince = Time.realtimeSinceStartup;
        }
        else
        {
            remanenceGroupPlane.material.SetFloat("_FrameFadeAmount", 0);
        }
    }

And the shader calculates the alpha as follow:

max(_CurrentTexture_var.a, saturate(_TrailTexture_var.a - _FrameFadeAmount));

…and yet it doesn’t work. The fade duration is always something like 0.2s, even if I set the fadeDuration to 10.

Any lead, any idea why that would do that? Are the Update calls always synchronized with the shader renders?

Edit: A small precision: only one of my cameras can see the plane using my “fading shader”, so it’s only rendered once per frame, right? (And thus the RenderTexture is faded only once per frame?)

8

(This thread can be closed, my initial question has been answered, and I’ll probably create a new thread later with a new specific title.)

9

You don’t need all those cameras. You just need Unity - Scripting API: Graphics.Blit for most of those steps.

10

Damn, I really have a lot to learn :smile: Thanks a lot, it will make my scene a lot clearer!

11

In case anyone comes here someday looking for the same thing, I have created a new thread here: Trail shader - Unity Engine - Unity Discussions