Convert texture to grayscale GC is 60Mb

[Tarrag]

I convert an image to grayscale with the code below and my GC per frame baloons to 60MB. I have several images so this becomes a killer. I’m getting these results on Unity 2019.2.2f1 and 2019.3 on a fresh standalone built-in render pipeline project.

Texture2D graph = thisTexture;

        //convert texture
        Texture 2D grayImg = new Texture2D(graph.width, graph.height, graph.format, false);
        Graphics.CopyTexture(graph, grayImg);
        Color32[] pixels = grayImg.GetPixels32();
        for (int x = 0; x < grayImg.width; x++)
        {
            for (int y = 0; y < grayImg.height; y++)
            {
                Color32 pixel = pixels[x + y * grayImg.width];
                int p = ((256 * 256 + pixel.r) * 256 + pixel.b) * 256 + pixel.g;
                int b = p % 256;
                p = Mathf.FloorToInt(p / 256);
                int g = p % 256;
                p = Mathf.FloorToInt(p / 256);
                int r = p % 256;
                float l = (0.2126f * r / 255f) + 0.7152f * (g / 255f) + 0.0722f * (b / 255f);
                Color c = new Color(l, l, l, 1);
                grayImg.SetPixel(x, y, c);
            }
        }
        grayImg.Apply(false);

        return grayImg;

I can’t quite understand what’s going on from profiler, can someone please shed some light what’s going on?

Image for this GC result is attached sampleImg.jpg
Thanks a bunch for your help

[LennartJohansen]

Getting and setting data from a texture like this allocates temporary memory that is then released when done and shows up as GC.

Have a look at doing this with a job modifying the internal data you can get in a nativeArray.

Unity - Scripting API: Texture2D.GetRawTextureData

This will only work with uncompressed textures.

[richardkettlewell]

Or use a single SetPixels call instead of 1 per pixel.

[Tarrag]

That’s really helpful @LennartJohansen @richardkettlewell . It seemed a way to use a single SetPixels could come from writing back to Color32 data obtained from GetRawTextureData.

But I can’t see any image after grayscale conversion, can you please give me some further guidance please:

//thisTexture is incoming ARGB32 texture obtained from LoadImage and renders correctly.

graph = thisTexture;
grayImg = new Texture2D(graph.width, graph.height, graph.format, false);
var data = grayImg.GetRawTextureData<Color32>();

int index = 0;
Color32 pixel;

for (int x = 0; x < grayImg.width; x++)
        {
            for (int y = 0; y < grayImg.height; y++)
            {
                pixel = data[index];
                int p = ((256 * 256 + pixel.r) * 256 + pixel.b) * 256 + pixel.g;
                int b = p % 256;
                p = Mathf.FloorToInt(p / 256);
                int g = p % 256;
                p = Mathf.FloorToInt(p / 256);
                int r = p % 256;

               
[INDENT][INDENT]byte l = (byte) ((0.2126f * r / 255f) + 0.7152f * (g / 255f) + 0.0722f * (b / 255f));[/INDENT][/INDENT]

                Color32 c = new Color32(l, l, l, 1);
                data[index++] = c;
            }
        }
        // upload to the GPU
        grayImg.Apply();

[richardkettlewell]

I’m not sure about your maths:

(0.2126f * r / 255f)

let’s say r = 200, for a pixel.

that’s (0.2126f * 200 / 255f)
which equals 0.16674f

which you then cast to a byte. which will be zero.

i dont think you want the “/ 255” bits.

tbh a lot of the maths preceding this looks strange to me too i’m not sure why you aren’t simply using pixel.r (etc) in the grayscale conversion, instead of all that multiplication and modulo stuff.

[mgear]

missing loadrawtexturedata before apply?

Unity - Scripting API: Texture2D.LoadRawTextureData

[richardkettlewell]

@mgear

missing loadrawtexturedata before apply?
https://docs.unity3d.com/ScriptReference/Texture2D.LoadRawTextureData.html

From that page:

For runtime texture generation, it is also possible to directly write into texture data via GetRawTextureData that returns a Unity.Collections.NativeArray. This can be faster since it avoids a memory copy that LoadRawTextureData would do.

[Tarrag]

Thanks a bunch @richardkettlewell , yes removing /255 I can now see an image - but it’s yellow

If I make no changes with the below code to show the image in the original color I get a yellow image. Can this be because LoadImage is returning an ARGB32 texture2d yet Color32 works on RGBA32?

//thisTexture is incoming ARGB32 texture obtained from LoadImage and renders correctly.
        
        graph = thisTexture;
        grayImg = new Texture2D(graph.width, graph.height, graph.format, false);
        var data = grayImg.GetRawTextureData<Color32>();
        
        int index = 0;
        Color32 pixel;
        
        for (int x = 0; x < grayImg.width; x++)
                {
                    for (int y = 0; y < grayImg.height; y++)
                    {
                        pixel = data[index];
                        Color32 c = new Color32(pixel.r, pixel.g, pixel.b, 1);
                        data[index++] = c;
                    }
                }
                // upload to the GPU
                grayImg.Apply();

All the multiplication is to get a grayscale texture, this worked with setpixel one by one, otherwise I can’t think how to get to a grayscale

[richardkettlewell]

Sorry i was unclear about what exact bit i think doesnt make sense:

                int p = ((256 * 256 + pixel.r) * 256 + pixel.b) * 256 + pixel.g;
                int b = p % 256;
                p = Mathf.FloorToInt(p / 256);
                int g = p % 256;
                p = Mathf.FloorToInt(p / 256);
                int r = p % 256;

I would get rid of all that and just do:

byte l = (byte) ((0.2126f * pixel.r) + (0.7152f * pixel.g) + (0.0722f * pixel.b));

but maybe i’m misunderstanding what it’s there for.

@Tarrag

Can this be because LoadImage is returning an ARGB32 texture2d yet Color32 works on RGBA32?

Yes perhaps - getting the Raw pixel data doesn’t perform any conversion to Color32 for you, it simply gives you the memory and it’s your responsibility to know how it is packed/organised. So it depends on your texture format.

GetPixels, on the other hand, converts to Color32 for you, which is slower, but easier to work with.
If you use GetPixels, remember to also add SetPixels afterwards, before Apply.

[Tarrag]

I see what you mean about the formulas @richardkettlewell - I’m trying to make the code really simple for me cos I’ve been looking at the same lines for hours

It’s weird because png with LoadImage on Unity 2019.2 should be returning RGBA32 (I’m on Unity 2019.2.2f1) but for me it’s returning ARGB32 which is what LoadImage legacy Unity 5.3 would have returned - is this maybe a bug or documentation error?

Return RGBA32 would make sense cos I guess it’d be in sync with Color32 and GetRawTextureData. Otherwise how can I do the equivalent of LoadImage to get RGB32 format so I can use more efficient Color32/GetRawTextureData?

Thanks again for your help!

[Tarrag]

Was going to settle in using SetPixels32 to create a grayscale and forget about the better GetRawTextureData.

But isolating the GC contribution to convert a grayscale only using the first script I posted in this thread (using SetPixel for each pixel) is 12MB (grayscale renders correctly)

Using SetPixels32 with the code below once is 24MB (grayscale renders correctly)

Anyone knows how I can improve performance please?

Texture2D graph;
Texture2D grayImg;
graph = thisTexture;//thisTexture is ARGB32 renders correctly

//convert texture
        grayImg = new Texture2D(graph.width, graph.height, graph.format, false);
        Graphics.CopyTexture(graph, grayImg);
        Color32[] pixels = grayImg.GetPixels32();
        Color32[] changedPixels = new Color32[grayImg.width*grayImg.height];
       
        for (int x = 0; x < grayImg.width; x++)
        {
            for (int y = 0; y < grayImg.height; y++)
            {
                Color32 pixel = pixels[x + y * grayImg.width];
                int p = ((256 * 256 + pixel.r) * 256 + pixel.b) * 256 + pixel.g;
                int b = p % 256;
                p = Mathf.FloorToInt(p / 256);
                int g = p % 256;
                p = Mathf.FloorToInt(p / 256);
                int r = p % 256;
                float l = (0.2126f * r / 255f) + 0.7152f * (g / 255f) + 0.0722f * (b / 255f);
                Color c = new Color(l, l, l, 1);
                changedPixels[x + y * grayImg.width] = c;
            }
        }
        grayImg.SetPixels32(changedPixels);
        grayImg.Apply(false);

[bgolus]

I think you want to use GetRawTextureData<byte>() instead of Color32. Iterate over the array in steps of 4. The color the 4 bytes refer to depends on the format, so ARGB32 will be that order.

NativeArray<byte> bytes = tex.GetRawTextureData<byte>();
for (int i=0; i<bytes.Length; i+=4)
{
    byte gray = (byte)(0.2126f * bytes[i+1] + 0.7152f * bytes[i+2] + 0.0722f * bytes[i+3]);
    bytes[i+3] = bytes[i+2] = bytes[i+1] = gray;
}
tex.Apply();

[Tarrag]

That works great @bgolus I also tested and see if it’s RGB24 it’s be in steps of 3, so cool !

Using this concept I tried to stitch into a new texture the original rgb on top of the gray image. But I’m getting two gray images (aligned correctly one on top of the other).

I also checked whether the rgb image was still in colour after creating the grayscale (testing if it had been overwritten) but it still has its original colors and both gray/rgb are ARGB32.

Could you please give me some guidance what may be wrong with this code please? Thanks a bunch for your help

Texture2D stitchedVerticalTexture;
Texture2D grayLayer;
Texture2D rgbLayer;
//grayTex2d is created grayscale image. rgbTex2d is original color image tested to be still color at this stage

grayLayer = new Texture2D(grayTex2d.width, grayTex2d.height, grayTex2d.format, false);
Graphics.CopyTexture(grayTex2d, grayLayer);
rgbLayer = new Texture2D(rgbTex2d.width, rgbTex2d.height, rgbTex2d.format, false);
Graphics.CopyTexture(rgbTex2d, rgbLayer);

stitchedVerticalTexture = new Texture2D(grayLayer.width, grayLayer.height * 2, TextureFormat.ARGB32, false);

NativeArray<byte> bytesGrayLayer = grayLayer.GetRawTextureData<byte>();
NativeArray<byte> bytesRGBLayer = rgbLayer.GetRawTextureData<byte>();
NativeArray<byte> bytesDestinationTexture = stitchedVerticalTexture.GetRawTextureData<byte>();

byte srcGray = 0;
byte srcRGB = 0;
for (int i = 0; i < bytesGrayLayer.Length; i += 4)
    {
            srcGray = (byte)(bytesGrayLayer[i + 1] + bytesGrayLayer[i + 2] + bytesGrayLayer[i + 3]);
            srcRGB = (byte)(bytesRGBLayer[i + 1] + bytesRGBLayer[i + 2] + bytesRGBLayer[i + 3]);

            bytesDestinationTexture[i + 3] = bytesDestinationTexture[i + 2] = bytesDestinationTexture[i + 1] = srcGray;
            bytesDestinationTexture[bytesGrayLayer.Length + i + 3] = bytesDestinationTexture[bytesGrayLayer.Length + i + 2] = bytesDestinationTexture[bytesGrayLayer.Length + i + 1] = srcRGB;
    }

stitchedVerticalTexture.Apply();

[bgolus]

What are the copy texture calls for? If you already have the texture in CPU memory, there is no need to make another copy of it before getting the raw bytes.

Why are you adding the individual channels’ bytes together into a new byte? You’re taking the red, green, and blue values, adding them together, and setting that as the value for all 3 channels. The original example code I’m doing that because it’s converting a color image into greyscale (also scaling each channel before adding together). You don’t want to do that if you want to keep the original color values, nor do you need to do it if you’re copying from an already greyscale image. Just copy the bytes exactly as they are.

Texture2D stitchedVerticalTexture;
//grayTex2d is created grayscale image. rgbTex2d is original color image tested to be still color at this stage
stitchedVerticalTexture = new Texture2D(grayTex2d.width, grayTex2d.height * 2, TextureFormat.ARGB32, false);
NativeArray<byte> bytesGray = grayTex2d.GetRawTextureData<byte>();
NativeArray<byte> bytesRGB = rgbTex2d.GetRawTextureData<byte>();
NativeArray<byte> bytesDestinationTexture = stitchedVerticalTexture.GetRawTextureData<byte>();

int numGrayBytes = bytesGray.Length;
for (int i = 0; i < numGrayBytes; i++)
{
    bytesDestinationTexture[i] = bytesGray[i];
    bytesDestinationTexture[i + numGrayBytes] = bytesRGB[i];
}

stitchedVerticalTexture.Apply();

This assumes the grey and RGB textures are both ARGB32 just like the stitched texture. And if that’s true then you don’t even need the above code at all because all of the above can be done with:

Texture2D stitchedVerticalTexture = new Texture2D(grayTex2d.width, grayTex2d.height * 2, TextureFormat.ARGB32, false);
Graphics.CopyTexture(grayTex2D, 0, 0, 0, 0, grayTex2d.width, grayTex2d.height, stitchedVerticalTexture, 0, 0, 0, grayTex2d.height);
Graphics.CopyTexture(rgbTex2D, 0, 0, 0, 0, grayTex2d.width, grayTex2d.height, stitchedVerticalTexture, 0, 0, 0, 0);

You may not even need to call Apply() on the stitched texture if you’ve already done so on the grey and rgb textures.

[Tarrag]

Oh wow, thanks so much @bgolus your post was an eye opener for me!

Im wondering why the channels had to be added to create a gray channel, is there any reading or online course you would recommend to learn more about this please? It’s really fascinating!

thanks again for your invaluable help!!

[bgolus]

@Tarrag

Im wondering why the channels had to be added to create a gray channel, is there any reading or online course you would recommend to learn more about this please? It’s really fascinating!

How else do you convert RGB to greyscale? My code just implements the same approach your original example did, just without the unnecessary divide by 255.

[Tarrag]

I see @bgolus , I take snippets from the net, mix and test and ask for help til I get it working but I don’t really know what Im doing, its be great learning how it works, its really awesome, gpu vs cpu, performance, raw byte vs color32, but don’t know where to start

[bgolus]

Honestly I don’t have a good answer for you. Some topics have a lot of material out there to read up on, others don’t and just require you understand stuff.

There are a ton of tutorials for how to go about doing an RGB to greyscale conversion, of varying levels of quality. Most will eventually multiply each RGB component by some per component multiplier and add them together. The value each component is multiplied by is based on the relative perceived brightness each component has to each other, with the three multipliers adding up to 1. Where those numbers come from … well, there are several sources that have been calculated for different use cases. I believe the most common numbers come from the NTSC standard (specifically Rec. 709), and was used for converting color TV broadcasts into luminance so they would display properly on black & white TVs. That gets into early color TV broadcasting here in the US and YUV color space … which is a discussion for elsewhere.

A byte is 8 bits. A Color32 is just 4 bytes (RGBA components, one byte each, aka 8 bits each = 32 bits). Each byte is used to represent a numerical integer value between 0 and 255. The ARGB32 format is also just 4 bytes, but the order the components are defined in is different than Color32’s RGBA. That’s really it.

[Tarrag]

Thanks a bunch for your explanation @bgolus , it gives me a lot to chew on and leads to further research