Faster method for getting a pixel color.

Unity Engine
,
1

I have created a system that paints terrain in play mode. Think Splatoon but for Unity’s terrain. What I’ve been working on requires me to occasionally know what color the painted terrain is. The many methods I have seen to do this work well enough and I’ve found one that works if my texture size is 512x512 or smaller but I would like to go larger as it looks sleeker, however, the larger I go in pixel size, the more it affects the framerate of the camera. I should add that I’m completely self-taught (good old YouTube academy) and reading Unity documentation makes my brain overheat.

Here’s what I currently have to determine my pixel information:

public Color ReturnColor(RaycastHit _hit)
{
    RenderTexture _splatmap = _hit.collider.GetComponent<PaintableTerrain>()._splatmap;
    RenderTexture temp = RenderTexture.GetTemporary(_splatmap.width, _splatmap.height, 0, RenderTextureFormat.ARGBFloat, RenderTextureReadWrite.Linear);

    Graphics.Blit(_splatmap, temp);

    RenderTexture previous = RenderTexture.active;
    RenderTexture.active = temp;

    if (myTexture2D == null || myTexture2D.width != _splatmap.width || myTexture2D.height != _splatmap.height)
    {
        myTexture2D.Reinitialize(_splatmap.width, _splatmap.height, TextureFormat.ARGB32, false);
    }

    myTexture2D.ReadPixels(new Rect(0, 0, temp.width, temp.height), 0, 0);
    myTexture2D.Apply();

    RenderTexture.active = previous;

    Vector2 pixelPos = _hit.textureCoord;
    pixelPos.x *= _splatmap.width;
    pixelPos.y *= _splatmap.height;

    Color pixelColor = myTexture2D.GetPixel((int)pixelPos.x, (int)pixelPos.y);
    RenderTexture.ReleaseTemporary(temp);


    return pixelColor;
}
2

Does it need to query the array every frame? First step might be to only get that ReadPixels array if you know it will be different from last frame.

“Know it will be different” can be a boolean set whenever you actually do paint on the ground, assuming that is less-frequently than you query the ground.

Otherwise… maybe keep your own Color[,] array in conjunction and use that for querying??

But as always…

For all performance and optimization issues, ALWAYS start by using the Profiler window:

Window → Analysis → Profiler

DO NOT OPTIMIZE “JUST BECAUSE…” If you don’t have a problem, DO NOT OPTIMIZE!

If you DO have a problem, there is only ONE way to find out: measuring with the profiler.

Failure to use the profiler first means you’re just guessing, making a mess of your code for no good reason.

Not only that but performance on platform A will likely be completely different than platform B. Test on the platform(s) that you care about, and test to the extent that it is worth your effort, and no more.

Remember that you are gathering information at this stage. You cannot FIX until you FIND.

Remember that optimized code is ALWAYS harder to work with and more brittle, making subsequent feature development difficult or impossible, or incurring massive technical debt on future development.

Don’t forget about the Frame Debugger window either, available right near the Profiler in the menu system.

Notes on optimizing UnityEngine.UI setups:

At a minimum you want to clearly understand what performance issues you are having:

  • running too slowly?
  • loading too slowly?
  • using too much runtime memory?
  • final bundle too large?
  • too much network traffic?
  • something else?

If you are unable to engage the profiler, then your next solution is gross guessing changes, such as “reimport all textures as 32x32 tiny textures” or “replace some complex 3D objects with cubes/capsules” to try and figure out what is bogging you down.

Each experiment you do may give you intel about what is causing the performance issue that you identified. More importantly let you eliminate candidates for optimization. For instance if you swap out your biggest textures with 32x32 stamps and you STILL have a problem, you may be able to eliminate textures as an issue and move onto something else.

This sort of speculative optimization assumes you’re properly using source control so it takes one click to revert to the way your project was before if there is no improvement, while carefully making notes about what you have tried and more importantly what results it has had.

“Software does not run in a magic fairy aether powered by the fevered dreams of CS PhDs.” - Mike Acton

3

I’ll add a ton of screenshots to help explain what’s happening. The dip in the CPU usage is while ReturnColor is being run every frame. The black is what’s being painted. It is alive, moving, and growing, or at least that’s how it is supposed to feel. As it stands, I could have anywhere between 1-4 player characters needing to know if they are standing in the darkness. Maybe some enemies too if I can decrease the frame drop. Keeping a Color[,] array might work but I’m not entirely sure where to start when I want the darkness to have such soft edges.

9576460--1355563--upload_2024-1-11_17-13-53.png
9576460--1355563--upload_2024-1-11_17-13-53.png1048×1128 119 KB

9576460--1355566--upload_2024-1-11_17-15-32.jpg
9576460--1355566--upload_2024-1-11_17-15-32.jpg2015×1215 1.14 MB

9576460--1355572--upload_2024-1-11_17-17-13.jpg
9576460--1355572--upload_2024-1-11_17-17-13.jpg2279×1169 642 KB

9576460--1355575--upload_2024-1-11_17-27-58.png
9576460--1355575--upload_2024-1-11_17-27-58.png925×573 888 KB

The top is just to the left of the grey NPC. The middle is just under the Player. the bottom is the dark hazy area. There is more paint than just darkness but this is a good example. When the numbers reach certain thresholds I want different things to happen so knowing the exact color is very helpful.
9576460--1355581--upload_2024-1-11_17-31-31.png

4

Didn’t realize I could reply directly. Using the ReturnColor method always seems to cause a noticeable frame drop.

5

Right… I noted that it gets ALL the pixels every time it needs to give you ONE pixel.

That is why I wrote these two specific ideas:

6

The painting occurs more frequently than the query and it affects more than just one area at a time. It paints gradually in one area over a few seconds with mathematical falloff around the edges. Ideally, I’d like to know what color everyone is standing on at all times but I could settle for every 1/4 - 1/2 second. As for trying to make an array, I’m concerned about losing the falloff effect of the painting process. I want areas of not complete darkness, the falloff is where my darkness gets its creepy tendril look (more impressive when not screenshots). Standing in darker areas should affect the players more than lighter areas but I don’t know how to include that without massive calculations bogging up the system every paint stroke. With the array system, I would have to keep track of the main area of darkness being painted and then figure out what the falloff is in the surrounding areas and store that info, but when another paint stroke goes over the same area it would change all those values, and this would happen frequently. Is the method I use for getting the pixel color the best way to get the pixel color? I’m wondering if there is a way to reduce the amount of pixels it has to count in the ReadPixels process rather than checking the whole texture every time I need to Return Color. What I thought might work is separating the main texture we’re checking with ReturnColor into more manageable pieces as if I were to crop the edges of the texture to the area my player characters are in or separate the texture into quadrants and then check the pixel colors because I notice a difference in frame drops with lower a resolution render texture.

7

No. GetPixel() has to decompress (assuming the texture is compressed) and convert the pixel data which is where the bulk of the slowdown is coming from. I’ve never bothered with any of these methods beyond playing around with them so I don’t know how effective they are but you could see if any of these help.

Note that these were written with the assistance of ChatGPT (GPT-4). Both of these approaches assume that the texture is stored as ARGB32. If they don’t work correctly I can try to fix any bugs but I don’t have an easy way to verify that they work on my end.

GetPixelData() approach

public Color ReturnColor(RaycastHit _hit)
{
    RenderTexture _splatmap = _hit.collider.GetComponent<PaintableTerrain>()._splatmap;
    RenderTexture temp = RenderTexture.GetTemporary(_splatmap.width, _splatmap.height, 0, RenderTextureFormat.ARGBFloat, RenderTextureReadWrite.Linear);

    Graphics.Blit(_splatmap, temp);

    RenderTexture previous = RenderTexture.active;
    RenderTexture.active = temp;

    if (myTexture2D == null || myTexture2D.width != _splatmap.width || myTexture2D.height != _splatmap.height)
    {
        myTexture2D.Reinitialize(_splatmap.width, _splatmap.height, TextureFormat.ARGB32, false);
    }

    myTexture2D.ReadPixels(new Rect(0, 0, temp.width, temp.height), 0, 0);
    myTexture2D.Apply();

    RenderTexture.active = previous;

    Vector2 pixelPos = _hit.textureCoord;
    pixelPos.x *= _splatmap.width;
    pixelPos.y *= _splatmap.height;

    // Using GetPixelData to access the pixel data
    var pixelData = myTexture2D.GetPixelData<Color32>(0);
    Color32 pixelColor32 = pixelData[(int)pixelPos.y * _splatmap.width + (int)pixelPos.x];
    Color pixelColor = pixelColor32;

    RenderTexture.ReleaseTemporary(temp);

    return pixelColor;
}

GetRawPixelData() approach

public Color ReturnColor(RaycastHit _hit)
{
    RenderTexture _splatmap = _hit.collider.GetComponent<PaintableTerrain>()._splatmap;
    RenderTexture temp = RenderTexture.GetTemporary(_splatmap.width, _splatmap.height, 0, RenderTextureFormat.ARGBFloat, RenderTextureReadWrite.Linear);

    Graphics.Blit(_splatmap, temp);

    RenderTexture previous = RenderTexture.active;
    RenderTexture.active = temp;

    if (myTexture2D == null || myTexture2D.width != _splatmap.width || myTexture2D.height != _splatmap.height)
    {
        myTexture2D = new Texture2D(_splatmap.width, _splatmap.height, TextureFormat.ARGB32, false, true);
    }

    myTexture2D.ReadPixels(new Rect(0, 0, temp.width, temp.height), 0, 0);
    myTexture2D.Apply();

    RenderTexture.active = previous;

    Vector2 pixelUV = _hit.textureCoord;
    pixelUV.x *= myTexture2D.width;
    pixelUV.y *= myTexture2D.height;

    Color pixelColor = Color.clear;
    NativeArray<byte> pixelData = myTexture2D.GetRawTextureData<byte>();
 
    // Assuming the texture is ARGB32, each pixel is represented by 4 bytes
    int index = (int)(pixelUV.y * myTexture2D.width + pixelUV.x) * 4;
    if (index < pixelData.Length - 4)
    {
        pixelColor.r = pixelData[index + 1] / 255f;
        pixelColor.g = pixelData[index + 2] / 255f;
        pixelColor.b = pixelData[index + 3] / 255f;
        pixelColor.a = pixelData[index] / 255f;
    }

    RenderTexture.ReleaseTemporary(temp);

    return pixelColor;
}
8

Please forgive my confusion, you believe that GetPixel is what’s causing the frame drop? Not the ReadPixels? I had assumed (and perhaps incorrectly) that ReadPixels was the culprit because when I lowered the render texture resolution the frame drops improved.

9

The profiler can answer this question for you. Deep-profile. Explore the call-stacks. See where most of the overhead is. Don’t assume, measure.

1 Like
10

Both of them can be problematic but for different reasons. GetPixel() has to do conversion work while ReadPixels() has to wait on the GPU. A well written alternative is more than just a few lines of code.

If I weren’t just going to take the second approach provided by @Kurt-Dekker (ie storing a second copy of my data whenever I’m writing to the terrain textures) I’d rewrite the code to not perform excessive operations. For example modifying my code to grab the data from the GPU once per frame instead of every call to ReturnColor().

Speaking of getting data from the GPU the best way to do it is with the following method not ReadPixels() as it isn’t necessary to wait on the GPU to be finished with its scheduled tasks before you can access the data.

https://docs.unity3d.com/ScriptReference/Rendering.AsyncGPUReadback.RequestIntoNativeArray.html

11

I’m rather unfamiliar with the profiler. What you see in my screenshots is my only practical experience with it. It’s a little hard for me to know where to look first but I think it is here?
Before ReturnColor:

9578851--1356037--upload_2024-1-12_19-27-52.png
9578851--1356037--upload_2024-1-12_19-27-52.png2159×1112 193 KB

During ReturnColor:
9578851--1356031--upload_2024-1-12_19-27-29.png
9578851--1356031--upload_2024-1-12_19-27-29.png2161×1122 202 KB

PlayerLoop is now taking up 75.1%

I’m most definitely looking to understand why what I’m doing is incorrect. I do not want a quick fix to this. All of these comments are incredibly helpful to me in better grasping this concept. I think you must have edited your post while writing this because I was going to follow up what you said with an additional question but now I don’t see what I was going to ask about. Originally I think you asked if it was necessary to call ReadPixels every time I needed the pixel color but I’ve tried to only use GetPixel to no avail and I never understood why.

As for the link to AsyncGPUReadback, do you know of any good youtube tutorials on the subject, with Unity Documentation I get lost and overwhelmed with what I’m looking for in relation to my often VERY specific use case.

12

You need to dig into the overview heirarchy (the part at the bottom) and see what time is being spent on each individual method. Not just look at the top level methods.

You may need to enable deep profiling (one of the buttons at the top) to help with this.

13

No, but that’s because I don’t bother with tutorials for anything that is older than approximately April 2023. If that sounds very specific it’s because that’s the current approximate cutoff date for ChatGPT (it’s dependent on the info though as to how old it covers).

It does have its drawbacks like the free version (GPT-3.5) being far weaker than the paid version (GPT-4), and it will hallucinate (https://www.ibm.com/topics/ai-hallucinations) creating incorrect code that it’s never quite able to make work even though it’s confident that it will, but if you can live with these limits it’s far beyond most tutorials.

For example here’s the result of my asking for an example of AsyncGPUReadback:

GPT-4 example code

using UnityEngine;
using UnityEngine.Rendering;

public class AsyncGPUReadbackExample : MonoBehaviour
{
    void Start()
    {
        // Create a texture
        Texture2D texture = new Texture2D(256, 256, TextureFormat.RGBA32, false);

        // Request Async GPU Readback
        AsyncGPUReadback.Request(texture, 0, TextureFormat.RGBA32, OnCompleteReadback);
    }

    void OnCompleteReadback(AsyncGPUReadbackRequest request)
    {
        if (request.hasError)
        {
            Debug.Log("GPU readback error detected.");
            return;
        }

        // Process the data
        Texture2D texture = new Texture2D(256, 256, TextureFormat.RGBA32, false);
        texture.LoadRawTextureData(request.GetData<uint>());
        texture.Apply();

        // Use the texture as needed
    }
}

Additionally, unlike a tutorial that you’ve stumbled upon online, you can ask questions. For example I asked for an explanation of the above code and it gave this.

Explain the code

It’s ability to explain the code doesn’t necessarily stop there either. You can ask for a more in-depth explanation of specific sections of the code. Whether it will be able to comply is another matter but that goes back to the issue of hallucinations. You have to be careful that it’s explained correctly. It’s always confident even when it’s wrong.

If you haven’t already I recommend trying your code out in a build rather than in the editor. Here’s the docs entry on how to attach the profiler to a build.

https://docs.unity3d.com/Manual/profiler-profiling-applications.html

1 Like
14

Does this look right? I activated the deep profiler and this is what appears to be taking up a majority of that percent
9578914--1356103--upload_2024-1-12_20-48-43.png

I wasn’t aware Chat-GPT had become so versatile and useful for learning code! I appreciate the suggestion! I knew that the editor loop was a big reason as to why you would make a build version but is there any other reason I’m unaware of?

15

This just means your CPU is waiting on the GPU to finish rendering. Quick google search could’ve told you this.

So whatever you’re doing is trying to access the GPU while it’s unavailable, holding up Unity’s main thread. So you probably need to look at the async methods that were mentioned by Ryiah.

Also word of caution on Chat GPT: It tends to make up stuff a lot. Seen a lot of newbies here with incorrect ideas because Chat GPT made up nonsense for them.

1 Like
16

Nothing that immediately comes to mind aside from general editor jank.

Yes, like @spiney199 mentioned Semaphore.WaitForSignal is the CPU waiting on the GPU, but the part that wasn’t mentioned is its parent Gfx.ReadbackImage which is ReadPixels(). It’s forcing the semaphore because it’s blocking (ie pausing) the main thread until data has been read.

Kinda. It’s a little more complex because the semaphore is a side effect rather than the actual culprit.

1 Like
17

Okay, having spent several days trying to understand AsyncGPUReadback and the examples presented to me in this forum, I think I reached the solution! Here’s my code slightly modified to include AsyncGPUReadback and exclude readpixels:

public Color ReturnColor(RaycastHit _hit)
{
    RenderTexture _splatmap = _hit.collider.GetComponent<PaintableTerrain>()._splatmap;
    RenderTexture temp = RenderTexture.GetTemporary(_splatmap.width, _splatmap.height, 0, RenderTextureFormat.ARGBFloat, RenderTextureReadWrite.Linear);

    Graphics.Blit(_splatmap, temp);

    RenderTexture previous = RenderTexture.active;
    RenderTexture.active = temp;

    if (myTexture2D == null || myTexture2D.width != _splatmap.width || myTexture2D.height != _splatmap.height)
    {
        myTexture2D.Reinitialize(_splatmap.width, _splatmap.height, TextureFormat.ARGB32, false);
    }

    AsyncGPUReadback.Request(_splatmap, 0, TextureFormat.ARGB32, OnCompleteReadback);

    //myTexture2D.ReadPixels(new Rect(0, 0, temp.width, temp.height), 0, 0);

    //myTexture2D.Apply();

    RenderTexture.active = previous;

    Vector2 pixelPos = _hit.textureCoord;
    pixelPos.x *= _splatmap.width;
    pixelPos.y *= _splatmap.height;

    Color pixelColor = myTexture2D.GetPixel((int)pixelPos.x, (int)pixelPos.y);
    RenderTexture.ReleaseTemporary(temp);


    return pixelColor;
}

void OnCompleteReadback(AsyncGPUReadbackRequest request)
{
    if (request.hasError)
    {
        Debug.Log("GPU readback error detected.");
        return;
    }

    myTexture2D.LoadRawTextureData(request.GetData<uint>());
    myTexture2D.Apply();
}

The frame drops have disappeared or at least become unnoticeable to the eye (and profiler!) The only remaining frame stutter I have appears to be the editor loop and should go away in a build version according to what I have read online.

Here’s the profiler with ReturnColor intermixed

9590548--1358878--upload_2024-1-18_13-4-43.png
9590548--1358878--upload_2024-1-18_13-4-43.png1662×1105 129 KB

The dip seems to be the editor loop:
9590548--1358881--upload_2024-1-18_13-7-59.png
9590548--1358881--upload_2024-1-18_13-7-59.png1674×1107 134 KB

Is there anything I am still doing that isn’t necessary or is amateurish that I should change? Understanding why or how I am making mistakes is a big help to a self-taught programmer like me! If everything looks good I want to thank all of you for your help!