Speeding up reading from a RenderTexture

I’m trying to read from a render texture every frame. In my scene, I have a camera that’s rendering to a RenderTexture. I need to determine the value of a pixel at a known location in that image.

I tried setting up some code to read a pixel out of the render texture into a 1x1 Texture2D, like so:

Texture2D texture = new Texture2D(1,1);
Rect rect = new Rect(Mathf.FloorToInt(pixel.x), Mathf.FloorToInt(pixel.y), 1, 1);
testTexture.ReadPixels(rect, 0, 0);
//testTexture.Apply();
Color32 pixelSample = testTexture.GetPixel(0,0);

This code, specifically ReadPixels(), is taking 3-5 ms each frame on the GPU. That’s actually quite a big deal for us, and I’m trying to figure out why it’s so large.

The code is copying from the GPU to RAM, so I get that there’s going to be a delay, but this seems large. I’m wondering if I’m blocking execution of Unity’s main thread or doing something else to gum up the works.

I’m calling this code from OnRenderImage - I’ve also tried Update() with no apparent change. Is there a specific point in the rendering I should be doing this, or perhaps a different way of going about it that’s faster?

Is there any other way to speed up the read of a single pixel from a RenderTexture? FYI, this has to work in the web player, so I can’t use texture pointers.

(P.S.: I don’t believe it’s possible to get information directly out of a shader, at least pre-DX11, but if I’m wrong that’s another avenue I could try)

When you render to the texture, then try to read it back immediately, you’re causing a pipeline stall. The GPU has to finish rendering all commands it has been given up to the “read pixels” call, then send that data back to the CPU. Pipeline stalls kill GPU performance.

Shawn Hargreave’s of XNA fame has a great explanation of what’s going on.

One way to deal with this is to render to the texture, then wait a frame or two before you attempt to copy the data back to the CPU. This gives the pipeline a chance to complete all of the commands normally without having to stall to fulfill your getdata request, at the cost of your needing to use slightly old data.

Another way to handle this is to figure out a way using shaders so you don’t ever have to pull the data out of GPU memory. Do you actually need to call GetPixels?

EDIT 2: From Unity 5.2+, you can no longer call glReadPixels from the main thread and expect the correct RenderTexture/FrameBuffer to be active. You have to move the call to a Low-level native plugin and bind the framebuffer manually in the plugin before reading the pixels back. For more information on this, see the newest comments to this answer.


EDIT: (Only works for Unity 5.1 and older) All of the below is still correct, but an easier way to implement it, is to just include the OpenGL library directly in C# (no need for building an extra C++ DLL).

[DllImport("opengl32")]
public static extern void glReadPixels (int x, int y, int width, int height, int format, int type, IntPtr buffer);

This might be a little late, but could be useful for someone else.

If the script is attached to the camera which renders to the render texture, you can put your ReadPixels() into

void OnPostRender () {}

Then the required RenderTexture is already active and you don’t need to spend time changing it.

Because Unity probably wants a copy of all textures in CPU memory, ReadPixels can be very slow - especially when dealing with large textures. To get around this, the only solution I could find was to run Unity in OpenGL-mode (add “-force-opengl” to the target path of Unity’s shortcut) and then write a plugin (DLL) which calls the OpenGL-function glReadPixels. With that function it is possible to transfer directly from the currently active frame buffer (RenderTexture) to CPU memory. With this solution it reduces the amount of time it takes to fetch a 1920x1080 RenderTexture to the CPU memory from 54ms to 12ms.

The C/C++ DLL source (place glext.h in the source folder):

#include "stdafx.h"

#include <gl\GL.h>
#include <gl\GLU.h>
#include <stdlib.h>
#include "glext.h"
#pragma comment(lib, "opengl32.lib")

using namespace std;

extern "C" __declspec(dllexport) int GetPixels(void* buffer, int x, int y, int width, int height) {
	if (glGetError())
		return -1;

	glReadPixels(x, y, width, height, GL_BGRA, GL_UNSIGNED_INT_8_8_8_8_REV, buffer);

	if (glGetError())
		return -2;
	return 0;
}

Add the function to Unity (C#) like this (place the DLL in Assets\Plugins\):

[DllImport ("NameOfYourDLL")]
	private static extern int GetPixels(IntPtr buffer, int x, int y, int width, int height);

And use it like any other function. The IntPtr buffer is a pointer to where you want to store your image. I get the pointer from another DLL - you could also get it by

unsafe{
	fixed(SomeClass* ptr = &someClass)
	{
		IntPtr buffer = (IntPtr)ptr;
		GetPixels (buffer, ...);
	}
}

However, that requires you to allow unsafe code. Alternativly you could probably use some Marshal-function to get a pointer to a byte array.

I am using this to fetch two full-HD renders each frame at ~24 fps and then keying those onto a 3D-SDI video stream using two Decklink hardware-keyers. The bottleneck is the fetching of data from the GPU as it stalls the rendering pipeline. If anyone finds a faster (real-time) solution for this, please let us know =)

@martijn-vegas hello, have u managed to solve the problem? I only get a picture entirely in black with your own code and I’ve checked and everything. I really need your help, Could u help me? Thanks in advance!