Use RenderTexture to sample pixels from camera?

Questions & Answers
1

I have a camera. How do I sample pixels from this camera? What call do I make to the camera to get a texture of what the camera is going to render?

2

Use OnRenderImage.

3

I did actually have to do this in one of my projects, where I used some off-screen rendering to detect mouse clicks on non-standard objects (cubic bezier curves to be precise). As far as I know, there is no way of directly accessing the data inside a RenderTexture, since the data is stored on the GPU. You have to transfer it to a Texture2D in order to actually sample the texture on the CPU (in scripts).

Here is a relevant code snippet from that project that does this (attached to a camera object):

public Camera offscreenCamera; // The camera that will do the offscreen rendering  

// Initialize everything properly in Start/Awake
private RenderTexture curveSelectTargetTexture; // The target texture
private Texture2D curveSelectTexture; // The texture we sample from
private bool shouldDoRendering; // Prevents the process from happening every frame

void OnPostRender() {
    if (shouldDoRendering) {
        // Make sure that the the target texture actually exists
        if (!curveSelectTargetTexture.IsCreated()) {
            curveSelectTargetTexture.Create();
        }
        
        // Set camera to render to our target texture
        offscreenCamera.targetTexture = curveSelectTargetTexture;
        
        /* Offscreen rendering was done here (removed from this code snippet) */
        
        // The following line transfers the data from the RenderTexture to the Texture2D
        curveSelectTexture.ReadPixels(new Rect(0, 0, Screen.width, Screen.height), 0, 0);

        Color colorOfPixelAtMouse = curveSelectTexture.GetPixel((int)Input.mousePosition.x, (int)Input.mousePosition.y);
        
        // Some other processing that is irrelevant
    }
}

Note that the pixels in the Texture2D will be anti-aliased. You can disable this by disabling anti-aliasing in the quality settings, but then your game would not have anti-aliasing. You can also disable anti-aliasing by making the camera use deferred rendering, though I have encountered a bug where anti-aliasing still occurs in orthographic view.

If you are rendering visuals, then this doesn’t really matter, but if you are rendering information (which is what I had to do), then you will need to either disable anti-aliasing or filter out garbage information caused by anti-aliasing.

You will probably be most interested in the line that contains GetPixel.