Taking a screenshot with depth information

Unity Engine
1

Good day everyone,
this is my first post in the Unity Community forums and I hope it is respecting the rules I have absorbed so far.
I am open for all kinds of constructive advice. Since I am a beginner in Unity I do not know if this post
is in the correct section, please move it, if it is not.
Now, my endeavor is to take a screenshot which contains the depth information of a camera’s view.

Open questions:
Do I need a shader for that?
(My knowledge about shaders is very limited.)

According to the forum entry

I need a shader.
But as far as I understood the depth is a byproduct that can be accessed when the frame is rendered by a camera so why do I need a shader?

Do I get something fundamentally wrong?
(Especially with regards to my code.)

Why is the screenshot I take “blank”?
(See provided picture material.)

The following code snippet shows what I coded so far:

using System.Collections;
using System.IO;
using UnityEngine;

public class DepthTextureScreenShot : MonoBehaviour
{

    public Camera camera;

    public int captureWidth = 1920;
    public int captureHeight = 1080;
    public enum Format { RAW, JPG, PNG, PPM };
    public Format format = Format.PNG;
    public string folder;

    private Rect rect;
    private RenderTexture renderTexture;
    private RenderTexture depthTexture;
    private Texture2D depthScreenShot;

    private int counter = 0; // image #

    void LateUpdate()
    {

        if (Input.GetKeyDown("o"))
        {
           
            rect = new Rect(0, 0, captureWidth, captureHeight);
            depthScreenShot = new Texture2D(captureWidth, captureHeight, TextureFormat.ARGB32, false);
            renderTexture = new RenderTexture(captureWidth, captureHeight, 0);
            depthTexture = new RenderTexture(captureWidth, captureHeight, 24, RenderTextureFormat.Depth);
           
            RenderTexture.active = depthTexture;
            Debug.Log(depthTexture.IsCreated());
            camera.depthTextureMode = DepthTextureMode.Depth;
            camera.SetTargetBuffers(depthTexture.colorBuffer, depthTexture.depthBuffer);
            camera.targetTexture = depthTexture;
            camera.Render();

            depthScreenShot.ReadPixels(rect, 0, 0);
            depthScreenShot.Apply();
       
            string fileName = UniqueFilename((int)rect.width, (int)rect.height, "l");

            WriteScreenShotToFile(depthScreenShot, fileName);

            //cleaning up
            camera.targetTexture = null;
            RenderTexture.active = null;
            Destroy(depthScreenShot);
        }
    }


    void WriteScreenShotToFile(Texture2D screenShot, string filename)
    {
        // pull in our file header/data bytes for the specified image format (has to be done from main thread)
        byte[] fileHeader = null;
        byte[] fileData = null;
        if (format == Format.RAW)
        {
            fileData = screenShot.GetRawTextureData();
        }
        else if (format == Format.PNG)
        {
            fileData = screenShot.EncodeToPNG();
        }
        else if (format == Format.JPG)
        {
            fileData = screenShot.EncodeToJPG();
        }
        else // ppm
        {
            // create a file header for ppm formatted file
            string headerStr = string.Format("P6\n{0} {1}\n255\n", rect.width, rect.height);
            fileHeader = System.Text.Encoding.ASCII.GetBytes(headerStr);
            fileData = screenShot.GetRawTextureData();
        }
        new System.Threading.Thread(() =>
        {
            // create file and write optional header with image bytes
            var f = System.IO.File.Create(filename);
            if (fileHeader != null) f.Write(fileHeader, 0, fileHeader.Length);
            f.Write(fileData, 0, fileData.Length);
            f.Close();
            Debug.Log(string.Format("Wrote screenshot {0} of size {1}", filename, fileData.Length));
        }).Start();

    }

    private string UniqueFilename(int width, int height, string camPointOfView)
    {
        // if folder not specified by now use a good default
        if (folder == null || folder.Length == 0)
        {
            folder = Application.dataPath;
            if (Application.isEditor)
            {
                // put screenshots in folder above asset path so unity doesn't index the files
                var stringPath = folder + "/..";
                folder = Path.GetFullPath(stringPath);
            }
            folder += "/screenshots";

            // make sure directory exists
            System.IO.Directory.CreateDirectory(folder);

            // count number of files of specified format in folder
            string mask = string.Format("depth_{0}x{1}*.{2}", width, height, format.ToString().ToLower());
            counter = Directory.GetFiles(folder, mask, SearchOption.TopDirectoryOnly).Length;
        }

        // use width, height, and counter for unique file name
        var filename = string.Format("{0}/depth_{1}x{2}_{3}_{4}.{5}", folder, width, height, counter, camPointOfView, format.ToString().ToLower());

        // up counter for next call
        ++counter;

        // return unique filename
        return filename;
    }
}

Thank you for your time!

Kind regards
BL

3256361--250892--depth_1920x1080_0_l.png
3256361--250892--depth_1920x1080_0_l.png1920×1080 37.8 KB

1 Like
2

Check this out.
https://github.com/unity3d-jp/FrameCapturer

That’s setup to capture the gbuffers when using deferred rendering, but it might help you on the right path.

The first issue you’re running into is just rendering the camera and copying the contents of the render texture won’t get you what you want. Part of the problem is Unity doesn’t provide a way to grab the depth directly, so setting up a camera and rendering out will only ever get you the color information. The way people work around this is to render the depth value to the color with a replacement shader pass, but that’s not necessary here. Unity handles rendering the scene depth and creating a texture of it for you. The only thing you need to do is grab a copy of that depth texture. Setting the depthTextureMode on the camera tells Unity to generate that texture for you.

If you look at that frame capture tool I linked to above, you’ll see it uses command buffers. Command buffers let you run bits of code at different times in the rendering pipeline. In this case you’d want to enable depth rendering on the main camera, then use a command buffer with the event CameraEvent.AfterDepthTexture to run a shader that copies _CameraDepthTexture to your own render texture, then save that to a texture.

Next problem is the depth texture is not linear, and is a 32 bit float. The range is nicely a 0.0 to 1.0 range, but it goes from the camera’s near clip plane to the far clip plane and 0.5 is not half way between the near and far. Saving to a png or jpg is going to loose a lot of data since it only has 8 bits of precision. You likely want to save to at least an exr file, though that’s still only 16 bit. Saving to raw is an option, but may be a hassle to use. You may want to transform the depth from non-linear to linear, and rescale that linear depth to a range you care about. That would all need to be done in the shader you use the copy the depth with.

4 Likes
3

is there a known way to do that in the HDRP? I’m trying for a while now to capture anything else than a single frame through the toolbox, but I’d like to have a depth capture as well as a 360° capture (not together), but the hdrp doesnt seem to like any extra components requesting capture data from the camera.

4

For anyone in the future that lands here from a google search, you may want to know about this
https://docs.unity3d.com/Packages/com.unity.simulation.capture@0.0/manual/index.html