Camera capture at deterministic intervals

Unity Engine
1

Hi. I have a camera rotating around a dynamic (moving) object for a full cycle, and then approaching a fixed amount to the object and keep rotation. After each 0.5s, I capture the camera view, assign it to a RenderTexture and write it to a PNG file. My goal is to run this multiple times with different texture width and heights and obtain sets of images in different resolution from different angles and distances.

The relevant part of my code look like this:

    private RenderTexture rt;
    private float nextActionTime = 0.0f; // start capturing at 0s
    public float period = 0.5f; // capture interval
    public int texWidth = 1920;
    public int texHeight = 1080;
    private void LateUpdate()
    {
        // Capture object periodically
        if (Time.time > nextActionTime)
        {
            nextActionTime += period;
            Capture();
        }
    }
    public void Capture()
    {
        Camera Cam = GetComponent<Camera>();
        // rt = new RenderTexture(1920, 1080, 24, RenderTextureFormat.ARGB32);
        rt = new RenderTexture(texWidth, texHeight, 24, RenderTextureFormat.ARGB32);
        rt.Create();
        Cam.targetTexture = rt;
        RenderTexture currentActiveRT = RenderTexture.active;
        RenderTexture.active = rt;
        Cam.Render();
        Texture2D tex = new Texture2D(Cam.targetTexture.width, Cam.targetTexture.height, TextureFormat.ARGB32, false);

        tex.ReadPixels(new Rect(0, 0, Cam.targetTexture.width, Cam.targetTexture.height), 0, 0);
        tex.Apply();
        RenderTexture.active = currentActiveRT;
        byte[] bytes = tex.EncodeToPNG();
        Destroy(tex);
        File.WriteAllBytes(capturePath + fileCounter + ".png", bytes);
        fileCounter++;
    }
}

For each “experiment”, I set texWidth and texHeight to new values and start the play mode. Another script attached to the camera rotates it and moves it closer to the object until a minimum distance is attached, and the above script captures the camera view at each 0.5s in the meantime.

My issue is, when I inspect the captured images after the experiments, I see that they are not exactly matching although, according to my setup, they are supposed to be taken at the same sampling times. It seems that the timing of the snapshots varies between each different run (i.e. restarting play mode). My best guess is that this might be caused by a slow GPU-to-CPU copy due to the usage of Texture2D.ReadPixels; however, I’m not entirely sure if this could be mitigated if I use AsyncGPUReadback to copy the data from GPU to CPU. I would quickly try it out but it seems that AsyncGPUReadback is not supported using OpenGL and might only work if a 3rd party plugin (such as this one) is used.

Another possibility is, such a deterministic sampling is not possible between different runs. Then I’d probably have to insert multiple cameras and save multiple textures of different sizes during one run. I’m not sure though how many textures I can write simultaneously to files using this approach.

2

How are you moving the camera around? If you’re computing it with anything related to Time.time or Time.deltaTime you won’t be able to replicate the behavior perfectly.

Instead you must drive the camera to the same angle and distance at each shot to get it to line up.

In other words, don’t orbit at X degrees per second and take shots every Y seconds.

Instead, identify the angles you want, move the camera to those positions and take the N number of shots you want.

3

Thanks, that’s exactly what I’m doing. I move the camera around using Time.deltaTime with a fixed speed. Then when I complete a full cycle, I move it towards the object by 1 unit. See the code below.

I will try the approach with the incremented angles but just curious, why are completion times of the frames different between different runs (as given by Time.deltaTime) ?

    private float nextMoveTime = 6.0f;
    public float movePeriod = 6.0f;
    void MoveCameraCloser()
    {  
        targetCenter = targetRenderer.bounds.center;
        transform.position = Vector3.MoveTowards(transform.position, targetCenter, 1.0f);
        transform.LookAt(targetCenter);
    }

    void LateUpdate()
    {  
        // Spin the camera around the target at 60 degrees/second
        transform.RotateAround(target.transform.position, Vector3.up, 60 * Time.deltaTime);
       
        // cover 360 deg for each distance, then move the camera forward
        if (Time.time > nextMoveTime) 
        {  
            MoveCameraCloser();
            nextMoveTime += movePeriod;
            distCamToObject = transform.InverseTransformPoint(targetCenter).magnitude;
        }

        if(distCamToObject < 0.5)   // don't get too close to the object
            UnityEditor.EditorApplication.isPlaying = false;
    }
4

It’s because that’s the point of Time.deltaTime: as your frame rate goes up and down over time, that lets you keep a constant speed of gameplay.

And by the very nature of general purpose multi-process multi-user computers running complex modern operating systems connected to the network, no two runs will ever be the same timing-wise.

If you want views from every 45 degrees, I would just do that explicitly:

for (int range = 10; range >= 5; range--)
{
   for (int angle = 0; angle < 360; angle += 45)
   {
      // somewhere out to the right and up
      Vector3 position = new Vector3( 1, 0.5f, 0).normalized * range;

      // spin it around the Y axis
      position = Quaternion.Euler( 0, angle,  0) * position;

      // TODO: use the position to position the camera
   }
}
5

Thanks. Now I’m using multiple cameras to record textures of different sizes synchronously and increment angles and distances in a coroutine. I attach script to all cameras: Main (render to scene) and Recorder cameras, and call my Capture function if the script is attached to a Recorder cameras. Capturing images works perfect without any sync problem because all views for different sizes are recorded at one run.

The only remaining issue is, the Main camera, which should only render to the display, does not rotate at all, although the script is attached to it too. If I disable the Capture for Recorder cameras (see below the if-clause in MoveCamera), it rotates without any problem. Am I using the yield statement somehow wrong such that it is blocking my Main camera?

    void Start()
    {  
        Camera cam = GetComponent<Camera>();
        targetRenderer = target.GetComponent<Renderer>();
        currentRange = transform.InverseTransformPoint(targetCenter).magnitude;
        StartCoroutine(MoveCamera(cam, currentRange, rangeStep, angleStep));
    }

    void Capture(Camera Cam)
    {
        rt = new RenderTexture(texWidth, texHeight, 24, RenderTextureFormat.ARGB32);
        rt.Create();
        Cam.targetTexture = rt;
        RenderTexture currentActiveRT = RenderTexture.active;
        RenderTexture.active = rt;
        Cam.Render();

        Texture2D tex = new Texture2D(Cam.targetTexture.width, Cam.targetTexture.height, TextureFormat.ARGB32, false);
        tex.ReadPixels(new Rect(0, 0, Cam.targetTexture.width, Cam.targetTexture.height), 0, 0);
        tex.Apply();
        RenderTexture.active = currentActiveRT;

        byte[] bytes = tex.EncodeToPNG();
        Destroy(tex);
        File.WriteAllBytes(capturePath + fileCounter + ".png", bytes);
        fileCounter++;
    }

    IEnumerator MoveCamera(Camera cam, float currentRange, float rangeStep, int angleStep)
    {
        do
        {  
            targetCenter = targetRenderer.bounds.center;
            transform.LookAt(targetCenter);
            transform.position = Vector3.MoveTowards(transform.position, targetCenter, rangeStep);

            for (int angle = 0; angle < 360; angle += angleStep)
            {               
                transform.RotateAround(target.transform.position, Vector3.up, angleStep);

                yield return new WaitForEndOfFrame();
                if(transform.parent.name == "RecorderCameras")
                  Capture(cam);
            }

            currentRange -= rangeStep;

        } while (currentRange > rangeMin);
    }
}