Virtual Production and Cinematography with Cinemachine: Suggested Additions

[Dirrogate]

Virtual Production with Unity's CineMachine

A one minute video, that barely scratches the surface of what can be achieved with induction of Cinemachine for real time filmmaking and Virtual Production (which is the rage in Hollywood right now due to lockdowns). The unity scene is being streamed to the Cellphone Vcam rig over wifi and driven by Unity WebRTC/Webstreaming package.

While I keep experimenting, here are a few areas that I think different teams from Unity can stitch together within a couple of weeks, given these already exist in ‘unity silos’

• • Expose Cinemachine Camera properties: Focal length etc - so they can be overridden if so desired by Cinematographers (using some kind of input such as midi/OSC, webrtc)
• • Cinemachine ability to trigger Unity Recorder to do a “take” - recording at least a camera path move and proxy rez mp4 video (quality configurable in recorder)
• • Cinemachine Extension Module: Manual Override using phone Gyro/ accelerometer: This could be implemented thereby giving an operator the ability to pan/tilt and add organic, human “camera shake”

Food for thought:

• - Cinemachine Webrtc / Webstreaming integration: The team at Cinemachine and Webrtc should work closer on creating a “Virtual Production” camera, driven by Arkit and Arcore. (UE4 already has one)

• Cinemachine Midi and OSC integration: Native OSC integration would allow for all parameters to be driven / overridden by hand for aspects of control while Cinemachine brain handles tracking and switching. In-fact, under OSC control, users can design their own Control surfaces using TouchOSC for Camera switching based on priority, Focal and ISO etc…

• - Cinemachine integration with FREE-D protocol: Cinemachine virtual camera drives for instance, the affordable (for broadcasters) Panasonic AW-UE150 PTZ camera.

Understanding Panasonic’s technological breakthrough for VR and AR applications

Trouble Shooting:
Free Look Cam and Exposure. I don’t know if this is a bug unique to me, but in using Unity 2019.3.9 with latest Cinemachine (2.5.0) and HDRP 7.3.1, I find that only on the Freelook camera I need to set ISO to some abnormally high number such as 25000, and open aperture wide, otherwise the scene is dark.

There is no Cinemachine postprocessing volume override on the Freelook camera and all other Cinemachine cameras respect the global Volume settings I have (ISO and Shutter are set to be responsive to the Exposure component in the Global volume)

This is something to check for, if anyone else can confirm?

Meanwhile, with Cinemachine, it’s like having AI driven camera people on call and is a huge asset to Virtual Production.

[marc_tanenbaum]

This is amazing feedback and super useful as we think about our roadmap. Thank you!

I’ll ask one of our engineers to comment on the “trouble shooting” questions.

[Dirrogate]

@marc_tanenbaum

I’ll ask one of our engineers to comment on the “trouble shooting” questions.

Thank you, Marc! Looking forward to interacting with the team soon.

[Gregoryl]

@Dirrogate Thanks for the excellent feedback. I’m having trouble reproducing your exposure issue. Can you put together a small project that demonstrates this and send it to me?

[Dirrogate]

@Gregoryl

@Dirrogate Thanks for the excellent feedback. I’m having trouble reproducing your exposure issue. Can you put together a small project that demonstrates this and send it to me?

@Gregoryl I’ll get back to this soon, since I’m thinking a Virtual Camera following a target and with Aim set to POV might better serve my purpose of creating a manual “Steadycam”

But I do want to escalate a request if possible:
Expose Cinemachine Camera properties: Focal length etc - so they can be overridden if so desired by Cinematographers (using some kind of input such as midi/OSC, webrtc)

Take a look at this project I’m working on as experiments to build a Virtual Production solution for myself in Unity.

https://www.youtube.com/watch?v=rGIQ5IDV4Ac

The camera switching is not done via timeline or script, but over OSC protocol driven by sliders and switches created on an android phone/tablet (via TouchOSC). The light flicker is also driven manually by a fader over OSC protocol.

I’m sorely missing functions like Focal length, Focus, aperture, Iso and shutter. These are not exposed by Cinemachine cameras (via reflection) as normal Unity cameras are.

I’d like Cinemachine to do it’s thing like following a target, dolly etc… while letting some settings have manual over-ride.

Kind Regards

[Gregoryl]

Assuming you’re using HDRP.
If you set this in your Camera:

then your Cinemachine vcams will have an expanded lens section:

you can write a script to drive all those fields.

Focus distance is trickier, due to the way the Unity physical camera is set up. If you add a CinemachineVolumeSettings extension to the vcam, you will have ability to pull focus by animating the Focus Offset field (offset is relative to the camera - you can change that by changing the Focus Tracking setting).

Set it up like this (make sure Resolution is set to Full or it will look bad):

[Dirrogate]

@Gregoryl

Assuming you’re using HDRP.
…you will have ability to pull focus by animating the Focus Offset field (offset is relative to the camera - you can change that by changing the Focus Tracking setting).
Set it up like this (make sure Resolution is set to Full or it will look bad):
…

Thank you for this tip @Gregoryl .
On the other aspects, yes I am using HDRP and the physical camera. My lament is that Cinemachines’ implementation of the physical camera has hidden/broken previously exposed parameters of the original Physical camera such as focal length.

For instance, these parameters were exposed (to non programmer cinematographers) by simply creating Unity Gui sliders/buttons and dragging functions such as Focal length etc as in the attached images.

It would be good to have this functionality restored.

For now, I tried this script - which works perfectly - for non cinemachine cameras

using UnityEngine;
using UnityEngine.Rendering.HighDefinition;

[RequireComponent(typeof(HDAdditionalCameraData))]
public class HDCameraAppatureMapping : MonoBehaviour
{
    [SerializeField] OscIn _oscIn = null;
    [SerializeField] string _appatureAddress = "/oscControl/aperture";

    void Awake()
    {
        HDAdditionalCameraData data = GetComponent<HDAdditionalCameraData>();
        _oscIn.MapFloat(_appatureAddress, (float value) => data.physicalParameters.aperture = value);
    }
}

But now when I add it to a basic (dummy) Cinemachine Vcam, it “adds” another camera component + the HDAdditionalCameraData component, resulting in a black output.

The code above was kindly given to me by the author of the excellent OSCsimpl asset from the asset store. I was/am successfully using this code (before incorporating Cinemachine) and OSC (a signalling protocol like midi) to drive Unity cameras for live Virtual Production.

I’m guessing the code needs to be modified to not load the HDAdditionalCameraData component now, to access Cinemachine physical camera properties.

Could you help please?
Kind Regards

[Olmi]

Hi @Dirrogate ,

I’m not sure if I understood your question completely, but here’s some pointers how to access the Cinemacine things:
You can access Cinemachine camera datas by just getting the component, like that CinemachineVirtualCamera:

CinemachineVirtualCamera virtualCamera = GetComponent<CinemachineVirtualCamera>();

You need to be also using the namespace where it’s at, so add the using statement to the beginning of your code:

using Cinemachine;

Then you access most of the parameters under m_Lens, i.e. if you do this, you’ll see the same values that are in your CinemachineVirtualCamera:

Debug.Log("Iso: " + virtualCamera.m_Lens.Iso);
Debug.Log("Aperture: " + virtualCamera.m_Lens.Aperture);

And to change the values, you just assign a new value, like this:

virtualCamera.m_Lens.Iso = 6400;

After this it should be quite straightforward to implement some system that changes the values runtime…

[Dirrogate]

Thanks so much @Olmi for setting me on the right path.
I’ve got it working thanks to your help.

using UnityEngine;
using UnityEngine.Rendering.HighDefinition;
using Cinemachine;

public class CMapertureChange : MonoBehaviour
{
    [SerializeField] OscIn _oscIn = null;
    [SerializeField] string _appatureAddress = "/oscControl/aperture";

    void Awake()
    {
        CinemachineVirtualCamera virtualCamera = GetComponent<CinemachineVirtualCamera>();
        _oscIn.MapFloat(_appatureAddress, (float value) => virtualCamera.m_Lens.Aperture = value);
    }
}

ApertureRealtimeOverOSC

Second half of the video shows the aperture values streaming into the scene (over OSC protocol) from a cellphone on the same wifi network

[Dirrogate]

@Gregoryl @Olmi would you know what the value for Focal length is? I can’t find it.
I’ve found m_Lens.Iso, m_Lens.ShutterSpeed, m_Lens_Aperture.
There’s one called field of view - which technically is only going to around 106 (and isn’t the same as focal length)

[Gregoryl]

Use m_Lens.FieldOfView. Focal length display is UI-only. The conversion formula is this:

      float FocalLengthToVerticalFOV(float focalLength)
        {
            if (focalLength < 0.001f)
                return 180f;
            return Mathf.Rad2Deg * 2.0f * Mathf.Atan(Lens.SensorSize.y * 0.5f / focalLength);
        }

and the inverse conversion:

        float VerticalFOVToFocalLength(float fov)
        {
            return Lens.SensorSize.y * 0.5f / Mathf.Tan(Mathf.Deg2Rad * fov * 0.5f);
        }

[Olmi]

Above can be found in LensSettingsPropertyDrawer.cs, I think? Although it’s not exact same code. (It uses epsilon constant instead of that small float value…) Anyway.

[Dirrogate]

Gaaah!
You guys assume I know coding!
Please could one of you let me know where that above code goes inside this? …

using UnityEngine;
using UnityEngine.Rendering.HighDefinition;
using Cinemachine;

public class CMapertureChange : MonoBehaviour
{
    [SerializeField] OscIn _oscIn = null;
    [SerializeField] string _appatureAddress = "/oscControl/aperture";
    [SerializeField] string _focallength = "/oscControl/focal";
    [SerializeField] string _shutter = "/oscControl/shutter";

    void Awake()
    {
        CinemachineVirtualCamera virtualCamera = GetComponent<CinemachineVirtualCamera>();
        _oscIn.MapFloat(_appatureAddress, (float value) => virtualCamera.m_Lens.Aperture = value);
        _oscIn.MapFloat(_shutter, (float value) => virtualCamera.m_Lens.ShutterSpeed = value);
        _oscIn.MapFloat(_focallength, (float value) => virtualCamera.m_Lens.FieldOfView = value);
    }
}

[Gregoryl]

Try this:

{
...
    _oscIn.MapFloat(_focallength, (float value) => virtualCamera.m_Lens.FieldOfView = FocalLengthToVerticalFOV(value, virtualCamera.m_Lens.SensorSize.y);
...
}

float FocalLengthToVerticalFOV(float focalLength, float sensorSize)
{
    if (focalLength < 0.001f)
        return 180f;
    return Mathf.Rad2Deg * 2.0f * Mathf.Atan(sensorSize * 0.5f / focalLength);
}

[Dirrogate]

Thankyou!!

[Dirrogate]

@Gregoryl

Assuming you’re using HDRP.
…
…If you add a CinemachineVolumeSettings extension to the vcam, you will have ability to pull focus by animating the Focus Offset field (offset is relative to the camera - you can change that by changing the Focus Tracking setting)…

@Gregoryl I’ve got one more important bit left. I’d like to do a rack focus via a remote slider.
Currently it works fine for regular Physical cameras with this script:

using UnityEngine;
using UnityEngine.Rendering;
using UnityEngine.Rendering.HighDefinition;
using Cinemachine;

public class CinemachineDofOsc : MonoBehaviour
{
    DepthOfField depthOfField;

    [SerializeField] OscIn _oscIn = null;
    [SerializeField] string _dofAddress = "/oscControl/dof";

    void Awake()
    {
        Volume volume = GetComponent<Volume>();
        DepthOfField tempDof;

        if (volume.profile.TryGet<DepthOfField>(out tempDof))
        {
            depthOfField = tempDof;
        }
        _oscIn.MapFloat(_dofAddress, (float value) => depthOfField.focusDistance.value = value);
    }
}

How would I go about setting this script to address the “Focus Offset field”. Could you help?

Also, in the image you posted above, above the offset field there is Focus tracking dropdown.
I don’t see that in the latest version of Cinemachine. 2.5.0

Kind Regards

[Dirrogate]

@Dirrogate

@Gregoryl
How would I go about setting this script to address the “Focus Offset field”. Could you help?

Also, in the image you posted above, above the offset field there is Focus tracking dropdown.
I don’t see that in the latest version of Cinemachine. 2.5.0

Kind Regards

bump^ please.

[Gregoryl]

1. Add a CinemachineVolumeSettings component to your vcam.
2. Create a profile for it with DepthOfFeied section, set it up as described above
3. Instead of GetComponent, do GetComponent(), and set its m_FocusOffset field.

FocusTracks target is a bool in the released version of CM. In CM 2.6, it’s a dropdown.

[Dirrogate]

I’ve just completed “RackFocus” with Cinemachine.
This allows for manual override via OSC or Midi and complements Cinemachines’ AI driven camera rather well.
Turns out it was GetComponent<Cinemachine.PostFX.CinemachineVolumeSettings>();

Here’s the code for anyone else that might need to control Cinemachine parameters manually.

using UnityEngine;
using UnityEngine.Rendering.HighDefinition;
using Cinemachine;


public class CMapertureChange : MonoBehaviour
{
    [SerializeField] OscIn _oscIn = null;
    [SerializeField] string _appatureAddress = "/oscControl/aperture";
    [SerializeField] string _focallength = "/oscControl/focal";
    [SerializeField] string _shutter = "/oscControl/shutter";
    [SerializeField] string _dofAddress = "/oscControl/dof";


    void Awake()
    {
        CinemachineVirtualCamera virtualCamera = GetComponent<CinemachineVirtualCamera>();
        Cinemachine.PostFX.CinemachineVolumeSettings rackfocus = GetComponent<Cinemachine.PostFX.CinemachineVolumeSettings>();

        _oscIn.MapFloat(_appatureAddress, (float value) => virtualCamera.m_Lens.Aperture = value);
        _oscIn.MapFloat(_shutter, (float value) => virtualCamera.m_Lens.ShutterSpeed = value);
        _oscIn.MapFloat(_focallength, (float value) => virtualCamera.m_Lens.FieldOfView = FocalLengthToVerticalFOV(value, virtualCamera.m_Lens.SensorSize.y));
        _oscIn.MapFloat(_dofAddress, (float value) => rackfocus.m_FocusOffset = value);
    }
    float FocalLengthToVerticalFOV(float _focalLength, float sensorSize)
    {
        if (_focalLength < 0.001f)
            return 180f;
        return Mathf.Rad2Deg * 2.0f * Mathf.Atan(sensorSize * 0.5f / _focalLength);
    }
}