Camera.SetStereoProjectionMatrix not working as expected

I am using Split-Screen Stereo VR Mode and have run into a problem applying projection matrices. If I use a one-camera setup with its target set to “Both” eyes, I can only set the projection matrix by setting the camera.projectionMatrix property directly – camera.SetStereoProjectionMatrix(…) does not seem to work. This means the projection matrix for each eye is forced to be the same (whatever camera.projectionMatrix is set to), which works for some VR displays but not all.

If I switch to a two-camera setup with targets set to Left and Right eyes on separate cameras, again I can only set a different projection matrix for each eye if I set the camera.projectionMatrix property directly. The two-camera setup code works (looks correct in the HMD) for all of my target VR displays, but the one-camera setup only works for displays where the left and right eye projection matrices are the same.

Here is code that works for me, but I would like the One-Camera setup to be able to set a different projection matrix for each eye, which currently isn’t working with camera.SetStereoProjectionMatrix. I have commented out the code that is not working:

private void SetProjectionMatrix()
            {
                if(stereoRigSetup == StereoRigSetup.OneCameraBothEyes)
                {
                    Matrix4x4 matrix0 = RenderManager.GetEyeProjectionMatrix(0);
                    _camera0.projectionMatrix = matrix0;

                    //@todo I thought this should achieve the same result as the line above, but it doesn't work that way
                    // _camera0.SetStereoProjectionMatrix(Camera.StereoscopicEye.Left, matrix0);
                    //_camera0.SetStereoProjectionMatrix(Camera.StereoscopicEye.Right, matrix0);

                    //this doesn't work as expected, either
                    //Matrix4x4 matrix1 = RenderManager.GetEyeProjectionMatrix(1);
                    // _camera0.SetStereoProjectionMatrix(Camera.StereoscopicEye.Left, matrix0);
                    //_camera0.SetStereoProjectionMatrix(Camera.StereoscopicEye.Right, matrix1);
                }
                else
                {
                    Matrix4x4 matrix0 = RenderManager.GetEyeProjectionMatrix(0);
                    Matrix4x4 matrix1 = RenderManager.GetEyeProjectionMatrix(1);
                    _camera0.projectionMatrix = matrix0;
                    _camera1.projectionMatrix = matrix1;

                    //this doesnt work either
                    // _camera0.SetStereoProjectionMatrix(Camera.StereoscopicEye.Left, matrix0);
                    //_camera1.SetStereoProjectionMatrix(Camera.StereoscopicEye.Right, matrix1);
                }
            }

Is this a bug with the SetStereoProjectionMatrix() call or am I using it incorrectly?

edit: using Unity 2017.1.0f3

Thanks for your help!

Late answer, I guess you won’t need the information anymore, but maybe it is helpful for someone else.
I don’t know if this is a bug, in any way it is weakly documented, but I am glad that Unity has this feature.

I use the non head-mounted stereo mode, but I am sure it is adoptable for the split-screen stereo mode.
To set the projection matrix, I basically do the following:

private void SetProjectionMatrix(Camera cam, Matrix4x4 projectionMatrix, Camera.StereoscopicEye? specifyBothEyeTarget = null)
{
#if UNITY_EDITOR // Editor only
    cam.projectionMatrix = projectionMatrix;
#elif UNITY_STANDALONE // Build only
    string loadedDeviceName = null;
#if UNITY_5 || UNITY_2017_1
    if(UnityEngine.VR.VRSettings.enabled == true)
    {
        loadedDeviceName = UnityEngine.VR.VRSettings.loadedDeviceName;
    }
#else // #elif UNITY_2017_2_OR_NEWER
    if (UnityEngine.XR.XRSettings.enabled)
    {
        loadedDeviceName = UnityEngine.XR.XRSettings.loadedDeviceName;
    }
#endif // End of version difference.
    if (loadedDeviceName.Equals("stereo")) // -vrmode stereo
    {
        if (cam.stereoTargetEye == StereoTargetEyeMask.Left)
        {
            cam.SetStereoProjectionMatrix(Camera.StereoscopicEye.Left, projectionMatrix);
        }
        else if (cam.stereoTargetEye == StereoTargetEyeMask.Right)
        {
            cam.SetStereoProjectionMatrix(Camera.StereoscopicEye.Right, projectionMatrix);
        }
        else // None or both:
        {
            if (specifyBothEyeTarget != null)
            {
                // Setting left or right eye for "both-eye" camera, depending on the specifyBothEyeTarget.Value
                cam.SetStereoProjectionMatrix(specifyBothEyeTarget.Value, projectionMatrix);
            }
        }
    }
    else if(string.IsNullOrEmpty(loadedDeviceName)) // -vrmode none or vr completely disabled
    {
        cam.projectionMatrix = projectionMatrix;
    }
#endif // End of UNITY_STANDALONE only
}

You can call this method dependently if you are using a single both-eye camera or two separate cameras. When calling it, for a both-eye cam, you need to call it twice (once for the left and once for the right side) and set the 3rd parameter (StereoscopicEye). If you call it for a left or right eye camera, just call it once and leave the 3rd parameter blank, the camera knows its target eye.

This also answers your other question: Yes, you need to set different projection matrices when using a camera with target eye both or when using Single Pass rendering. (Using separate cameras means Multi Pass rendering implicitly.)

An additional note: If you work / develop on a machine with a video card that does not support 120Hz stereo (e.g. Nvidia GTX, RTX, etc.), you can still use the “-vrmode stereo” command line argument. This will of course not really activate stereo rendering, but in this case, the projection matrix can be set with cam.SetStereoProjectionMatrix(), whereas, if the application is started with “-vrmode none”, the projection matrix must be set with cam.projectionMatrix. The above written code takes care of these things.

All this is still true for Unity 2018.3.6 (latest version at the time of writing).

Additionally, you can quickly try out this demo script:

using UnityEngine;

[RequireComponent(typeof(Camera))]
public class ProjectionMatrixTest : MonoBehaviour
{
    private Camera _cam;

    private Matrix4x4 _projMat;
    private Matrix4x4 _projMat2;

    private void Awake()
    {
        _cam = GetComponent<Camera>();
    }

    private void Start()
    {
#if UNITY_EDITOR
        _projMat = _cam.projectionMatrix;
#else
        if (_cam.stereoTargetEye == StereoTargetEyeMask.Left)
        {
            _projMat = _cam.GetStereoProjectionMatrix(Camera.StereoscopicEye.Left);
        }
        else if(_cam.stereoTargetEye == StereoTargetEyeMask.Right)
        {
            _projMat = _cam.GetStereoProjectionMatrix(Camera.StereoscopicEye.Right);
        }
        else // Both or None
        {
            //_projMat = _cam.projectionMatrix;

            _projMat = _cam.GetStereoProjectionMatrix(Camera.StereoscopicEye.Left);
            _projMat2 = _cam.GetStereoProjectionMatrix(Camera.StereoscopicEye.Right);
        }

#endif
    }

    private void Update()
    {
        Debug.Log("Updating: " + _cam.stereoTargetEye.ToString());

        _projMat.m01 += Mathf.Sin(Time.time * 1.2F) * 0.1F;
        _projMat.m10 += Mathf.Sin(Time.time * 1.5F) * 0.1F;
        _projMat.m21 += Mathf.Sin(Time.time * 0.8F) * 0.1F;

        _projMat2.m01 += Mathf.Sin(Time.time * 1.2F) * 0.1F;
        _projMat2.m10 += Mathf.Sin(Time.time * 1.5F) * 0.1F;
        _projMat2.m21 += Mathf.Sin(Time.time * 0.8F) * 0.1F;
        

#if UNITY_EDITOR
        _cam.projectionMatrix = _projMat;
#else
        if (_cam.stereoTargetEye == StereoTargetEyeMask.Left)
        {
            _cam.SetStereoProjectionMatrix(Camera.StereoscopicEye.Left, _projMat);
        }
        else if(_cam.stereoTargetEye == StereoTargetEyeMask.Right)
        {
            _cam.SetStereoProjectionMatrix(Camera.StereoscopicEye.Right, _projMat);
        }
        else // Both or None
        {
            //_cam.projectionMatrix = _projMat; // This does not work.
			
            _cam.SetStereoProjectionMatrix(Camera.StereoscopicEye.Left, _projMat);
            _cam.SetStereoProjectionMatrix(Camera.StereoscopicEye.Right, _projMat2);
        }

#endif
    }
}

Just add it to a camera (Single- or both-eye) and add a huge cube in front of the camera, so that you can see the effect better. The projection matrix gets messed up, but you can see, that the API works. If you add it to a both-eye camera and comment out one of the two projection matrix apply lines (SetStereoProjectionMatrix…) and create a build, you can see, that only one eye goes wild, but the other one stays normal.