I’m writing a Unity plugin which uses a separate renderer. The idea is to combine both images based on depth. To get the images to align I take Unity’s View- and Projectionmatrices and put them into the external renderer. The problem is, that they have different coordinate systems with left- and right-handedness respectively. Unity has a left-handed, the other renderer a right-handed one. Initially I got it to work. I simply flipped the up-axis and rotated around one axis to make it fit. Then I switched to VR. Since that point I can’t get it to work. There is always some offset between the two renderers scenes.
I am using OpenXR with a Oculus Rift S and tried multiple Unity versions (2022 LTS, 6000 LTS, 2023)
To see the behaviour please watch the following video:
Sorry for the shakiness, I had to record it holding the VR glasses in my hand. The cube that is generally further up on the y-axis is the one rendered by Unity. As you can see, moving away from the cubes increases their distance on the y-axis. When very close to the origin, the cubes positions almost align. Rotating the VR-glasses around its local axis, seems to also rotate the cube rendered by the external renderer around the given axis relative to the origin.
Both renderers use the same cube, with the center at the origin (0/0/0)
I retrieve the Unity view matrices like this:
private Matrix4x4 GetStereoViewMatrix(Camera.StereoscopicEye _eye)
{
if (xrDisplaySubsystem != null)
{
Matrix4x4 viewMatrix = cam.GetStereoViewMatrix(_eye);
Matrix4x4 conversionMatrix = new Matrix4x4(
new Vector4(scaleX, 0.0f, 0.0f, 0.0f),
new Vector4(0.0f, scaleY, 0.0f, 0.0f),
new Vector4(0.0f, 0.0f, scaleZ, 0.0f),
new Vector4(0.0f, 0.0f, 0.0f, 1.0f)
);
Matrix4x4 conversionMatrixAfter = new Matrix4x4(
new Vector4(1.0f, 0.0f, 0.0f, 0.0f),
new Vector4(0.0f, -1.0f, 0.0f, 0.0f),
new Vector4(0.0f, 0.0f, 1.0f, 0.0f),
new Vector4(0.0f, 0.0f, 0.0f, 1.0f)
);
// Transpose because external renderer uses column-major
viewMatrix = viewMatrix.transpose;
// This matrix rotates the object around its center
viewMatrix = conversionMatrix * viewMatrix;
// Inverse because external renderer requires world to camera matrix
viewMatrix = viewMatrix.inverse;
// This matrix flips the y-axis
viewMatrix = conversionMatrixAfter * viewMatrix;
return viewMatrix;
}
return Matrix4x4.identity;
}
The projection matrices are simply retrieved and transposed to column-major:
Matrix4x4 leftEyeProj = cam.GetStereoProjectionMatrix(Camera.StereoscopicEye.Left);
Matrix4x4 rightEyeProj = cam.GetStereoProjectionMatrix(Camera.StereoscopicEye.Right);
float[] leftProj = new float[]
{
leftEyeProj.m00, leftEyeProj.m10, leftEyeProj.m20, leftEyeProj.m30, // Column 1
leftEyeProj.m01, leftEyeProj.m11, leftEyeProj.m21, leftEyeProj.m31, // Column 2
leftEyeProj.m02, leftEyeProj.m12, leftEyeProj.m22, leftEyeProj.m32, // Column 3
leftEyeProj.m03, leftEyeProj.m13, leftEyeProj.m23, leftEyeProj.m33 // Column 4
};
float[] rightProj = new float[]
{
rightEyeProj.m00, rightEyeProj.m10, rightEyeProj.m20, rightEyeProj.m30, // Column 1
rightEyeProj.m01, rightEyeProj.m11, rightEyeProj.m21, rightEyeProj.m31, // Column 2
rightEyeProj.m02, rightEyeProj.m12, rightEyeProj.m22, rightEyeProj.m32, // Column 3
rightEyeProj.m03, rightEyeProj.m13, rightEyeProj.m23, rightEyeProj.m33 // Column 4
};
Plugin.setUnityProjectionMatrix(leftProj, rightProj);
If anybody has an idea I’d be super thankful, this problem has been bothering me for over a month.