How to correct camera after modifying "view matrix"?

I have a 3D scene with some quads and a sprite

my camera is setup to look at the scene at a 45º angle like so

now I have the camera following the sprite around, and it all works as expected.

But I am not happy with the perspective, so I found that a nice correction was to alter the transform matrix or (view matrix) of the camera, fixing the tilt of the sprite by basically billboarding it (and avoiding geometry clipping) and skewing the geometry to create a pleasing effect

the problem is that after I alter the values of the matrix, the camera follow stops working correctly and the character starts going further than the bounds of the camera

when I alter the m11 component of the matrix, jumping on the Y axis eventually sends the character offscreen

when I alter the m12 component, walking north eventually sends the character offscreen

here is the code I am using for camera following, and for changing the view matrix of the camera

    // Public fields to assign the actual camera and dummy camera
    public Camera camz;
    public Camera dummyCam;

    // Target that the camera should follow
    public Transform target;
    public Vector3 positionOffset;

    // Matrix offsets exposed as public fields for modification in the Inspector
    public float offset00, offset01, offset02, offset03;
    public float offset10, offset11, offset12, offset13;
    public float offset20, offset21, offset22, offset23;
    public float offset30, offset31, offset32, offset33;


    void Update()
    {
        // Update the dummy camera's position to follow the target
        dummyCam.transform.position = target.position + positionOffset;

        // Get the current worldToCameraMatrix from the dummy camera
        Matrix4x4 currentMatrix = dummyCam.worldToCameraMatrix;

        // Apply offsets to the current matrix
        currentMatrix.m00 += offset00; currentMatrix.m01 += offset01; currentMatrix.m02 += offset02; currentMatrix.m03 += offset03;
        currentMatrix.m10 += offset10; currentMatrix.m11 += offset11; currentMatrix.m12 += offset12; currentMatrix.m13 += offset13;
        currentMatrix.m20 += offset20; currentMatrix.m21 += offset21; currentMatrix.m22 += offset22; currentMatrix.m23 += offset23;
        currentMatrix.m30 += offset30; currentMatrix.m31 += offset31; currentMatrix.m32 += offset32; currentMatrix.m33 += offset33;

        // Apply the modified matrix to the actual camera's worldToCameraMatrix
        camz.worldToCameraMatrix = currentMatrix;

    
    }

It seems the distortion I am creating on the camera is messing with the math that keeps the character in the center of the screen

I was wondering if someone knew if there was some formula I could use to compensate the offsets I am inserting and allow the camera to keep up with the character

sorry for the long post and thanks for your time

Use Matrix4x4.TRS (or SetTRS) instead of inventing your own numbers. By fudging the values in the first column you’re introducing inconsistent rotational values, and I believe these won’t even render correctly which can explain the discrepancy.

You can also check for validity by using ValidTRS.

2 Likes

thank you for the reply

is there any clue on how to use this to achieve the effect I am getting by changing the m11 value?

I dont know what parameters I am supposed to plug into SetTRS() to get this result

(because the effect doesnt feel like it translates to a legit camera transform?)

Hey perth, it feels like maybe you’re looking for an oblique camera frustum??

This would let you keep the camera facing horizontal but skew the view cone downwards.

Your eyes are actually using an oblique frustum: you can see many more degrees down than up from the center of your gaze line forward. With math you can make it as ridiculously downwards as you want.

Play with values… it’s kinda cool. Green dots are the “facing line” of the camera, going horizontal here.

9861642--1420791--oblique.png

using UnityEngine;
using System.Collections;

// @kurtdekker - 0, 0 is a "normal" camera, try negative and positive obliquities to see what it does.

public static class CameraObliqueFrustum
{
    public static void SetObliqueness (Camera cam, float horizObl, float vertObl)
    {
        cam.ResetAspect ();
        cam.ResetCullingMatrix ();
        cam.ResetProjectionMatrix ();
        Matrix4x4 mat = cam.projectionMatrix;
        mat [0, 2] = horizObl;
        mat [1, 2] = vertObl;
        cam.projectionMatrix = mat;
    }
}
1 Like

You need to provide a clear picture (with a checkered pattern) to show the desired distortion of your image.
The way you explained your problem suggests that you’ve fixed the perspective somehow, but doesn’t tell us how, and also it appears to have never really worked.

1 Like

thanks! It looks super interesting I will try this out

I thought of this as well, but wasn’t sure if that’s what he asked.
Using an oblique frustum in the manual has some more illustrations.

And here’s some more info.

1 Like

ideally I would like to have some freedom to have this be dynamic and easy to change like this, its just the camera follow that is not keeping up with the changing of the values

Have you tried setting the field of view to a lower value? Outside of potentially needing ‘obliqueness’ I mostly see this change as something field of view might correct. Though there is some slight stretching as well, which you can introduce by doing TSR.

I mean why don’t you simply expose TRS values and experiment with that? That way you get what you wanted – experimentation – without having to learn how to mathematically validate your matrices.

TRS is called like that because it assumes translation, rotation, and scaling. The latter two are applied onto the 3x3 matrix, which is the actual transformation matrix. The 4th row and column are added to include translation (you can basically extract their values as is from the 4th column).

So you can simply expose 3 vector3s

using UnityEngine;

[ExecuteAlways]
public class TRSExperiment : MonoBehaviour {
 
  [SerializeField] Camera _camera;
  [SerializeField] Vector3 _translation;
  [SerializeField] Vector3 _rotation; // in degrees
  [SerializeField] Vector3 _scaling;

#if UNITY_EDITOR
  void OnValidate() => updateCamera();
#endif

  void Awake() => updateCamera();

  void updateCamera() {
    if(_camera == null) return;
    var mtx = Matrix4x4.TRS(_translation, Quaternion.Euler(_rotation), _scaling);
    if(!mtx.ValidTRS()) mtx = Matrix4x4.identity;
    _camera.projectionMatrix = mtx;
  }

}

and see what’s what.

Edit: changed the example slightly.

1 Like

thanks, I will give both of these a try, tbh I dont know how to describe what I want with math

I want 3D terrain,
I want the field of view high because I really want the walls on the side to shift a lot while navigating the map,
and in the end the spin is that the characters are flat pixel art (2D top down, like ff6, zelda minish cap, etc)

but I dont want a flat orthographic camera, I want it to have a sense of depth, and when going high you can see far away terrain and chars very zoomed out, so thats why I want a perspective camera

here is what im going for:

link with timestamp

Here FOV is set to 25 and there is something called a skewFactor. So it likely skews the render on the Z axis, and as explained in the video, the effect comes across more strongly with the rotation, so someone coded an expression that takes something like a sine of a double angle and multiplies that with the applied skew factor.

Another thing to consider is how matrices get computed in the first place. And you should probably read more on the subject if you want to customize a projection matrix. I’m sure you’re aware that the game dev business is very complicated affair and many such questions don’t have easy answers.

In particular, you need to know about affine transformations if you want to produce skewed (aka shear) projections. But beyond that, you can very easily produce such a transformation by finding another, simpler matrix, and then multiplying it with the camera’s one.

Anyway to cut the long story short, the point of a matrix is to transform one vector space to another.
Here’s a good tutorial on what is going on. However, in Unity things run slightly differently, namely the matrix is transposed compared to those used in mathematics. It looks like this (in the tutorial you’ll see Tx, Ty, Tz at the bottom row, whereas in Unity those terms belong to the red box).

(Edit: Image is from this article, also a very useful read, and it even contains a hint, namely:
“The green part of the matrix is rarely used, except for shear operations and weird projections.”)

And if you check out this article (and this Q/A), it explains what skewing is, in the context of matrix terms.

If you find proper terms (which is already explained in that StackOverflow thread), you can apply it to your existing projection matrix by multiplying the two, thanks to this operator.

1 Like

thanks a lot for the info

thanks for spending the time to give me advice, i really appreciate it

1 Like