Rotate single axis of object to nearest 90 degrees

I’m working on a 6 DOF game, so I have a character that flies around, with mouselook, and rotates properly.

What I want to do now, is if the player isn’t rolling (by pressing Q and E), I want to “correct” the roll of the quaternion to the nearest 90 degree angle (so the ship levels itself). So, I guess I want to modify the rotation, along the transform.forward axis, to go to the next increment of 90.

It’s easy to do if I just modify the eulerAngles, but it doesn’t quite have the effect I want - it doesn’t take into account the forward vector, so as soon as you turn upside down or at a strange angle, it freaks out and doesn’t rotate properly.

Here is what I have now:

    private void HandleMouseLook()
        float mouseInputX = Input.GetAxis("Mouse X") * mouseSensitivityX;
        float mouseInputY = -Input.GetAxis("Mouse Y") * mouseSensitivityY;
        float rollInput = Input.GetAxis("Roll") * rollSpeed; // Roll is Q and E keys

        var rotation = new Vector3(mouseInputY, mouseInputX, rollInput) * Time.deltaTime;

        // If rollInput == 0.0f, correct rotation along transform.forward axis to nearest 90 degrees

I’d recommend splitting your GameObject into two, and making one of them the parent of the other.

The parent GO would handle the mouse look rotation you already have, while the child rolls relatively to it. The child also contains the model, so both rotations get combined for it. Moving the child’s local rotation towards the nearest 90 degree step is rather easy.

Now you can choose: Does rolling affect how the mouse look rotation works, or is it rather just a visual thing? If it’s just visual, you can stop here. If not, you will have to read the child’s local rotation and modify your mouse look rotation vector in the parent’s script accordingly. Transform.TransformDirection will probably help there.

var rotation = new Vector3(mouseInputY, mouseInputX, 0);
ShipComponent.RigidbodyComponent.AddRelativeTorque(rotation* Time.deltaTime);

I haven’t tested this, but it should give you a direction to go for.