Tilt character with the ground and rotate it


I am making a game where the character tilts with the ground. I managed to do it pretty smoothly with the code below which uses Raycasts from 4 sub-objects. For some reasons, it is best I keep a Character Controller.

public Transform backLeft;
public Transform backRight;
public Transform frontLeft;
public Transform frontRight;
private RaycastHit lr;
private RaycastHit rr;
private RaycastHit lf;
private RaycastHit rf;
private Vector3 upDir;
void Update () {
    Physics.Raycast(backLeft.position + Vector3.up, Vector3.down, out lr);
    Physics.Raycast(backRight.position + Vector3.up, Vector3.down, out rr);
    Physics.Raycast(frontLeft.position + Vector3.up, Vector3.down, out lf);
    Physics.Raycast(frontRight.position + Vector3.up, Vector3.down, out rf);
    upDir = (Vector3.Cross(rr.point - Vector3.up, lr.point - Vector3.up) +
      Vector3.Cross(lr.point - Vector3.up, lf.point - Vector3.up) +
      Vector3.Cross(lf.point - Vector3.up, rf.point - Vector3.up) +
      Vector3.Cross(rf.point - Vector3.up, rr.point - Vector3.up)
    transform.up = upDir;

But the problem now is that the character can’t rotate anymore (I am using transform.Rotate with left/right arrow keys). I understand this is because I reassign transform.up, but I haven’t found a solution for now.
I’ve tried several things, but none worked. The most satisfying was using a temporary variable to keep the rotation on y axis, but then the character isn’t aligned with the correct normal anymore after he changes his starting rotation.

Has someone got an idea?

Rotate the position of the up vector like this:

   transform.rotation =  Quaternion.FromToRotation(transform.up, upDir) * transform.rotation;