ClampMagnitude but with different speeds

So im trying to fix my character moving faster on the diagonals and it is still not working and i dont know what i am doing wrong. My character moves also moves slower when going backwards also going sideways. This is what i have tried:

float axisX = joystickInput.axis.x;
float axisY = joystickInput.axis.y;
            
// Speed and axis
if (axisY < 0) { speed = Mathf.Abs(-axisY) * backSpeed; } // Speed on the negative y-axis       
else { speed = Mathf.Abs(axisY) * forwardSpeed; } // Speed on the y-axis
speed += Mathf.Abs(axisX) * sideSpeed; // Speed on the x-axis

// Direction of the VR headset
Vector3 direction = Player.instance.hmdTransform.TransformDirection(new Vector3(axisX, 0, axisY));

// Fixes faster movement on diagonals
Vector3 clamp = Vector3.ClampMagnitude(direction, speed);

//Apply
CharController.Move(Vector3.ProjectOnPlane(clamp, Vector3.up) * speed * Time.deltaTime);

I like to work visually so I would recommend an AnimationCurve that shows how angle and speed relate. You can tweak this in the Inspector for better results:

    // from 0° = forward to 180° = backwards 
    public AnimationCurve speedByAngle = new AnimationCurve(new Keyframe(0, 1f), new Keyframe(90f, 0.5f), new Keyframe(180, 0.5f));

    void MoveController()
    {
        float axisX = joystickInput.axis.x; // assuming this goes from -1 to 1
        float axisY = joystickInput.axis.y; // assuming this goes from -1 to 1

        // create a direction vector directly from the input
        Vector3 dir = new Vector3(axisX, 0, axisY).normalized;

        // now calculate the angle between forward and your move vector
        // and use this to get the speed from the animation curve
        float speed = speedByAngle.Evaluate(Vector3.Angle(dir, Vector3.forward));

        Vector3 global = Player.instance.hmdTransform.TransformDirection(dir);

        CharController.Move(global * speed * Time.deltaTime);
    }