Rigidbody rotating too slowly

So basically I “made” a script that controls the hand movement, and has proper collisions too. Most of it works, however, the rotation seems to have a limited speed, even if its speed is set to high values.

The speed of the rotation is slow, no matter what I change the multiplier to.

Here is the code:

using System.Collections;
using System.Collections.Generic;
using UnityEngine;

public class Hand: MonoBehaviour
{
    // Physics Movement
    [SerializeField] private GameObject followObject;
    [SerializeField] private float followSpeed = 30f;
    [SerializeField] private float rotateSpeed = 100f;
    [SerializeField] private float teleportDistance = 0.75f;
    [SerializeField] private Vector3 positionOffset;
    [SerializeField] private Vector3 rotationOffset;
    private Transform _followTarget;
    private Rigidbody _body;
    private bool _inGrab = false;

    void Start()
    {
        // Physics Movement
        _followTarget = followObject.transform;
        _body = GetComponent<Rigidbody>();
        _body.collisionDetectionMode = CollisionDetectionMode.Continuous;
        _body.interpolation = RigidbodyInterpolation.Interpolate;
        _body.mass = 20f;

        // Teleport Hands
        _body.position = _followTarget.position;
        _body.rotation = _followTarget.rotation;
    }

    void Update()
    {
        PhysicsMove();
    }

    private void PhysicsMove()
    {
        // Position
        var positionWithOffset = _followTarget.position + positionOffset;
        var distance = Vector3.Distance(positionWithOffset, transform.position);
        _body.velocity = (positionWithOffset - transform.position).normalized * (followSpeed * distance);

        // Rotation
        var rotationWithOffset = _followTarget.rotation * Quaternion.Euler(rotationOffset);
        var q = rotationWithOffset * Quaternion.Inverse(_body.rotation);
        q.ToAngleAxis(out float angle, out Vector3 axis);
        _body.angularVelocity = axis * (angle * Mathf.Deg2Rad * rotateSpeed);

        // Teleport back when too far
        if(distance > teleportDistance)
        {
            _body.position = _followTarget.position;
            _body.rotation = _followTarget.rotation;
        }
    }
}

The script is attached to a hand model with a rigidbody, and the follow-target is the XR Controller. Everything seems to be working, except the speed of the rotation.

Rarely worked with quaternions, so it might just be some basic issue.

Configure Rigidbody.maxAngularVelocity to a larger value. Default is 7, you may use 20 for example.