Spring or configurable joint - fluent moving of connected rigidbody

HI guys,
I’m trying to fluently move connected rigidbody using spring or configurable joint when player rotates around the Y axis.

Basically I grab an item on mouse down and drag it until mouse up triggers. I’m pratically trying to simulate the object drag and drop from Stranded Deep survival game (for those who know it).

My problem is Fixed update is not sufficient for this operation - the jittering of the connected rigidbody is pretty visible = rotating player in FixedUpdate is not a good option.
Not sure whether there is a way to move/sync the physics of the connected rigidbody with the player transform
in Update. I tried to use Physics.SyncTransforms method just before rotating the player transform without any success.

Only the way I’m able to do the movement more fluent is shortening of the fixed update interval significantly which would highly probably lead to a considerable performance loss.

Every advice is welcome!

Setting interpolation for carried rigidbody has completely fixed the problem.

However I just found a better overall solution for carrying items anyway.

Attaching the first draft code for those interested:

using UnityEngine;

public class ItemCarrying : MonoBehaviour
{
    [SerializeField] private Transform _grabHolder;
    [SerializeField] private float _grabReach = 4f;
    [SerializeField] private float _followSpeed = 5f;
    [SerializeField] private float _carriedItemAngularDrag = 10f;
    [SerializeField] private Transform _camera;

    private bool _carrying;
    private Rigidbody _carriedRigidbody;
    private Vector3 _offset;
    private float _originalAngularDrag;

    private void Update()
    {
        //also use layer mask to avoid colliding with character controller or other collider which is part of player
        if (Input.GetMouseButtonDown(0) && Physics.Raycast(_camera.position, _camera.forward, out RaycastHit hit, _grabReach))
        {
            _carriedRigidbody = hit.collider.GetComponent<Rigidbody>();
            if (_carriedRigidbody)
            {
                _carrying = true;
                _offset = hit.point - _carriedRigidbody.transform.position;
                _originalAngularDrag = _carriedRigidbody.angularDrag;
                _carriedRigidbody.angularDrag = _carriedItemAngularDrag;
            }
        }
        else if (Input.GetMouseButtonUp(0) && _carrying)
        {
            _carrying = false;
            _carriedRigidbody.angularDrag = _originalAngularDrag;
            _carriedRigidbody = null;
        }

        if (_carrying)
        {
            _carriedRigidbody.velocity = (_grabHolder.position - _offset - _carriedRigidbody.transform.position) * _followSpeed;
        }
    }
}