Destroy and add new rigidbody breaks it

Unity Engine
1

We have some features that require us to delete rigidbodies after they have been attached as childs on other transforms, for example when a magazine is inserted into the firearm.

When it becomes a standalone item again we will readd the rigidbody. Here is the code

public class RigidBodyTemplate
    {
        private float angularDrag;
        private float mass;
        private float drag;
        private RigidbodyInterpolation interpolate;
        private CollisionDetectionMode collisionDetectionMode;
        private bool isKinematic;
        private bool useGravity;

        public RigidBodyTemplate(Rigidbody org) {
            mass = org.mass;
            drag = org.drag;
            interpolate = org.interpolation;
            collisionDetectionMode = org.collisionDetectionMode;
            angularDrag = org.angularDrag;
            isKinematic = org.isKinematic;
            useGravity = org.useGravity;
        }

        public Rigidbody CopyTo(GameObject obj)
        {
            var rigidbody = obj.AddComponent<Rigidbody>();

            rigidbody.mass = mass;
            rigidbody.drag = drag;
            rigidbody.interpolation = interpolate;
            rigidbody.collisionDetectionMode = collisionDetectionMode;
            rigidbody.angularDrag = angularDrag;
            rigidbody.isKinematic = isKinematic;
            rigidbody.useGravity = useGravity;

            return rigidbody;
        }
    }

I have also tried messing about with centerOfMass, inertiaTensor, inertiaTensorRotation but no diffrence.

The new rigidbody does not behave as it should.

Example here. First with original rigidbody, than I drop the item and the code above is executed. See how it behaves after. (The speed should be as fast as in teh first example but it lags behind my real movement)

2

Here is a more organic example, the one above I forced the bug unnatural, here it is like it it is in game

The item is first freefloating and workign correctly. Then its attached to another body, first using a config joint and finally we delete the joint and the body and let the transform be a complete slave to the parent transform. Then when you detach the item we recreate the body and the problem starts.