Configurable joint keeps distance when forced

Hi, I’m using a configurable joint between a vr controller and a hand mesh, the hand has a collider and a dynamic rigidbody when the controller has a kinematic rigidbody and the joint.
This way when I try to move the controller though an object the hand stop when touches the object.

The problem is that if I move the controller deep inside the object (far from the entry point) and move it back outside the object the hand keeps the offset it has when the controller was inside the object.

Can someone see the problem ?
Thanks in advance

I forgot to show the linear limit that is 0, then the hand should return to the controller position with the linear limit spring…