Non-Kinematic Rigidbody movement via Transform

Hey all, my setting is in VR and I am trying to have good character control (hand model moving with wand controller) via rigidbody (hand model is able to interact with items or other characters realistically).

My problem: Any physics interaction causes the model collider to “move out of place”, that is the model will eventually no longer be where the actual user’s hand is.

I would like to use Kinematic so I can set the transform directly, if it were possible for kinematic rigidbodies to collide with each other, but unfortunately they do not.

In FixedUpdate:
My initial attempt was using a mixture of Rigidbody, Rigidbody.MovePosition and Rigidbody.MoveRotation, and a lerp between the current position and rotation to the target position and rotation of the user’s hand. This causes the above problem. Rigidbody is non-kinematic, all constraints unlocked.

Second attempt was using Rigidbody, rigidBody.Position and rotation and again using lerp. Same settings, same issue. Also tried using a Vector3.MoveTowards to no avail.

Third attempt, setting the rigidbody.position/rotation directly to the target position/rotation. Same issue.

Couple other attempts with a mixture of these implementations

I really don’t understand why the model collider can’t just stay onto the target position. What am I missing here? Any advice and insights are appreciate.

EDIT: Also to note, it’s a multiplayer game so the other characters have rigidbody too.

Thanks,
Andy

There is a magical thing called a PID controller
Enjoy!