Combining IK and Physics for VR

I’m developing a VR game in Unity and have set up my player with a rig, using two bone IK for each arm. I’ve placed colliders on the right hand and want the hand to move towards the VR controller’s position using physics.

However, I don’t want to just attach a Rigidbody and collider to the target and move the target because the collider on the target won’t be as accurate as the one on the hand.

please give me advice im lost

I have never worked with VR but I think I know what you can do:

  1. We can achieve this by adding a force to the right hand
    first get the position of the VR right hand, lets call it vrRightPos (again ive never worked with VR so idk how you would do this)
  2. Now get the position of the characters right hand, lets call it rightHandPos
  3. Now calculate the direction of force by:
    Vector3 direction = (vrRightPos - rightHandPos).normalized;
    
    Normalize for this use case is optional, but I like to use it.
  4. now simply add an empty object, add an RigidBody, add character joint so it pivots around shoulder joint, change position of right hand to that empty object using IK and
  5. Now add an force to this empty object:
    emptyObj.rb.AddForce(direction * force); note, force is a public variable you need to set

This should work, you can also try increasing the magnitude of force with distance, that would make it smoother