Picking up physical objects

I’m trying to implement a mechanic for picking up rigidbodies, but am not sure which solution to pursue.
I am looking for a mechanic that would pick an object at the current point in space (for example grab a cube by it’s corner if hand is currently there) and hold the object, while the object is still able to rotate around the point that the cursor goes through. Something like shown in the picture - grabbing the cube and moving it upwards makes it rotate around the grab point downwards.

Should I treat the cube as rigidbody with full physics and gravity (not kinematic object) and use forces to move it towards the point somehow? Or should I use non-physics based position and rotation translation to manually implement some system to emulate physics-like behaviour?

For any other people looking for answer to similar problems: I think I managed to prototype something pretty usable. What I’m doing in short:

  1. Doing Physics.Raycast on grab action (OnMouseButtonDown) and saving RaycastHit (it contains hitpoint position).
  2. Spawning (or just moving) an empty GameObject to the hitpoint position and locking it to that object as a child (to maintain relative position to the object). This empty transform will be our point where we apply the force.
  3. I apply a force with AddForceAtPosition from the empty gameobject to current cursor position. Tweaking the force function is key to making it behave correctly.

One of the problems I stumbled upon was really shaky behaviour of the object, especially when it got close to cursor. The solution was to incorporate a change to drag and angularDrag properites of the rigidbody based on current force being applied (the more force is applied, the higher the drag and angularDrag). This will help with the twitching of the rigidbody.