Rotating an Object with a custom vector

Unity Engine
1

So im developing a touch game, where the player can rotate objects by touching them then moving their finger.

The way it works is that im raycasting from camera, first to see if the raycast hits the object (to make sure the player clicked on it), then it rotates the object into the the new raycast point.

I calculate the difference between the objects position and the raycast hit position to create a vector that points at the new position, then set the objects transform.up (or transform.right) to the new vector, which gives me the wanted rotation.

this system is working for me, it is giving me the kinda controls i want, but my problem with it is that its very snappy; for example, if i touch an object thats pointing upwards on its bottom, the code will immediately set the transforms up vector to the new vector i calculated which will make it look downwards.
I dont want this to happen, i want the change to happen only after i start moving my finger.

2

To be honest, this sounds less like a question of physics and more of a question of Scripting how to. Setting Transforms isn’t physics, in-fact it bypasses physics completely and instantaneously sets a post.

That said, something so visual isn’t really easy to follow from a rough description.