Moving RigidBody to a point in a spherical world

I have a planet ( a spherical planet) that have units on the surface , that units are attracted by the planet with this

  public void Attract(Transform body,Rigidbody rigidBody)
        Vector3 gravityUp =  (body.position - transform.position).normalized;
        Vector3 bodyup = body.up;
        Quaternion targetRotation = Quaternion.FromToRotation(bodyup, gravityUp) * body.rotation;

        body.rotation = Quaternion.Slerp(body.rotation, targetRotation, 50*Time.deltaTime);

I want them to also move to a specific point in the planet but I can’t make it work, I tryed lot of things, the most “near” to the solution was this code on the Update method, but his make unit to “bounce”,

Vector3 dir = (positionToAchieve - transform.position).normalized * speed;
rigidBody.velocity = rigidBody.velocity + dir;

Any idea /suggestion will be welcome, I am new in Unity and I am learning it

public Transform target;
public float speed;
void Update() {
float step = speed * Time.deltaTime;
transform.position = Vector3.MoveTowards(transform.position, target.position, step);