Moving an object only goes halfway a second move.

I have this script to move a object to a position.
The first time works great,
but when I move the object to a 2nd location it only gets halfway, and the 3rd time it moves even less.

What did I do wrong?
Also how can I move it at a fixed speed?, now it speeds up fast and slows down afterwards.

private var speed 	:float = 0.4;

function MoveToo(toPos :Vector3) :IEnumerator
{
	var distanceToTarget : float = Vector3.Distance(rTransform.position, toPos);
	
 	while(distanceToTarget > 0.1)
 	{
		rTransform.position = Vector3.Lerp(rTransform.position, toPos, speed * Time.deltaTime);
		yield;
	}
}

edit
It looks like it stil moves but only at such low unnoticable speed after a first move.

Edit2
Turns out this is what I was looking for:

private var speed 	:float = 0.1;

function TravelToo(toPos :Vector3) :IEnumerator
{
	var distanceToTarget : float = Vector3.Distance(rTransform.position, toPos);
	
 	while(distanceToTarget > 0)
 	{
 		distanceToTarget = Vector3.Distance(rTransform.position, toPos);
		rTransform.position = Vector3.MoveTowards(rTransform.position, toPos, speed);
		yield WaitForFixedUpdate;
	}
}

The problem is that you’re not using Lerp() correctly. Lerp() takes the Start and End values, as well as a Position along the transition.
As such,

Lerp(10, 50, 1.0) == 50 

while the opposite is also true

Lerp(10, 50, 0.0) == 10.

If you want to use the Lerp() function, you need to increment a variable from 0.0 to 1.0 and pass that to Lerp() as the third parameter. There are many, many ways to do this, but here’s one example.

(Beware, not actual code.)

Progress: float = 0.0f;

while()
   Progress+= speed * Time.deltaTime;
   transform.position = Vector3.Lerp(StartPos, EndPos, Progress);

Source: