How to switch 2 objects' positions gradually?

GameObject o1, o2;
Vector3 o1Position, o2Position;
bool access = false;
float speed = 0.50f;

void Update()
{
   if ( /* it's necessary to switch these objects */ )
   {
      o1Position = o1.transform.position;
      o2Position = o2.transform.position;
      access = true;
   }

   if(access)
   {
      o1.transform.position = Vector3.Lerp(o1Position, o2Position, speed);
      o2.transform.position = Vector3.Lerp(o2Position, o1Position, speed);
   
      if(o1.transform.position == o2Position)
      {
         access = false;
      }

   }
}

I try Lerp as you can see above but that only moves my objects half way to their destinations and they end up sharing the same positions. Is there another way to switch their positions slowly or how can I fix one? I played around with speed variable but it didn’t help. Thanks in advance.

If you don’t want eased movement at the end, do:

 o1.transform.position = Vector3.MoveTowards(o1.transform.position, o2Position, speed * Time.deltaTime);
 o2.transform.position = Vector3.MoveTowards(o2.transform.position, o1Position, speed * Time.deltaTime);

‘speed’ will be units per second for movement.