I think that Slerp will inherently cause problems when interacting with a physics object because Slerp operates outside the bounds of the physics simulation. Rigidbodies and colliders move and interact with each other during Unity’s FixedUpdate phase, which is when the physics simulation updates. At that time, physics objects move, collisions are resolved, and everything finds a valid position to be in given the colliders, rigidbodies, and forces at play in the scene.
Now, consider using Slerp. Slerp is just an operation you’re applying to the switcher’s transform. There’s nothing to stop you from telling the switcher’s transform to move into the center of the rigidbody…or to move right through the rigidbody. Part of the reason the rigidbody jumps all over the place in your bug.gif is because the switcher’s collider is getting embedded inside the rigidbody’s collider, which causes the simulation to freak out.
To fix this, you should NOT move the switcher’s transform directly, but move the switcher’s rigidbody instead. This should allow the physics simulation to update correctly, but I can’t be totally sure. See Unity - Scripting API: Rigidbody2D.MoveRotation for an example of rotating the rigidbody directly. It also offers an explanation of why physics events still work correctly when you use this method.