Attempting to Set Player Position Start of Scene But Player Behaves Strangely

I recorded the instance in above video. Left window is server, right is client. Player hits the collider to additively load the next scene, moves strangely, and then after a pause I give it a new positional input and the player snaps to the expected point.

-There is a starting room amongst many rooms, when the scene starts the room finds X players and sets their location to defined spawn points
-Currently there is no client authoritative movement, so this is all done on the server
-When a client hits the scene loading box, all players in the scene have their input locked and RigidBody velocities cleared via ClientRPCs
-Input is unlocked after the new position is set

Things I have tried that have not resulted in any changes:
-Starting a coroutine to wait a frame after setting the new location
-Setting the location via a ClientRPC called by the server after setting the location on the server in hopes of better syncage
-Stalling the call to set the player's location in case its conflicting with something happening during the scene load, I am not sure how to tell 100% but as far as I know I am calling it after everything is loaded
-Turning off the colliders after additively loading the scene as I noticed the player model and the rigid body seem to desync, so I was worried the server was trying to 'move' the player to the new location but actually was colliding with walls along the way

The part that is leaving me really perplexed is it sometimes works OK, but I am unsure what is happening with the packets that is causing such a tug of war between two position. Since after additional frames of input the correct location eventually is shown on both the server and the client, I am unsure where the issue lies.

I think I solved it, the rigidbody was colliding with the wall, so setting it to Kinematic, then moving the player, then turning it back, prevented the collision and correctly set the player's location.

The above did not fully solve it. I got fairly close with turning off the network object's interpolation, but if I do something like this psudo-code on the server:

rigidbody.kinetic = true;
networkobject.interpolation = false;
transform.position = new_position;
networkobject.interpolation = true;

It will not work, however if I omit turning back on the interpolation, it works perfectly. I believe what is happening is the interpolation is turning on too fast due to something with how its interacting with the stack? I tried with ServerRPCs to put the steps on the stack in order, so if I did the first two lines as one RPC, the third as another, and the fourth as another, I hoped it would execute it stepwise with only doing the next while the previous is done, but this was not the case. I am unsure how to slow it down.