Players can't move each other in multiplayer?

Seems like Rigidbody players can’t move each other due to local player authority. Collision occurs (even visually for a moment), but other player reverts back in place immediately. How to disable or fix this?

Kind of fixed by setting movement, velocity, snap thresholds and interpolation to 0 and enabling rigidbody extrapolation. Still want to know the right way to do this (client side).