Rigidbody2D jitter on collision. Photon Pun 2 sync issue?

Whenever one character(rigidbody2D) collides with another one, the other character starts to jitter and move along the velocity it was pushed with until it runs into something else, then snaps back to the position it should be. This only happens in the view of the player who pushed, in the other players view everything moves as expected.
If I set the Rigidbody2D to Kinematic then the collisions no longer cause jitter, however the characters no longer recognize walls/etc. I’m fairly new to coding and taking my time to learn everything but this one has got me stumped.

video of problem:

Movement Script

private void FixedUpdate()
        if (photonView.IsMine)

            movement = movementanim;

            //prevent unexpected movement from this rigidbody
            rb.velocity = Vector2.zero;

            movementanim.x = Input.GetAxisRaw("Horizontal");
            movementanim.y = Input.GetAxisRaw("Vertical");

            animator.SetFloat("Horizontal", movementanim.x);
            animator.SetFloat("Vertical", movementanim.y);
            animator.SetFloat("Speed", movementanim.sqrMagnitude);


            rb.MovePosition(Vector2.Lerp(rb.position, rb.position + movement * moveSpeed, 0.1f));

using Photon Pun 2 to sync transform. Movement controls are on Fixed Update and I’ve tried setting Interpolate to Interpolate to no success.

Not sure if there was a better solution but I was able to work around the issue by replacing the Rigidbody with a Character Controller. I had to make a couple small changes to the code of course, but the one that caught me by surprise is that the environmental colliders need to be 3D colliders now since the CC is a 3D component.