How do I sync XR Toolkit grabbing over Photon 2 (Pun 2) for multiplayer VR?

I have been trying to use XR Toolkit and Photon 2 Networking to create a multiplayer experience. When Player A grabs an object it looks fine. Player B, watching that interaction, sees the object that Player A grabbed fighting against gravity/kinestic. I have tried multiple ways to fix this but have not succeeded. Any ideas on a solution for this? Thank you in advance.

Did you find any solution ?

Anyone figured out how to fix this?