Syncing large amounts of colliders in UNET/Photon?

I need to sync many game objects’ positions (rigidbodies) across the network using UNET/Photon, and these game objects are essentially voxel structures that contain many box colliders (so I can easily update the colliders without rebaking the collision mesh.)

However, CPU usage goes through the roof when I try to sync these objects across the network. In the profiler, the main culprit is PhysX.PxsSap.sapUpdateWork.

I think that sapUpdateWork is caused by changing physics structs around because certain colliders move away from each other. (https://answers.unity.com/questions/1363522/moving-many-static-colliders-bad-performance.html)

This only happens to the client who does not own the objects, and when bad, it decreases FPS to 2-4 FPS. How can I fix this?
Here is a link to a picture of the profiler.

Okay, after messing around with settings for a while, I found out that Unity does not like it when you move a bunch of colliders while a rigid body is trying to do physics. I simply set the rigid body to be kinematic, and this fixed it. It is not as smooth as I would have liked it, but it works for now.