Sphere (Ball) Collision Over Network

I currently have 2 vehicles which are controlled by two players over network. I have managed to smooth out the vehicle movement and to some extent collision between them using interpolation/extrapolation.

However, I also have a ball which when the clients vehicle collides with it, massive problems occur. For example, the vehicle will be thrown into the air or the ball will just go through the mesh of the vehicle etc.

Could I get some pointers as to what I should be looking at to try and get it react as close to what the server’s vehicle does when colliding with the ball but on the client side.

Would it be interpolation/extrapolation or is there more to it when working with objects such as a sphere with physics?

Thinking about this some more. I guess my problem is that, currently the server owns the ball whereas I should make each client own the ball and then send the collision data to the server and update each client. Yes or no?

Another idea.

When a player joins, add another (duplicate) sphere collider to the ball and make it local to that player? Would that work? Can you make a collider local or only the object itself?