So dynamic bodies are just not stable enough when we factor in buoyancy and collisions using a simple single body setup. Our boats need mostly the same level of predictable behaviour that characters have.
So formulating a plan and looking for other ideas I might have missed. Mostly are there simpler approaches I missed as this is all getting a bit complex.
The basic idea is start with a very simple stable body, a cylinder and move it on flat plane. I setup a test controller for this and it is extremely stable with some basic tweaking of mass/damping/friction/restitution.
Boats have to interact with terrain as well as runtime placed structures that can be in the water, and other boats. For stability all surfaces boats interact with other then the ground plane have to be 90 degrees, straight up. So I’d need to build out special colliders just for boats to interact with. Items are easy just add box colliders and increase the y axis by some amount. Terrain I was thinking build a convex hull from points so the terrain edges are all vertical faces. Not a lot of accuracy needed there, faces within 10 meters or so of where water meets terrain would be fine.
For buoyancy create dynamic bodies that are only affected by buoyancy and don’t interact with anything. Transform the force positions from where the ship really is to these bodies.
Finally the body that is displayed and that characters interact with is kinematic. We combined the simple dynamic moving body with the buoyancy results and the result of that is the kinematic body movement. So we move the kinematic body via changing Translation/Rotation directly, but we also have to track the combined linear velocity as the character controller solver will need that.
Buoyancy it looks like I could just rip some of the logic from Unity.Physics for applying and then integrating velocities, skip bodies entirely there.