My colliders seem to jump up unwanted at start. I do not apply any forces at start, only at a raycast. After start, the colliders behave as they should.
I have multiple colliders per object, although they do not touch.
I accidentally duplicated them over each other. Stupid, stupid, stupid.