Object falling from high are passing through ground

First, you can see the bug in action here: Unity 3D gravity collision bug - YouTube

When rigidbody objects are falling from high position (300 in my test), the ground collision doesn’t works, the objects falls through ground.
As you can see in the video, when I change the height to 30, it works.
The spheres have rigidbodies using gravity and a sphere collider.
For info, the grounds have mesh collider for the two on the right, and a box collider for the one on the left.
I’ve tried to add kinematics rigidbodies to the grounds but it doesn’t changes anything.

Using Unity 2020.3.28f1.

You can find a minimal reproduction here: GitHub - BenDevelopment/Unity-3D-gravity-collision-bug: Unity 3D gravity collision bug

To prevent objects from flying through other objects at high speed, you need to specify continuous detection mode in the rigidbody settings. More details about this can be found, for example, here:

Collision detection modes in Unity’s Rigidbody component

I don’t see that you have tried adjusting the “collision detection” options
“discrete” is the default and maybe something like “Continuous” would work better in your case.