The issue you described is expected issue of having speculative contacts for fast bodies. More about the issue you can find in this talk from the referenced time:
So if ground has to be mesh there are 2 approaches that I see:
- Try to implement customer contact modified like ModifyNormalsJob in UnityPhysicsSamples\Assets\Demos\5. Modify\Scripts\ModifyNarrowphaseContactsBehaviour.cs
- Switch to Havok physics