I have simple scene where a small cube is falling towards my terrain.
My terrain was generated from a height map taken from one of the unity free terrain assets.
It has a 2049x2049 resolution. (raw data size 8MB).
The terrain is added a terrain collider built from the same raw data.
The cube has a rigidbody and a box collider.
Small cube : all good
In the scene the cube falls, and starts bouncing around the terrain.
When the cube is small, all is well, including the FPS.
Large cube : FPS = 3
If I make the cube a lot bigger, so its base is about 15% the area of the terrain, and I run the same scene, once the cube touches and starts bouncing off the terrain, the FPS drops to 3 FPS.
In general read the Documentation about the Physics settings in the Unity Documentation. You can adjust the solver iteration count for example or the distance at which a collsion is detected as such.
BUT
From my experience no matter what you do will really make this “good”. The elefant in the room is that you have a giant high resolution mesh collider that Unity will check for each frame if there is any collision with any of the million of vertices. So in the end if you want to really speed this up: Reduce the quality of your terrain. This would be the proper solution here as the detail in your map will not even have any noticable impact on this. If you had only 1% of the total verices you currently have the cube would probably still behave almost the same.
If you have to keep the visuals then perhaps you can find a way to simplify only the collider into a less high-res version and keep the visuals of the map you currently have but that is for you to decide which path you want to follow.
