There are multiple ways of optimizing your scene with respect to colliders and rendering as well.
One option is to add mesh colliders and set them to Is Convex. This will reduce the level of detail of your mesh collider, whilst keeping a sub-accurate representation of your model’s collider. On square-like buildings, mesh colliders set to IsConvex still function exactly the same, but reduce the verts on your collider by a lot.
Another option is to use occlusion culling. You can do this manually or using Unity’s automatic system. To do this manually, when a building is out of the player’s view (so like 100m away), disable the building object entirely (or just the collider/renderer). This way, less objects need to be simulated at once. To do this automatically, set the object to static (occlusion static), and make sure occlusion culling is enabled on your camera. It will automatically disable renderers of objects out the camera view for you.
For rendering, you can merge all the building’s meshes into one mesh (or sub meshes per material, if they have different materials), then add a mesh collider on the combined object/s. This reduces draw calls and optimizes physics calculations, as there are less objects for the GPU to worry about drawing at once.
Another tip is to set the building game objects to static. Lightmap static optimizes lighting calculations (since the buildings don’t move so realtime lighting isn’t needed), and batching static allows similar materials to be batched together (which reduces draw calls). If you need dynamic batching (objects can’t be static), in the material properties, in advanced, enable GPU instancing. This produces the same performance increase as batching the objects together.
Finally, LOD’s. You can set up an automatic level-of-detail system for your buildings using the Unity LOD camera component. By setting buildings further away to a lower level of detail, you can optimize the collider and mesh geometry by a whole lot. There are a few tutorials of using LOD systems, and simplifying geometry.
Hope these tips are useful. I know there are a whole lot more that I’m not aware of, but good luck 
@CerebralStatic