Low framerate on physics initialization.

I have a relatively small scene (256x256) that has a lot of colliders and rigidbodies (200-300). These objects are being used to create a large completely destructible structure.

On initialization i am getting some serious lag (1-2 fps). After everything initializes the frames jump back up to normal. I am assuming this is due to initializing the physics for all of these objects. What exactly can i do here?

I was able to fix this by changing the timestep and max timestep to .02 in the time settings.

this menu is located at edit > project settings > time

This still is not ideal as it causes in game physic to start slowly, it’s is much better than a choppy 2 fps.