Frame-rate Drop After Pausing Game

After setting my Time.timeScale = 0; the game pauses fine. However, after I resume the game and set it back to 1, the frame rate of the game drops considerably. The frame rate resets to normal everytime I reload the level though. Is there any way I can fix this?

A bit of necrophilia but does anyone has any idea what would cause this? Building a VR game for Oculus and have the same issue.

Game runs fine until timeScale is altered, then it runs poorly after