does interpolation full up some kind of cache over time

It seems that performance (frame rate) degrades after a certain amount of time playing with my rig. But if I go make a cup of tea, pour a glass of wine or take a piss and come back, the performance is back up where it started at, but degrades quickly again once I start moving things around again. All objects in my scene (test rig) are connected to one another with physics, there's nothing else going on.

Physics objects go to sleep if they don't move for a certain lenghth of time, most likely you're seeing a large frame rate boost when all of your objects are asleep, then you moving them again wakes them up