Random phyics spikes

Hi! I get random spikes related to physics as shown in the profiler. This happens in the main menu too when there is no physics at all (except there is a background which has a meshcollider with a quad mesh and a physics raycast, otherwise this does not do anything).

Game is 3D, using win7, unity 5.3.5. Physics processing shows 99% activity but only for a spike and then drops. Weird thing is this sometimes happens often as shown in the image, but sometimes i have to wait 5-6 seconds for a spike (while just staring at the profiler).

I have tried turning off the raycaster. I don’t remember having this issue with earlier versions of Unity but maybe i just didn’t notice…

Problem seems to solve itself somehow… If i have unity running for a while this problem disappears.