The simplest test scenario possible, navmesh baked on cube, 120 agents doing absolutely nothing, no code involved. Bug? Unity 5.4.
Looks like despite being idle, agents are aware of their neighbors. Crowd.Proximity.Collect looks like a method to find each agent’s neighbors (to check if they are colliding with each other and such).