Collisions performance question

Maybe while developing it’s not necessary, but is it recommended to disable collisions between all objects who won’t collide or you don’t care if they do, when deploying a release build? Or the performance gain is not worth the effort?

The physics should be pretty optimized in this regard, so I would spend time looking for performance bottlenecks elsewhere.