Scenario is Collider bounds all fit in a grid. Bounds extents are all multiples of 0.25. Most colliders are box colliders a few are mesh colliders. Rotation is always at 90 degree angles.

I create an undirected graph of 'connected' colliders. Simplest way to think of connected is assume all colliders are boxes and ones that have faces that share the same space are connected.

Currently for every bounds I take a box collider of the same size/rotation and inflate it by 0.125 and do a CollisionWorld.ColliderCast.

The current approach has one functional issue which is it hits colliders that are only connected by a corner not face. I haven't implemented a fix for this part yet. Current idea is that since everything is at 90 degrees for each collider that I hit I could then take it's aabb and move it both directions on each axis and see if it intersects the source non inflated aabb. That's 6 bounds checks. Average number of connected colliders is probably in the 3-6 range.

The scale of this is into the low hundreds of thousands, although it's partitioned so never that many colliders in the scene. Building connections is incremental and persisted. But even with that there is a worst case where we have no connection data so it's worth optimizing more if we can.