If virtual offset worked reliably for most cases, it would solve a lot of issues and reduce the amount of probe adjustment volumes you need to use for leak free GI with APV, but often it does nothing for probes insides of geometry.
Video: xd1n85
Not sure what the issue in particular is, but it seems to be mesh collider related.
I tested this many different virtual offset settings on video, but also off video including test bakes, virtual offset seems to never run on most probes inside of the scene walls.
I show the meshes/colliders at the end of the video.
Unity: 2023.1.0f1