VisionOS and PolySpatial are fudnamentally different beasts than standard XR as you get with HoloLens. On the HoloLens, and all other XR devices that Unity supports, Unity is entirely in control of all visuals. For VisionOS, you can get the same effect if you use the “Virtual Reality - Fully Immersive” mode for your app mode.
If you want to use the “Mixed Reality” mode for your application, though, things are much different. In this case, Unity is not doing ANY of the rendering that you are seeing and has no control over what you actually see or interact with. RealityKit is the rendering API and we have to do our best to work within that API to get things to work as best we can. There are significant limitations with the API, as well as device and user security/privacy when using MR mode that limit what we can do, display and react to.
If you feel that there is a bug that needs correcting, then please by all means file a bug report for that. Please make sure that you supply a small, repro project demonstrating the issue.
If you feel that there are missing features, or want to see other features that we may not know of, you can report that in the 
Unity visionOS Support Roadmap discussion area just for that.