Hybrid Mode Switching Bugs/Questions

When I load into my hybrid app I start in Polyspatial where I display my main menu floating in a passthrough environment, this works.

Then I load into a 3D scene which uses Metal rendering that takes over your vision so you don’t see the passthrough video feed anymore. This also works.

Then I try transitioning back to my Polyspatial menu and I get back, but now I get no passthough video. Just white, I can see my UI though so rendering isn’t completely broken

I can somewhat replicate the issue in the editor, but I need to loop through the process twice for some reason.

So: Menu (PolySpatial) > 3D Scene (Metal) > Menu (PolySpatial) > 3D Scene (Metal) > Menu (PolySpatial)
Then I get a white background instead of blue (editor doesn’t have passthrough)

I think this might be related: If I look in the editor in my initial polyspatial use I can see this “PolySpatialRoot”

Menu (PolySpatial)

When I go into metal mode it remains, which maybe isn’t good? Do I even need that here in metal mode?

3D Scene (Metal)

Then go into the menu, then back to the 3d model. Now I’m in the metal again, I see a flash of white before the scene loads that I didn’t see before at the exact time the polyspatial root gets destroyed.

3D Scene Second Attempt (Metal)

Then I return back to the main menu, polyspatail root is gone. White background.

Menu (PolySpatial?)

If I were to guess, I’d say the polyspatial root is important for the passthrough camera to work, is that correct?

Is there a way to restore it?

Also I get this stack trade the second it gets destroyed I think:

[Diagnostics] EXCEPTION InvalidOperationException in PolySpatialCore:
  at Unity.PolySpatial.Internals.AssetRepresentation.get_Resource () [0x00017] in /Users/bokken/build/output/unity/quantum/Packages/com.unity.polyspatial/Runtime/AssetManagement/AssetRepresentation.cs:36 
  at Unity.PolySpatial.Internals.LocalAssetManager.ProcessChangedAsset (Unity.PolySpatial.Internals.AssetRepresentation representation) [0x00050] in /Users/bokken/build/output/unity/quantum/Packages/com.unity.polyspatial/Runtime/AssetManagement/LocalAssetManager.cs:1194 
  at Unity.PolySpatial.Internals.LocalAssetManager.FetchAssetChangesCallback (Unity.PolySpatial.Internals.ObjectDispatcherProxy+TypeDispatchData data) [0x00076] in /Users/bokken/build/output/unity/quantum/Packages/com.unity.polyspatial/Runtime/AssetManagement/LocalAssetManager.cs:1140 
  at Unity.PolySpatial.Internals.ObjectDispatcherProxy.DispatchTypeChangesAndClear (System.Type type) [0x0006f] in ./Library/PackageCache/com.unity.polyspatial/Runtime/InternalBridge/BridgeUtils.cs:205 
  at Unity.PolySpatial.Internals.LocalAssetManager.ProcessChanges () [0x00055] in /Users/bokken/build/output/unity/quantum/Packages/com.unity.polyspatial/Runtime/AssetManagement/LocalAssetManager.cs:817 
  at Unity.PolySpatial.Internals.PolySpatialUnitySimulation.UpdateInternal () [0x000f8] in /Users/bokken/build/output/unity/quantum/Packages/com.unity.polyspatial/Runtime/Trackers/PolySpatialUnitySimulation.cs:257 
  at Unity.PolySpatial.Internals.PolySpatialUnitySimulation.Update () [0x00014] in /Users/bokken/build/output/unity/quantum/Packages/com.unity.polyspatial/Runtime/Trackers/PolySpatialUnitySimulation.cs:195 
  at Unity.PolySpatial.Internals.PolySpatialCore.PolySpatialAfterLateUpdate () [0x000e8] in /Users/bokken/build/output/unity/quantum/Packages/com.unity.polyspatial/Runtime/PolySpatialCore.cs:698 

Using the debugger I realised that it’s this load scene additively function.

Somehow causes polyspatial core to think it should unload things and the exception kills it

I think the root disappearing might relate to this post here.

I removed all the objects that were in the scene that was being additively loaded and the root didn’t throw the exceptions and wasn’t destroyed. (I didn’t need the things in the scene, it was just legacy stuff).

Can’t test on the real headset until tomorrow.

Maybe worked out my issue. Thinking about what that root was doing was maybe a red herring.

While metal rendering seems to visually work fine when I use static object, non-readable textures and meshes. It must kill something somewhere.

I can see when I’m in my scenes that I get errors every frame complaining about issues like that coming from the polyspatial core.

This must break polyspatial in some why. So when I try switching back to RealityKit the camera feed no longer works.

I’d be great if I could use read only textures/meshes and static batching in metal rendering because it clearly visually works, without breaking polyspatial when I try and return to that mode.

Anyway that sort of unblocks me for now other than the performance and memory negatives.

Hey there! Happy 2025 :smile:

There’s a lot going on here so let me try to parse through what you’re seeing and answer your questions in aggregate. As always, we’ll be able to identify and solve any problems you’re seeing if we are able to replicate them on our end. The easiest way to do that is with your full project attached to a bug report (Help > Report a bug...) but short of that, a simple scene that you can instruct us on how to build step-by-step, or an isolation project with that scene is ideal. Suffice it to say, what you’re seeing sounds like a bug (or possibly a few different bugs), but it could be coming from anywhere. While Hybrid mode ends up giving you the best of both worlds (RealityKit and Metal) you also end up facing bugs from both worlds, including bugs unique to the “switch” between rendering with Metal and RealityKit. Note that we never actually “switch off” PolySpatial in Hybrid mode. Seeing the PolySpatialRoot disappear may not be entirely right, but it is not due to PolySpatial being “turned off.” More likely, this is something happening because volumes are being created and destroyed, or as part of cleanup being done during scene loads.

Now let’s get to some individual questions:

PolySpatialRoot and many of the objects in the DontDestroyOnLoad scene are not created in visionOS builds. They are “backing objects” we create in the Editor in order to simulate what the RealityKit backend is doing for debugging purposes. These objects are not required for metal mode, and pertain specifically to what PolySpatial is doing. It is fine if you see them while using metal rendering in Hybrid mode, because they are likely just the result of PolySpatial doing its thing. Even if you never trigger the creation of a RealityKit volume or immersive space, there are some circumstances where you may still see a PolySpatialRoot and you can safely ignore it.

This is incorrect. PolySpatial/Unity does not directly control passthrough rendering on visionOS. The only way to fully disable passthrough on visionOS is by opening an immersive space with the “Full” immersion style. But in that case, the background is always black, and you will still sometimes see your forearms and objects or people vignetted to “poke through” if Apple decides you need to see that stuff. In Unity, immersion style is a build-time setting under Project Settings > XR Plug-in Management > Apple visionOS, just in case you’re curious.

Since you’re seeing a fully white background, you must be somehow still rendering with Metal, or your RealityKit content includes a giant white sphere (or something) that is rendering white in an immersive RealityKit space. In PolySpatial terms, that would mean you have an active volume camera somewhere whose output mode is set to Unbounded or Metal. Bear in mind that, with default settings, when you load a scene without a volume camera we create one for you, and its default output configuration is Unbounded.

As you discovered in your later posts… this is the real issue. We shouldn’t be seeing exceptions within PolySpatial code, and that means you’re hitting a bug that we need to fix in PolySpatial. We do our best to fail gracefully in this case, but from a certain POV, “all bets are off” once you hit an exception like this. In other words, once you start hitting exceptions in PolySpatial, you will likely see “undefined behavior” like this.

Any of the scene load APIs will trigger Unity to internally evaluate and garbage-collect resources, so it’s not too surprising that even an Additive load would trigger some assets to unload. It’s possible that they were destroyed previously but some reference was keeping them alive until recently, or they just weren’t cleaned up properly at some point, and the scene load is “kicking” Unity/PolySpatial to actually unload them at the moment of scene load. Without being able to replicate these exact circumstances, it’s impossible to know. Stack traces won’t help, because the problem likely occurred well before you hit the actual exception.

Kind of? Anything causing exceptions within PolySpatial core code can lead to odd behavior like this. But yeah it sounds like this person was also seeing the PSL root object disappear when resources were loaded/unloaded properly. With any luck, you might see better results after upgrading the PolySpatial package, but I have a hunch this is something we haven’t accounted for.

Removed what objects from what scene? You removed all the objects from the additive scene… so you’re additively loading an empty scene? Did you just remove objects that referenced assets? Is it possible that those objects included scripts that created/destroyed assets and that is what was causing the exceptions?

Could you clarify what you mean here? Is there a texture import setting or scene configuration that you can change to eliminate the issue? Or are you just pointing out that the scene renders properly in Metal in isolation? Is the script calling LoadSceneAsync still involved?

I should point out that loading/unloading scenes is not required to turn Metal rendering on and off. All you need to do is change the volume camera output configuration used for your immersive scene from Unbounded to Metal (or just add a Metal volume camera). In other words, a single scene with just a volume camera changing modes is enough to tell Unity we want to start/stop rendering with Metal. This may help to work around your PolySpatial exceptions while we work on a fix.

I can’t think of a reason why rendering with Metal would be “killing” anything related to PolySpatial. From what I can tell, it’s far more likely the scene load is the cause. I’m curious to know if you see the same issues if you build you app in RealityKit app mode, instead of Hybrid. I wouldn’t expect metal rendering to work in that case, but will probably still see the PolySpatial exceptions.

Can you share the stack trace for one of these errors? Is it just one error every frame? Is it the same error all the time?

Again, this won’t be “camera feed doesn’t work,” but “something is rendering on top of the camera feed.” Either we failed to properly shut down the Compositor Layer used for Metal Rendering, or PolySpatial is rendering some large immersive space that is a single solid color. Note that on visionOS 2, passthrough video can be used with metal rendering, so just having a compositor layer active is not enough to block the camera feed.

I don’t think read-only textures and static batching are relevant here. If you turn off texture reads does it fix anything? Or are you just talking about the benefits of using Metal rendering for parts of your experience?

Forgive my confusion here–what unblocks you? Was it the removing additively loaded objects that you mentioned in your previous post? There’s definitely a bug to be solved here, and I’d love to learn more about how to replicate your issue so that we can solve it for you and any others who might run into the same kind of trouble.

I started this message by requesting a bug report but it bears repeating. Please share your project if you are able to do so. Even if you have been able to work around the issue, it’s a gnarled root waiting to trip up anyone else who walks the same path. We want to make sure PolySpatial is robust enough to survive anything users can throw at it! If you are unable to share your project due to IP ownership issues, I highly encourage you to take the time to create a fresh project with sample assets and trigger the same exception. Of course, please make sure you are doing so on the latest version of Unity and PolySpatial packages (and check that same upgrade on your project), to ensure that it’s not a bug that we’ve already fixed. Once we have a way to replicate the issue over here, we can surely find and fix the bug in the next release.

Thanks for reporting, and happy new year!