Has anyone managed to get SwiftUI windows working with the Metal rendering mode?
I used the SwiftUI sample and made it bare-bone so that it doesn’t depend on PolySpatial. A MonoBehaviour just tries to open the “HelloWorld” SwiftUI window group 2 seconds after the game starts.
This setup works with PolySpatial. But some issues appear when switching to the Metal rendering mode:
The injected scenes/view are assigned to the target UnityFramework instead of Unity-VisionOS
When the C# back-end invokes openSwiftUIWindow, an error pops which says “No scene found with id ‘HelloWorld’”. This happens even when changing the scene/view target back to Unity-VisionOS
If I copy the window and view code inside UnityCompositorSpace, then the window appears. But obviously at this point, maintenance is an issue
It would be really helpful to have this feature. SwiftUI is the only possible way to have gaze feedback with Metal!
Are you using the SwiftUI sample as a starting point? The PolySpatial visionOS build processor contains special logic for Swift plugin files named FooBarInjectedScene.swift (in other words, files that end in InjectedScene.Swift). Check out com.unity.polyspatial.visionos/Editor/VisionOSBuildProcessor.cs. Specifically, (on the latest version… your line numbers may vary), we check for those filenames on line 104, and we process the scenes on line 379. This logic doesn’t exist in the com.unity.xr.visionos processor.
Yeah, manually copying code around in the Xcode project isn’t sustainable. But, hopefully the PolySpatial build processor code is enought of a demonstration for what you might want to throw together for your own build processor. We have a couple of carveouts for RealityKit mode in the xr.visionos postprocessor code, and of course you’re able to make similar customizations. But there’s also nothing stopping you from just deleting UnityCompositorSpace.swift in your own build postprocessor code, and replacing it with your own version, or setting up whatever structure you want to your main app and custom Swift trampoline. Feel free to copy/paste some of the code from com.unity.polyspatial.visionos to get started.
com.unity.xr.visionos contains the “bootstrap” Swift code necessary to integrate with Unity, but both xr.visionos and polyspatial.visionos override the default trampoline with their own main App type. By removing main.mm, they establish a SwiftUI scene as being the starting point for the application. Without these packages, Unity builds an Xcode project very similar to what you get on iOS. Incidentally, this is what “Windowed Mode” in the XR Plug-in does. It just turns of the build postrpocessors and lets Unity build to a 2D window like it does by default without any packages installed. You’ll probably still want to keep the App Mode set to Metal (or if you’re feeling bold, go ahead and introduce your own app mode!) but write your own build IPostprocessBuildWithReport implementation with a higher callbackOrder so that you can do your own build automation.
And, of course, there are tons of other options for ways you can automate this process outside of Unity. At its core, a build postprocessor is just a script that runs after the build, giving you an opportunity to move files around and parse/rewrite the Xcode project definition. You could do something similar with a shell script, a small console program, some other build system like make, etc. It all depends on your use case and what else you might need to integrate into the final build. If you just want an extra SwiftUI window, a build postprocessor should do the trick.
Hi, I came across your post about issues with SwiftUI and Metal rendering in Unity. I was wondering if you’ve found a solution to these problems. I’m facing a similar challenge, and any insights you could share would be greatly appreciated.
Hi! Apologies for returning to this topic after so long.
@lgarvi I could hack a configuration that works up to a point, detailed below.
@mtschoen Thanks for your detailed answer. Indeed, I didn’t realize the PolySpatial build processor had that extra bit to handle SwiftUI scenes. I did some experiments to introduce SwiftUI into a Metal build, but then encountered issues related to the app lifecycle. May I ask some more questions here?
Here’s what I did:
Create a basic Unity visionOS project. I used the Metal sample from com.unity.xr.visionos.
Create a very basic SwiftUIDriver that opens a SwiftUI window on Start. Attach that script to an empty GameObject in the sample scene.
using System.Runtime.InteropServices;
using UnityEngine;
public class SwiftUIDriver : MonoBehaviour
{
void Start()
{
OpenSwiftUIWindow("HelloWorld");
}
#if UNITY_VISIONOS && !UNITY_EDITOR
[DllImport("__Internal")]
static extern void OpenSwiftUIWindow(string name);
#else
static void OpenSwiftUIWindow(string name) {}
#endif
}
Build the visionOS XCode project.
In the XCode project, create a Swift file inside target UnityFramework to support the c-declaration of the C# code above:
import Foundation
import SwiftUI
@_cdecl("OpenSwiftUIWindow")
func openSwiftUIWindow(_ cname: UnsafePointer<CChar>)
{
let openWindow = EnvironmentValues().openWindow
let name = String(cString: cname)
print("############ OPEN WINDOW \(name)")
openWindow(id: name)
}
Then in UnityMetalMainApp.swift, declare a simple HelloWorld window next to the compositor space:
var body: some Scene {
unityVisionOSCompositorSpace
WindowGroup(id: "HelloWorld") {
Text("Hello World!")
}
}
If we launch the app at this point, the HelloWorld window appears on top of the immersive space with the Unity Metal sample running. Neat. However, closing the window (either with the native cross button or programmatically) causes the immersive space to stop responding to Inputs. Console logs show that Unity was notified to enter background.
Just to mess around further, I went to UnityAppController.mm and erased the logic inside applicationDidEnterBackground and applicationWillResignActive.
With that rather brutal change, closing the SwiftUI window no longer causes the Unity game loop to pause. Yay. However, if we close and reopen the entire app, visionOS will then only open the SwiftUI window and not the immersive space with the Unity sample. A force quit is required to restart everything properly.
I’m assuming the last problem originates from visionOS itself. But is there some proper fix I could do at step 5? Any way I can skip applicationDidEnterBackround and/or applicationWillResignActive if the event was triggered by something else than the immersive space? Alternatively, can I make the entire app quit if the SwiftUI window is closed (and only then, not when the app is otherwise sent to background)?
IIRC those events are forwarded from the SwiftUI app container to Unity explicitly in UnitySwiftUISceneDelegate, and you can either delete/comment the lines there or try and detect your Window’s scene and early-out, so you only forward events from the immersive scene.
Something else to consider, if you find that you’re doing more surgery to the Swift/Objective-C trampoline than you’d like, is Hybrid mode. That would let you use PolySpatial’s build processor and window logic alongside an immersive space with Metal rendering. You’d pay a little extra CPU overhead for PolySpatial, but depending on your use-case that might be fine.
Hybrid mode also gives you the ability to manually re-start the Metal immersive space by cycling enabled on whatever VolumeCamera is set to Metal mode. You should be able to re-start the metal immersive space in regular Metal mode by calling openImmersiveSpace("CompositorSpace") somewhere in Swift code, but it might be more trouble than it’s worth to hunt down all those edge cases.
Another advantage to the Hybrid approach is that it falls more squarely within the “Unity supported” realm where we’re on the hook to fix any bugs you encounter with our built-in systems and solutions. I’m not able to provide official support for Swift code you add to the Xcode project after it comes out of Unity. We support your ability to do this kind of thing (a.k.a. Unity as a Library) but we don’t support specifically what you do once the app leaves Unity.
