Best approach to re-use UI between iOS and visionOS?

I’m porting an existing iOS AR app to visionOS. The app has a couple of screen-space UI dialogs, mainly used for onboarding purposes.

I know that screen-space UI is not supported (well, it kinda works, but looks pretty weird and huge), so I want to convert it into world-space UI on visionOS. However, to save double-work, I’d prefer to share as much of the UI and logic behind the dialogs between platforms.

What’s the best approach for doing this? Putting the UI elements in prefabs and re-using them on different canvases?

Another approach I’m exploring is to re-build these dialogs natively, which would yield the advantage of having them look awesome (Unity’s UX looks pretty weird on visionOS, IMHO). Has anyone already tried out to use regular SwiftUI views in a Unity-exported app and communicate with them (C# to Swift and back)?

I managed to make my dialogs show up in world-space. However, they are behaving pretty weird. Although I transformed all texts into TextMesh Pro, some of them just do not show up, even if the prefab containing them is just the same and shows up elsewhere. Same goes for images which tend to not show up. And no input clicks are detected on the TextMesh Input fields (I set the world camera programmatically). Is there an in-depth explanation or a sample somewhere how to use UI correctly on visionOS? I’m pretty stuck.

Edit: I just noticed that this problem only exists if a game object containing a UI screen is activated at runtime. If the screen is visible on start, all items show up. They are not interactable, though. Just the buttons highlight on hover.

Edit 2: Ok. Not always. Some dialogs work if they are active on startup. Others not.

Referencing this issue How to use Localization package on visionOS? - #17 by mtschoen, where we had a related discussion.

Summary of the other thread: The dialogs now show up correctly, but they are not responsive (neither in the simulator, nor in the Editor). Might be related to PolySpatial disabling InputSystemUIInputModules events in favor of SpatialPointerDevice, which I am not using yet.

Still a bit unclear to me what the best setup would be for a project that shall support both iOS (plus Android) and visionOS.

It would be nice if there was some documentation on the best-practices for UI on visionOS.

I’d also love to see a way to use SwiftUI in parallel, so I could implement parts of the UI in Shared Space, using Apple’s tech, and then switch to Unity UI in Immersive Space. However, I would not want to have separate projects (e.g. in Xcode and Unity) for this. It would be awesome to be able to inject the Swift files during Unity’s build process, like I already do for .mm and .h files in my iOS export.

1 Like

I tried to add the SpatialPointerDevice to my project now, but could not find it? There is no such component. I have both PolySpatial and the Input System imported.

In the Mixed Reality sample scene, I can see that a click on the “3DButton” under “Home Button Text and Icon” is recognized by some code, but I don’t think this is really the (only) way to do this? I would not want to setup a separate script for each UI element of my 2D dialogs.

In the PolySpatial docs, there is also no information about how to setup this SpatialPointerDevice. Instead, it states:

“Existing actions bound to a touchscreen device should work for 2D input.”

To me this sounds as if things “just work” out-of-the-box for 2D interfaces.

SpatialPointerDevice isn’t a MonoBehavior, it’s an InputSystem device which exposes controls to action maps for the gaze/pinch interaction. I’ll go into more detail below, but you shouldn’t have to use this if you don’t need the extra information it provides.

That is correct. We have some test scenes that we use internally to validate UI functionality, and I just double-checked that they use the standard input actions for InputSystemUIInputModule. The way this works is that we process gaze/pinch events both to SpatialPointerDevice and a normal TouchScreen, using the main camera or a “backing camera” for the VolumeCamera to come up with an equivalent screen position for the input. Basically it reverse-engineers where a user might have touched a touch screen to get a raycast hit at the interaction location we get from visionOS. This should send input events through a normally-configured InputSystemUIInputModule.

No, SpatialUIInputManager is just an alternative approach. As you say, normal UI input should work. I’ve made a note for us to add a sample scene to the PolySpatial package samples that uses InputSystemUIInputModule which should help clarify some of this stuff in the future.

Yep! Handling UI input on VR is handled differently (a sample is on the way), but for MR/PolySpatial, you should be able to use an input module and event system with the default configuration. It is important, though, that your UI is visible by the main camera in order for the events to come through.

That would be nice. :slight_smile: This is a pretty common request but it would be a big lift on our end. We actually have this listed on our roadmap as a feature we are considering. Please give it a vote if you haven’t already.

You technically wouldn’t need separate Xcode projects if you used a build postprocessor to just dump/modify swift code in the generated Unity Xcode project, but developing the swift stuff in a separate project and building it out as a library is probably a better approach.

Yeah, I think in principle this should be as easy as copying the .mm and .h over to the Xcode project. I haven’t looked into the details, though. This would be a feature of the Unity plugin importer, which is outside of the scope of our team, but certainly a good feature request! I’ll see where I can direct this feedback. :thinking:

2 Likes

Thanks for your fast and detailed response, highly appreciated.

Maybe this is the issue in my case. In our existing iOS app (which I am porting to visionOS), we use a separate “Startup Camera” which renders the onboarding and login dialog screens. Then, after login is complete, we switch to our “AR camera” (as part of activating AR in general at the same time) and disable the “Startup Camera”. Maybe PolySpatial picks up the wrong camera, although only one is active at a time?

I also set the “Event Camera” for the world-space canvas to the “Startup Camera”, as the hint in the canvas suggested. I even implemented a camera switch when we switch over to the AR Camera. But I can’t even get that far, because the login dialogs are not working, so I cannot login first.

BTW: I also double-checked that the “Startup Camera” can see my UI. I even moved it a bit, so the UI is visible in full in the Editor preview.

And, as I said: the UI doesn’t work in Editor as well, not just in the Simulator. Both do not recognize any UI elements.

That would be amazing. I actually checked the other samples now (especially the “Project Launcher” where there is some UI elements) - just to learn that it uses the programmatic approach as well. And it only shows it for buttons, not for input fields (although I assume this would work the same way).

Yes, that would be even better. If the library could be bundled with the Unity project, so it would be auto-included in the exported Xcode project. We automated the build process as much as possible, using custom build scripts, since I want to avoid any manual post-export setup in Xcode.

I left a comment on the roadmap.

1 Like

Another comment on the Editor part: If I run the app in the Editor, it still uses the mobile part of my user-interface, i.e. a “Screen-Space Overlay” canvas.

Actually, I have two nodes below my central “Dialogs” tree node. One is for mobile, one is for headset. Both host the same dialog prefabs, the only difference between these nodes is the canvas setup on the respective “mobile” or “headset” node.

Is this maybe the reason why the dialogs don’t work in the Editor, because PolySpatial does not support screen-overlay canvases, even in the Editor?

It would be awesome to have an official “multi-platform” sample project that shows how to setup things if you want to use the same code-base for mobile and visionOS, including playability in the Editor.

Here’s a screenshot of my setup, for clarification:

On startup, the whole AR node is inactive. It includes the AR session etc. Only the dialogs including the login dialogs are active, and rendering is done using the Startup Camera. This is to avoid unnecessary device load with an active AR camera on mobile devices, while AR is not needed yet.

Once the user is logged in, the AR node becomes active, and the AR Camera takes over. Maybe this approach does not work well on visionOS and the whole AR subsystem has to be active all the time? But this won’t explain why things don’t work in the Editor.

I tried to replicate things in the visionOS sample project, using the Mixed Reality scene.

I added a canvas and a button on it, as well as a script that reports a button click:

Interestingly, this setup works in the Editor, but not in the simulator.

I also tried to set the event camera of the canvas to the main camera of the scene, and I also tried out to set the canvas layer to “PolySpatial UI”. None of these changes yielded any results.

Is there anything else what I have do to make the buttons work?!

Alright. The reason why the button press did not work in the sample scene was that the sample scene still used the “old input system”. After I changed this, it works.

Is this maybe the reason why the dialogs don’t work in the Editor, because PolySpatial does not support screen-overlay canvases, even in the Editor?

Yes, like all XR systems there is no support for screen overlay at all.

This cannot be. We’re using screen overlays in our iOS and Android AR apps for years now. Works perfectly. Or do you mean VR?

Somehow magically, my screens work in iOS Simulator now. I do not really know what happened. They still won’t work in the Editor, though.

Also: I noticed that no keyboard shows up for input fields. How can I enter text into these fields?

Yes, sorry, VR.

1 Like

From a question in your prior thread:
I said:

and you asked:

I was a bit unclear with my explanation. UnityPlayModeInput will override the input actions asset only in play mode, which means it has no effect in iOS (or visionOS) builds. It is simply there to avoid false positives, where your UI works because it’s getting mouse input, but it won’t work on visionOS with gaze/pinch input.

If you would like to test your project without UnityPlayModeInput in the loop, you can uncheck Enable PolySpatial Runtime under Project Settings > PolySpatial. That will run your scene in play mode just as it is on iOS or any platform other than visionOS.

None of this will help you fix your UI issues in simulator/device builds, but I wanted to answer this dangling question. :slight_smile:

This is a known issue and will be fixed in the next release.

Ok, thanks for the clarification. So there is no way to keep the PolySpatial runtime on, but still test things in the Editor’s Play mode?

My problems in the simulator vanished somehow, magically. Without even having changed anything (according to git). I can now interact with the UI. But I can’t type. Glad to hear that this is already being addressed.

do you mean it works in play to device editor (?)

i cannot get input field to work in either editor/play to device or simulator

No, it works in the simulator now, but not in Unity Editor. If I understand @mtschoen, this is intentional, to avoid “false positive” events that would not happen in the simulator or a real device.

If you want to use the dialogs in the Editor, you have to disable PolySpatial altogether. I guess this would also disable Play to Device. I am not using this anyways, since I learned that it is basically “streaming” events from the simulator to the Editor, but it is not really behaving like if the app would run on the simulator directly.

To be honest, I have no real clue why it worked in the simulator all of the sudden. I had tried a lot of things, and I could not see any difference in my git repository when it suddenly worked. Before, I had also set the world camera for the events, but this did not change anything. I even tried to orient this camera in the Editor, so it looked as if it would “see” the canvas.

It’s all pretty confusing.

have you been able to get just a regular UI Input Field to work (not a editor dialogue, runtime UGUI or TMPro)

No, UI Input Fields do not work. I had these in my app before and had to switch everything (including regular UI texts) to Text Mesh Pro. If you use UI texts, they show up as some random illegible text blocks.