[VisionOS] Choosing Between Metal and Reality Kit App Modes for Vision Pro

Unity Engine
, , , , , ,
1

I can’t seem to find much documentation laying out the pros and cons of each option, or what goes into making the decision between selecting Metal or RealityKit App Modes. I’d be much obliged if someone could help me understand!

From what I can tell, choosing RealityKit gives you the ability to make Bounded apps. You can also access the Crown for Progressive Immersion, and your hands/arms can be occluded by virtual objects. You can also use Play-to-Device in this mode.

But with all the benefits of RealityKit mode, why choose Metal? At first I assumed it would unlock shader possibilities currently restricted by RealityKit (e.g., only Shader Graph shaders are supported), but looking at the documentation, it looks like the same restrictions apply. Am I missing something?

I’m on PolySpatial 2.0.4 by the way.

Edit: This page talks about it a little, but I’m still confused.

The description for Metal Rendering with Compositor Services:

“…allows apps to create content using metal-based graphics features supported by Unity. Rendering and simulation are entirely managed by Unity, resulting in low overhead. PolySpatial and RealityKit will not be used in applications set to this mode. This mode allows the quickest path to port existing XR applications to visionOS.”

Does this mean we don’t have shader limitations if we go with Metal? And what do we lose by not having PolySpatial and RealityKit?

2

Hey there! Hopefully we can help clear some things up.

Yup. Sounds good so far… Although you can control upper limb visibility in both modes.

Yup! You had it right the first time. :slight_smile: Metal mode does not limit you to ShaderGraph shaders. You can also do post processing effects and draw a skybox or solid color background. Essentially, it’s just regular Unity graphics, where you can draw whatever you want to the full screen (or “framebuffer” if we’re getting technical). In Metal mode, you don’t have any restrictions on the kinds of features you can use, as long as they work on other Metal platforms like iOS and macOS. Of course, there are some platform-specific capabilities and constraints like differences between ARKit on iOS and visionOS, but you don’t have to use PolySpatial to create RealityKit entities in order to get graphics on screen.

Could you please let us know where the documentation implies that shader restrictions apply to Metal mode? One thing that has changed since visionOS 1 is that Compositor Services (Metal) apps weren’t able to utilize pass-through video. In other words, on visionOS 1, you could only use Metal for “VR” apps that didn’t give users a view into the real world. At the time, this was a major differentiator for PolySpatial/RealityKit, and it’s possible that we still have some language in our docs talking abou this kind of limitation for Metal apps.

Yup! No shader limitations with Metal. What do you lose? You can’t use bounded volumes or have your app exist in the “shared space” alongside other apps like Safari. Apps using Metal Rendering with Compositor Services always use an ImmersiveSpace, so they will fade away when the user presses the home button or switches to another app (like taking a FaceTime call).

Using PolySpatial/RealityKit, you can still do something like put a giant sphere in the scene and use an Unbounded VolumeCamera to block out passthrough video and achieve a similar “fully immersive” effect, but you can’t write shaders that write arbitrary colors to the screen. And you can call up or dismiss this immersive space alongside content in volumes, which allows your app to sometimes exist alongside other apps, and sometimes “go immersive”, like a desktop app entering an exclusive fullscreen mode. And, of course, the actual rendering quality of RealityKit may be something you want to leverage, whether it’s because you want your app to have the same look and feel as other apps rendered with RealityKit, or you want to use the higher-quality reflection probes, which aren’t available to Metal apps. I’m not sure if we have a full breakdown of Metal vs. RealityKit from a rendering perspective, but it might make for an interesting blog post some day :slight_smile:

Now… I don’t want to confuse you even further, but I should also mention Hybrid Mode, which is a way of combining the two modes to have the best of both worlds. You can pull up a compositor layer which allows you to take full advantage of Unity rendering features while also putting content in volumes and using RealityKit rendering. It comes at a performance cost, but if you want the ultimate flexibitlity of using all of Unity’s render features as well as RealityKit, it’s worth giving Hybrid Mode a try.

I hope this helps! Let me know if you have any more questions, and good luck!

3

Hey, thanks for the help! This is a really good breakdown and it’s going to be super helpful for determining which app we go with.

To answer your question about documentation:

Could you please let us know where the documentation implies that shader restrictions apply to Metal mode?

Here’s one document that could use some extra clarification, though there may be others. One thing I find confusing is that visIonOS support is all categorized under PolySpatial, even though the Metal App Mode doesn’t technically use PolySpatial (I think?). If it’s a choice between ‘RealityKit and PolySpatial’ and ‘Metal and Compositor Services,’ then why is everything categorized under PolySpatial in the documentation?

Another question I have is whether one mode or the other is slated as the more long-term, “future proof” option? In other words, is Metal Mode intended to be a temporary solution to make porting easier while RealityKit support is further developed? Or is Metal Mode considered to be the ‘future’, with more and more support added to it. Or are they both considered to be equally supported for the forseeable future? (I know you don’t have a crystal ball and may not be able to share roadmap info, but if you can illuminate this at all it would be helpful!)

Edit:

In the document I linked above, it would be great to know which limitations apply to Metal (if any). For example, there are a lot of Particle System limitations – do those apply to Metal as well?

4

A lot of this depends on Apple and what they choose to support at a lower level in future versions (for instance, they didn’t support Metal passthrough in visionOS 1, but added support for it in visionOS 2, making Metal usable for AR apps). Generally speaking, though, it’s probably fair to say that the Metal path is more “future proof” from Unity’s perspective because it relies on (and supports) all of Unity’s existing functionality, which is battle-tested across multiple platforms. PolySpatial/RealityKit is more of the special case (not necessarily a temporary solution, though, unless Apple provides access to everything currently exclusive to RealityKit).

However, if you limit yourself to the subset of features supported in PolySpatial, then it should be relatively easy to switch between the two modes.

No, and that document is definitely specific to PolySpatial. Metal mode uses the standard Unity particle system engine.

One feature of RealityKit/PolySpatial that I didn’t see mentioned in this thread is the gaze hover behavior (and, related to that, gaze-based foveated rendering). Apple doesn’t provide CPU access to the gaze vectors, so this feature can’t currently be supported in Metal mode.

5

Super helpful, thank you!

6

Is my understanding correct? （PolySpatial 2.1.2）
If I want to make an app, I can choose the technical route as follows.

vr → Metal
mr → RealityKit
Just the content you make →Metal
Your content + the real world →RealityKit

The lit material rendering result is the same as the editor →Metal
post processing effects →Metal
visionOs light effect →RealityKit
world track →RealityKit
eye track →RealityKit
hand track →RealityKit
image track →RealityKit
bounded volumes →RealityKit
Unbounded VolumeCamera →RealityKit

I want them all, and they can be split into unrelated scenes → Hybrid Mode

7

That’s right; we changed the terminology in PolySpatial 2.X/Unity 6000 because Metal mode (previously “VR”) added support for passthrough.

As of PolySpatial 2.X/Unity 6000/visionOS 2, Metal mode can show the real world (passthrough) as well.

Yes, Metal mode uses the usual Unity shaders, so it should look exactly as it does in the editor (versus our having to convert to MateralX/use RealityKit lit shaders in RealityKit), and allows post-processing effects.

Assuming you mean the image-based lighting, yes, only RealityKit mode can really show that. There is a way to get environment probes in Metal mode that you can use for image-based lighting, but my understanding is that these are very low resolution on the Apple Vision Pro.

These should all work in either Metal or RealityKit mode.

This is correct; only RealityKit can use eye tracking (and not in a way that allows you to get the eye tracking information on the CPU: RealityKit uses it internally for gaze-based foveation and hover effects).

Correct; Metal mode doesn’t use volume cameras. Instead, it uses the standard Unity Cameras (typically with an XR rig that tracks the head/device pose).

I believe Hybrid mode actually allows you to switch within a scene (using volume cameras) or even include bounded content on top of Metal content.

8

Thank you very much for your detailed analysis! Your explanation has made it much clearer and faster for me to choose the right technical solution for my project.

Can I understand it this way:

RealityKit and Metal only provide high-level rendering and interaction features, so switching between these two modes will only affect related aspects (such as physical materials, animations, particles).

ARKit, which operates at a lower level, supports both modes, meaning features like spatial awareness, world aranchor, motion tracking, environmental lighting estimation, plane detection, image tracking, and object tracking are not impacted by the rendering mode. Unity integrates ARKit functionalities through the AR Foundation framework.

As for eye tracking being unavailable, it’s simply because Apple hasn’t opened up its API to ARKit, so Unity can’t integrate it through Metal.

Is there any part that I missed or misunderstood?

9

Sounds about right to me!

10

New project encounters new confusion :sweat_smile:
Does “Metal mode + passthrough” not support “post-processing effects”? For example, bloom effect?

unity60000.37f1
“com.unity.xr.visionos”: “2.3.1”,
“com.unity.xr.arfoundation”: “6.1.1”

Am I missing any setting?

image
image1608×1733 237 KB

image
image1624×1622 459 KB

11

It does support post-processing effects, AFAIK. In the Unity version that you’re using, you may need to disable foveated rendering (see this thread).

12

Thank you for your response; disabling that option brought the post-processing into effect.