Fully immersive app - camera control in simulator not working

Hi, I don’t have much experience with AR/VR, let alone VisionOS, but I’m trying to get a basic sample working where you can just look around in the simulator in a fully immersed app. (Mixed reality apps are working)

I’ve tried doing this through creating a new project using the VR template and then adding the `com.unity.xr.visionos’ package and setting ‘App Mode’ to ‘Virtual Reality - Fully Immersive Space’ and setting the ‘Device Target’ to ‘Simulator’. But this results in a static view of the scene which does not change when looking around in the simulator (except for the loading window that popups over it that does move around):

I also tried from the visionOS template 0.2.2 with the SampleSceneUnbounded scene and with different settings but getting the same results as above.

Am I missing something in the input settings? Does it need a custom path for tracking head position/rotation in VisionOS?

Can you check that your camera or its parent has a Tracked Pose Driver component added to it?

Hey, sorry for the slow reply.

Still no luck.
I’m using an empty scene (standard URP scene template) and added a cube. Then added the XR/VR Origin (VR), which has the Tracked Pose Driver component on the camera. I’ve even tried adding centerEyePosition [XR Simulated HMD] for the Position Input, but that also didn’t help.

I’m now using Unity 2022.3.9f1 and Apple visionOS XR Plugin 0.3.1 and building a visionOS app with the Device Target set to Simulator.
And since the target is set to fully immersive, I’ve disabled the PolySpatial Runtime 0.3.2 (but leaving it enabled makes no difference).

Editor:

image

Simulator:


Rotating the camera in the Simulator moves the ‘loading’ window around in space as expected, but the rest (cube) remains static.

Xcode log:

[UnityMemory] Configuration Parameters - Can be set up in boot.config
    "memorysetup-allocator-temp-initial-block-size-main=262144"
    "memorysetup-allocator-temp-initial-block-size-worker=262144"
    "memorysetup-bucket-allocator-granularity=16"
    "memorysetup-bucket-allocator-bucket-count=8"
    "memorysetup-bucket-allocator-block-size=4194304"
    "memorysetup-bucket-allocator-block-count=1"
    "memorysetup-main-allocator-block-size=16777216"
    "memorysetup-thread-allocator-block-size=16777216"
    "memorysetup-gfx-main-allocator-block-size=16777216"
    "memorysetup-gfx-thread-allocator-block-size=16777216"
    "memorysetup-cache-allocator-block-size=4194304"
    "memorysetup-typetree-allocator-block-size=2097152"
    "memorysetup-profiler-bucket-allocator-granularity=16"
    "memorysetup-profiler-bucket-allocator-bucket-count=8"
    "memorysetup-profiler-bucket-allocator-block-size=4194304"
    "memorysetup-profiler-bucket-allocator-block-count=1"
    "memorysetup-profiler-allocator-block-size=16777216"
    "memorysetup-profiler-editor-allocator-block-size=1048576"
    "memorysetup-temp-allocator-size-main=4194304"
    "memorysetup-job-temp-allocator-block-size=2097152"
    "memorysetup-job-temp-allocator-block-size-background=1048576"
    "memorysetup-job-temp-allocator-reduction-small-platforms=262144"
    "memorysetup-temp-allocator-size-audio-worker=65536"
    "memorysetup-temp-allocator-size-preload-manager=262144"
    "memorysetup-temp-allocator-size-background-worker=32768"
    "memorysetup-temp-allocator-size-job-worker=262144"
    "memorysetup-temp-allocator-size-gfx=262144"
    "memorysetup-temp-allocator-size-cloud-worker=32768"
    "memorysetup-temp-allocator-size-nav-mesh-worker=65536"
Built from '2022.3/staging' branch, Version '2022.3.9f1 (ea401c316338)', Build type 'Release', Scripting Backend 'il2cpp'
MemoryManager: Using 'Default' Allocator.
-> applicationDidFinishLaunching()
[Physics::Module] Initialized MultithreadedJobDispatcher with 9 workers.
[Subsystems] Discovering subsystems at path /Users/stijn/Library/Developer/CoreSimulator/Devices/A633CE5D-F472-46C4-9C94-B45F9C6574A5/data/Containers/Bundle/Application/6E994CC1-B1CE-4A74-8D96-886A2556C803/PolySpatial Template.app/Data/UnitySubsystems
[Subsystems] No descriptors matched for  examples in UnitySubsystems/UnityVisionOS/UnitySubsystemsManifest.json.
[Subsystems] 1 'inputs' descriptors matched in UnitySubsystems/UnityVisionOS/UnitySubsystemsManifest.json
[Subsystems] 1 'displays' descriptors matched in UnitySubsystems/UnityVisionOS/UnitySubsystemsManifest.json
[Subsystems] 1 'meshings' descriptors matched in UnitySubsystems/UnityVisionOS/UnitySubsystemsManifest.json
GfxDevice: creating device client; threaded=1; jobified=0
Initializing Metal device caps: Apple xrOS simulator GPU
Initialize engine version: 2022.3.9f1 (ea401c316338)
AddInstanceForFactory: No factory registered for id <CFUUID 0x600000248e80> F8BB1C28-BAE8-11D6-9C31-00039315CD46
[0x104396c50|InputElement #0|Initialize] Number of channels = 0 in AudioChannelLayout does not match number of channels = 2 in stream format.
<0x600002612e20>: handleMessage:withType: - invalid sequenceNumber: 0 for orientation change.
The referenced script (Unknown) on this Behaviour is missing!
The referenced script on this Behaviour (Game Object '<null>') is missing!
WARNING: RGBA Compressed ASTC6X6 UNorm format is not supported, decompressing texture
XRGeneral Settings awakening...
UnityEngine.XR.Management.XRGeneralSettings:Awake()

[Subsystems] Loading plugin UnityVisionOS for subsystem VisionOS-Display...
[Subsystems] VisionOS XR Plugin successfully registered Provider for VisionOS-Display
[Subsystems] VisionOS XR Plugin successfully registered Provider for VisionOS-Input
[Subsystems] VisionOS XR Plugin successfully registered Provider for VisionOS-Display
[Subsystems] VisionOS XR Plugin successfully registered Provider for VisionOS-Input
[Subsystems] VisionOS XR Plugin successfully registered Provider for VisionOS-Display
[Subsystems] VisionOS XR Plugin successfully registered Provider for VisionOS-Meshing
[Subsystems] Loading plugin UnityVisionOS for subsystem VisionOS-Input...
[Subsystems] Loading plugin UnityVisionOS for subsystem VisionOS-Meshing...
New input system (experimental) initialized
[XR] Display Start
UnloadTime: 0.841000 ms
####### EnablePolySpatialRuntime is false, not enabling!
No active UnityEngine.XR.ARSubsystems.XRRaycastSubsystem is available. Please ensure that a valid loader configuration exists in the XR project settings.
UnityEngine.XR.ARFoundation.SubsystemLifecycleManager`3:GetActiveSubsystemInstance()
UnityEngine.XR.ARFoundation.SubsystemLifecycleManager`3:OnEnable()
UnityEngine.XR.ARFoundation.ARTrackableManager`5:OnEnable()
UnityEngine.GameObject:AddComponent()
UnityEngine.XR.Interaction.Toolkit.XRRayInteractor:FindCreateARRaycastManager()

-> applicationWillEnterForeground()
-> applicationDidBecomeActive()
100.103 HALC_ProxyIOContext.cpp:1.328 HALC_ProxyIOContext::IOWorkLoop: skipping cycle due to overload
100.103 HALC_ProxyIOContext.cpp:1.328 HALC_ProxyIOContext::IOWorkLoop: skipping cycle due to overload
nw_socket_copy_info [C1:2] getsockopt TCP_INFO failed 102
nw_socket_copy_info getsockopt TCP_INFO failed 102
100.103 HALC_ProxyIOContext.cpp:1.328 HALC_ProxyIOContext::IOWorkLoop: skipping cycle due to overload
100.103 HALC_ProxyIOContext.cpp:1.328 HALC_ProxyIOContext::IOWorkLoop: skipping cycle due to overload
RequestCreateTexture
SetupRenderTextureFromXRRequest (id: 1 col: 0x112bf1c40 d: 0x112b7fd10 sr: 0x0)
RenderTexture::Create (id: 1)
IVRDeviceCallback_CreateEyeTextureResources 0x112bf1c40
IVRDeviceCallback_CreateEyeTextureResources 0x112b7fd10
Presenting a drawable without a device anchor. This drawable won't be presented.
RequestCreateTexture
SetupRenderTextureFromXRRequest (id: 2 col: 0x1159bd140 d: 0x1043bf0f0 sr: 0x0)
RenderTexture::Create (id: 2)
IVRDeviceCallback_CreateEyeTextureResources 0x1159bd140
IVRDeviceCallback_CreateEyeTextureResources 0x1043bf0f0
Presenting a drawable without a device anchor. This drawable won't be presented.
RequestCreateTexture
SetupRenderTextureFromXRRequest (id: 3 col: 0x112de52c0 d: 0x112dccb90 sr: 0x0)
RenderTexture::Create (id: 3)
IVRDeviceCallback_CreateEyeTextureResources 0x112de52c0
IVRDeviceCallback_CreateEyeTextureResources 0x112dccb90
Presenting a drawable without a device anchor. This drawable won't be presented.
Presenting a drawable without a device anchor. This drawable won't be presented.
Presenting a drawable without a device anchor. This drawable won't be presented.
Presenting a drawable without a device anchor. This drawable won't be presented.
Presenting a drawable without a device anchor. This drawable won't be presented.
Presenting a drawable without a device anchor. This drawable won't be presented.
Presenting a drawable without a device anchor. This drawable won't be presented.
Presenting a drawable without a device anchor. This drawable won't be presented.
Presenting a drawable without a device anchor. This drawable won't be presented.
Presenting a drawable without a device anchor. This drawable won't be presented.
Presenting a drawable without a device anchor. This drawable won't be presented.
Presenting a drawable without a device anchor. This drawable won't be presented.
Presenting a drawable without a device anchor. This drawable won't be presented.
Presenting a drawable without a device anchor. This drawable won't be presented.
Presenting a drawable without a device anchor. This drawable won't be presented.
Presenting a drawable without a device anchor. This drawable won't be presented.
Presenting a drawable without a device anchor. This drawable won't be presented.
Presenting a drawable without a device anchor. This drawable won't be presented.
Presenting a drawable without a device anchor. This drawable won't be presented.
Presenting a drawable without a device anchor. This drawable won't be presented.
Presenting a drawable without a device anchor. This drawable won't be presented.
Presenting a drawable without a device anchor. This drawable won't be presented.
Presenting a drawable without a device anchor. This drawable won't be presented.
Presenting a drawable without a device anchor. This drawable won't be presented.
Presenting a drawable without a device anchor. This drawable won't be presented.
Presenting a drawable without a device anchor. This drawable won't be presented.
Presenting a drawable without a device anchor. This drawable won't be presented.
Presenting a drawable without a device anchor. This drawable won't be presented.
Presenting a drawable without a device anchor. This drawable won't be presented.
Presenting a drawable without a device anchor. This drawable won't be presented.
Presenting a drawable without a device anchor. This drawable won't be presented.

It looks like your cube is not a child of the camera object. The tracked pose driver on the camera is what updates the transform of the camera object to be the device / headset position. If you make the cube a child of the Main Camera object and move around the simulator does it stay locked to the camera?

We have a similar question/experience with the same setup and versions using the VR Template project.

In fully immersive space on the visionOS simulator moving the loading window with the drag bar (or WASD/arrowkeys) does not move the orange cube around the scene even if attached to the Main Camera under XR Origin.

What is the recommended way to simulate camera control in the visionOS simulator?




The cube shouldn’t be a child of the camera object (unless you want the cube to stay static in front of the camera wherever you look). But I tried your suggestion and the result is the same. The same result as NicoleLazzaro posted.

1 Like

Just to be clear, made a gif of it, cube remains static (whether parented to camera or unparented), same as skybox. Left the ‘loading’ window open, so you can see the camera is being rotated.
immersive_static

1 Like

I was confused about the issue but was able to test a few more things this morning and figured out what was wrong. Device position is part of ARKit so you need to add an AR Session to your scene (Create > XR > AR Session) currently that is required to get the data for the device (HMD) position. The input bindings for the camera tracked pose driver should reference devicePosition [Handheld AR Device] and deviceRotation [Handheld AR Device].

2 Likes

Ah this works, thanks! Wasn’t expecting an AR Session was needed for VR.

2 Likes

I had that same thought, but it does make sense after thinking about it since all of the AVP data for the head and hands comes through ARKit on the AVP side.

With all that said, we can certainly improve our documentation, samples, and workflow to make this clearer. We still need to do a pass on the visionOS XR plugin to add Project Validation rules, and I think the AR Session requirement is a prime candidate for that. For now, though… spread the word! :slight_smile:

2 Likes

Hi

I’m trying to render object always in front of camera in Mixed Reality, but it doesn’t appear on device, I have ARSession added to scene and I’m using XR Rig with a camera under. Have you idea what i might doing wrong?

I’ve had the same problem here. Where does ARSession go in the hierarchy? I’ve tried it as a parent to the camera, as a child to the camera and as a sibling. Plus, I am running 2022.3.12f1 and see no reference to an “[Handheld AR Device]” as a source for the rotation. Only Touchscreen, Head/Eye tracking and action maps.

Do you have the Apple visionOS loader checked in XR Management Settings?

It doesn’t matter. You just need an ARSession component on any active GameObject, and it needs to be enabled. You should see some messages in the log about starting data providers based on your scene setup.

What version of com.unity.xr.visionos are you using? We updated the minimum supported Unity version to 2022.3.15f1 in the 0.7.1 version of that package, although earlier versions should still provide head pose through that input device. If you are using the legacy TrackedPoseDriver from com.unity.xr.legacyinputhelpers, you won’t be able to map actions to that device, but the bindings for Generic XR Device and Center Eye - HMD Reference should work.

Please take a look at the samples in com.unity.xr.visionos for a known working camera rig setup, and compare it with what you have in your scene. The Project Validation feature in Project Settings is also a good place to check if you are having trouble.