Standalone Holographic Remoting to Hololens 2 with XR SDK Package

[Ozelotl]

Hello,

I want to create a standalone UWP application that can connect to the Hololens 2 Holographic Remoting Player.

I am using Unity 2019.4.5f1, XR Plugin Management Package 3.2.16, Windows XR Plugin 2.4.1 and Holographic Remoting Player v2.2.3.0.

I can use the Window>XR>Windows XR Plugin Remoting functionality as intended (connect, then press play, then see the hologram on my Hololens). I can also execute the code from the scripting api or the same call the Windows XR Plugin Remoting window uses. When connecting like this before entering Playmode, everything works as if I had used the Windows XR Plugin Remoting window. When executing the code during Playmode, a connection is established but the screen in the Hololens stays black.

Additionally, I have found no way to make even the black screen connection work in a standalone UWP desktop application.
If I check “Windows Mixed Reality” in the XR Plug-in Management settings for UWP, my built application will ask me to connect a MR device when starting.
If I uncheck it, I can start the application but will get an error upon trying to execute the connection code, as XR dlls are not loaded.

Unfortunately, I can only find tutorials for getting Holographic Remoting to work with Legacy XR (by including the “none” sdk and then setting “XRSettings.enabled = true;”). I would love to be able to use Holographic Remoting in a standalone application with the new XR Pipeline, or at least know if this is currently not possible.

Thanks in advance for your help and best regards,
Stella

[JimboJonesBL]

Would love to know the exact same thing. I’m using Unity 2019.4.13, Windows XR Plugin 2.4.1, XR Plugin Management 3.2.17, Holographic Remoting Player v24.0.0. Holographic Remoting via Editor Playmode works flawless. But when building a standalone UWP application, the application asks to connect a MR device just like @Ozelotl .

Settings in Unity:
Edit->Project Settings-> XR Plug-in Management
PC, Standalone:

• Enable Initialize XR on Startup
• Enable Windows Mixed Reality
  UWP:
• Enable Initialize XR on Startup
• Enable Windows Mixed Reality

Edit->Project Settings-> XR Plug-in Management->Windows Mixed Reality
PC, Standalone:
-Enable Shared Depth Buffer (24bit)
UWP:
-Enable Holographic Remoting
-Enable Shared Depth Buffer (24bit)

Edit->Project Settings ->Player → Publishing Settings under Capabilities additionally

• Enable InternetClientServer
• Enable PrivateNetworkClientServer

Add the holographic Remoting prefab from the MRTK.Tutorials.PCHolographicRemoting.unitypackage from Erstellen einer Holographic Remoting-App zur Visualisierung von 3D-Inhalten in HoloLens 2 - Training | Microsoft Learn

Settings in Visual Studio:
-Release
-x64
-LocalMachine

When hitting LocalMachine build while having the Holographic Remoting Player opened, following Error appears:
“Cant open app
To use this app, youll need to set up Mixed Reality. Go to Start > Mixed Reality Portal or plug in a compatible headset to get started.” I don’t even get to the point where I can enter an IP.

PS: I can access Windows Device Portal via WiFi

Is it possible to have a standalone application with the XRSDK setup? I would really appreciate an answer, as there are no real tutorials or information on this topic.

[joejo]

Would love to know the exact same thing. I’m using Unity 2019.4.13, Windows XR Plugin 2.4.1, XR Plugin Management 3.2.17,

I know it’s tangential to the root issue, but 3.2.17 should be used with 2019.4.15f1 or later. XR Plug-In Management 3.2.17 - Unity Forum

[JasonCostanza]

Hello there,

Can you try changing your settings to match the following and let me know if it works better, differently, worse?

The reason it’s hitting the Headset plug in step is because “Initialize on startup” may be active, if I’m following your descriptions correctly. That will cause the app to look for any valid VR display at launch, but we’re not ready to give it one yet so we need to disable that. The loader MUST be enabled because otherwise our remoting API’s won’t be accessible, as Ozelotl mentioned seeing in their post about the dlls not loaded.

If you can try the above and let me know if you’re still hitting issues, that would be fantastic. I did scour our bug database and found a matching bug, but it has yet to be verified by our internal QA team and due to COVID-19, some of our bug verification times have been longer than usual as we shift bugs and resources around local government lockdowns and such.

[JimboJonesBL]

Hey guys,

thank you very much for your fast answers!

@joejo Stupid me can’t read version names. I’m sorry, I’m running XR Plugin Management 3.2.17 with Unity 2019.4.16f1. I double checked the other versions, these are correct. Thank you for pointing out!

@JasonCostanza Okay, I see progress! Now I’m getting a game preview window with a Connect button and text field to enter the IP address. Unfortunately, the development console gives me a strange error:

Following this error message I researched two things.

1. Why is the error for “Hololens ‘1’ remoting plugin”

• In checked the HolographicRemoting prefab from MRTK.Tutorials.PCHolographicRemoting.unitypackage, that is responsible for the remote connection. It has a script attached which has the following lines for connecting:

    public void Connect()
    {
        if (HolographicRemoting.ConnectionState != HolographicStreamerConnectionState.Connected)
        {
            //HolographicRemoting.Connect(IP); //For HL1
            HolographicRemoting.Connect(IP, 99999, RemoteDeviceVersion.V2);

        }
    }

Full code in attachment.

Peeking the definition of RemoteDeviceVersion…

namespace UnityEngine.XR.WSA
{
    [NativeHeader("Modules/VR/HoloLens/PerceptionRemoting.h")]
    public enum RemoteDeviceVersion
    {
        V1 = 0,
        V2 = 1
    }
}

… gives probably the answer to why it says “Hololens ‘1’”?

2. “Did you enable remoting in Player Settings”

• Well, because I’m using the new XR Plugin, the options in the Player Settings are greyed out of course. As stated in my first post Holographic Remoting is enabled in the Mixed Reality settings.

Am I missing something here?

6635836–756964–HolographicRemoteConnect.cs (1.46 KB)

[JasonCostanza]

Hey @JimboJonesBL

Sorry just saw this, I apologize!

The option for Holographic remoting is also in the XR Plugin system. If you look in my screenshots, that setting is found under the Windows Mixed Reality loader settings. Go to Project Settings > (under) XR Management > Windows Mixed Reality. Select that and you will get settings pertaining to the WMR loader specifically, in which Holographic Remoting now lives. If that is disabled, that could be the cause of the error you’re getting though I do also not like the text you’re receiving. I will ask our engineer if he can track down that error and confirm my thinking or not.

Note, a lot of our engineers have already taken time off for the holidays so it may take us a while to get a confirmation from them as many are not checking emails or messages at this time.

[JimboJonesBL]

Hey @JasonCostanza ,

citing my first post, I guess this already complies with my settings:
Edit->Project Settings-> XR Plug-in Management->Windows Mixed Reality
PC, Standalone:
-Enable Shared Depth Buffer (24bit)
UWP:
-Enable Holographic Remoting
-Enable Shared Depth Buffer (24bit)

I really appreciate the effort and am looking forward to receiving some news on this after the Christmas break. Happy holidays to you guys

[JasonCostanza]

Oh I apologize I misread that first post. I just got back from holiday so let me get this back on my radar and I’ll have updates for you soon. I hope you had a happy holiday and new year!

[JasonCostanza]

By soon, turns out that means 20 minutes in this case. Updates!

@JimboJonesBL can you search in your project in Visual Studio for MicrosoftHolographicAppRemoting.dll. If that file is missing then somehow your plugin did not install the remoting stuff you need and you should purge the package from your project fully, then re-download the WMR package and re-enable that holographic remoting checkbox and try again.

[JimboJonesBL]

Hey @JasonCostanza ,

happy new year! Thanks for getting back on the topic so fast!

Searching through the built project, I found a dll file with almost the same name (see below). I assume that’s the one?

[JasonCostanza]

Hey @JimboJonesBL Let’s get you to file a bug report attaching your project and we will take a look to eliminate the back-and-forth exchange. I’m hoping by looking at your project we can give you exact guidance or realize we broke something and will make a fix for you that you can track using your bug report. How’s that sound? Provide the bug number here when you have it and I’ll go intercept it.

[JimboJonesBL]

Hey @JasonCostanza ,

I filed a bug report with the project attached. Hopefully, you can tell me where it’s going wrong with this!

It’s Case 1304280

[JasonCostanza]

@JimboJonesBL

Hey @JasonCostanza ,

I filed a bug report with the project attached. Hopefully, you can tell me where it’s going wrong with this!

It’s Case 1304280

Thank you, got the bug. I’ll take a look at it real soon and post an update here or email you with any info or questions

[JasonCostanza]

I approved your bug, definitely something going wrong in there and I couldn’t even find what it was. I’m going to collaborate with our remoting engineer and see if we can spot what’s the problem. Unity Issue Tracker - Standalone holographic remoting application with XRSDK cannot connect to Hololens2

[JimboJonesBL]

Hey @JasonCostanza ,

is there any progress to report on this issue yet? Unfortunately, the issue status didn’t change since it went live.

[JasonCostanza]

@JimboJonesBL

Hey @JasonCostanza ,

is there any progress to report on this issue yet? Unfortunately, the issue status didn’t change since it went live.

I looked into the status more and the bug is assigned to our engineer for fixing. I am not sure if they have begun work on investigating it yet but it is on their plate.

[keveleigh_1]

It looks like you’re using the legacy UnityEngine.XR.WSA API, but for XR SDK there’s a new API in the Windows XR Plugin package: Class WindowsMRRemoting | Windows XR Plugin | 2.6.1. I think you’ll want to migrate to that one.

[JimboJonesBL]

Hey @keveleigh_1 ,

thanks for jumping on this topic.

I adjusted the connection script with the WindowsMRRemoting class, see attachment. The error from above disappears, unfortunately, another weird error appears:

Checking the log:
Begin showing splash screen.
The referenced script (Microsoft.MixedReality.Toolkit.XRSDK.Oculus.Input.OculusXRSDKDeviceManagerProfile) on this Behaviour is missing!
(Filename: C:\buildslave\unity\build\Runtime/Scripting/ManagedReference/SerializableManagedRef.cpp Line: 199)

The referenced script on this Behaviour (Game Object ‘’) is missing!
(Filename: C:\buildslave\unity\build\Runtime/Mono/ManagedMonoBehaviourRef.cpp Line: 334)

A scripted object (probably Microsoft.MixedReality.Toolkit.XRSDK.Oculus.Input.OculusXRSDKDeviceManagerProfile?) has a different serialization layout when loading. (Read 64 bytes but expected 148 bytes)
Did you #ifdef UNITY_EDITOR a section of your serialized properties in any of your scripts?
(Filename: C:\buildslave\unity\build\Runtime/Serialize/SerializedFile.cpp Line: 2324)

This error did not halt the application though. I still could enter an IP address and click connect. WindowsMRRemoting.isConnected also returns true, but nothing is displayed on the Hololens2.

Starting the application in Debug mode, I get some exceptions, but they don’t give me a direct hint what’s going on (Full debug log in attachment). The log also shows, that the Connect() call returns false?!

Unfortunately, I won’t be able to debug this project any longer in depth. But if there is only an error in the connection script I can still check if a solution works.

Thank you for your help!

6822473–792665–HolographicRemoteConnect.cs (1.7 KB)
6822473–792668–Log.txt (11 KB)

[manurocker95]

Sorry for reviving an old thread but does anyone know why HolographicRemoting doesn’t exist under UnityEngine.XR, UnityEngine.XR.WindowsMR, UnityEngine.XR.WSA nor UnityEngine.XR.OpenXR? I’m using OpenXR but can’t connect to the hololense2 in standalone…

[JimboJonesBL]

Hey @manurocker95 , I’m way off topic these days and I didn’t follow the introduction of OpenXR to Hololens2.

I assume you already followed these steps on the Microsoft Website(?):

Use PC resources to power your app with Holographic Remoting remote app -...