VR Desktop Mirror asset - x Windows and HTC Vive

Unity Engine
1


This asset allow viewing and interacting with Windows Desktop (mainly Unity Editor) within VR.
Useful for example for tuning variables in inspector without need to leave the VR headset.

All source-code (C# Unity and C++ DLL) is released on GitHub under a LGPL licence: GitHub - Clodo76/vr-desktop-mirror: Windows Desktop Mirror - DLL and Unity Asset

Requirements

  • HTC Vive or Oculus Rift CV1/DK2
  • Windows 64 bit
  • SteamVR asset if used with HTC Vive

How to

  • Install this package: Download (41 KB!) - v1.0
    If you have a compilation error about missing SteamVR_Controller, comment the first line
#define VDM_SteamVR

This will remove the support of HTC Vive.

  • Drag ā€œVR Desktop Mirror > Prefabs > VR Desktop Mirrorā€ in project tree.

  • Start PlayMode, pick your VR headset.

With Keyboard / Mouse / Oculus Rift:

  • Press the left control to show the desktop. Quick press to hide. Hold press to move. During hold press, use the scroll wheel to adjust distance.
  • Press the left alt to switch to zoom mode (for better aiming). Use the scroll wheel to adjust the zoom distance.
  • Important*: If you interact with Unity Editor UI, the detection of keyboard for show/zoom donā€™t work. Re-give the focus to the Game View window.
    Note also that it reflect the mouse event in the REAL screen position, NOT the mouse over the plane/texture.

With HTC Vive:

  • Click the Grip button on any controller. It will show the desktop.
  • Point with the controller the desktop and retain press the Grip for positioning.
  • Point and short-click the Grip to hide.
  • Interact with the desktop with any controllers.
  • Trigger act as mouse left click.
  • Thumbpad act as mouse left or right click, depend on the side pressed.

Properties of the asset have tooltip with specific documentation.

The BIG issue

Some functionality of Unity (for example, menu and context menu) pause the playback, and so the VR environment. I consider this issue an Unity bug, because occur even if ā€œPlay in backgroundā€ is enabled.
I opened a topic in the Editor forum, and i also fill a bugreport.

Until the bug is fixed, the asset detect and close any menu and context-menu to avoid the VR environment lock. This can be disabled by unchecking the property ā€œHackUnityBugā€.

The following itā€™s a list (not exaustive) of action not allowed:

  • Unity Editor Menu (for example File, Window etc).
  • Any Context Menu (for example right click on a gameobject).
  • Any enum property in inspector (that are managed with a context menu).
  • Removing a component (that are managed with a context menu; Adding a component work).
  • etc.

A little issue

Unity for a Input.GetKeyDown(KeyCode.X) return always false also if the X key is pressed, but Game View in Play Mode donā€™t have the focus. This issue donā€™t occur with HTC Vive buttons. If anyone know how to resolve this, please write a post. Thanks.

Faq

Q- What if i use ā€œDragā€ or ā€œTouchpad clickā€ in my game?
A- To avoid collisions, in your script check the bool ā€œVdmDesktopManager.ActionInThisFrameā€, and skip your behiavour if is True (Input accepted for interaction with the desktop).

Q- Future of this asset?
A- There are a lot of possible improvements:

  • Support of multiple monitors
  • More buttons configurations preset
  • Simulation of mouse wheel or center mouse click
  • Virtual keyboard

But honestly the Unity bug described above itā€™s serious, i hope that someone can help me to point out to Unity team the issue, otherwise it seem useless to improve this project.

Thanks for any feedback.

5 Likes
2

Hey there! I was searching around for something like what youā€™ve created and found your post. Iā€™m working with the DK2 and I was trying to have the screen as a material on a plane, rather than a pop-up menu that can only be reached through the controllers. Do you think this would be possible with your script? It looks really cool so far. :slight_smile:

3

This is a new way to see the technology. It actually make virtual reality enjoyable and impressive. This is a part of standardize experience to the customerā€™s home.

4

Hi, i read your comment, i will implement support for Oculus (and an interface that not require Vive controller) as soon as possible. Thanks for the feedback.

5

Project updated:

  • Oculus DK2/CV1 support. HTC Vive support can be removed by commenting a #define directives.
  • Support of mouse / keyboard for seated headset (Oculus)
  • Minor improvement and bugfixes

Topic rebuilded after the forum rollback.

p.s. i cannot update the title of the topic. Can a moderator change it to

? thanks.

6

So I tried this in Unity 5.3.5 and the virtual desktop plane is white. I can use the controller as a mouse but I canā€™t see anything in game.

Iā€™m using Windows and the HTC Vive.

1 Like
7

Just wanted to say this is awesome and Iā€™ve been looking to implement something similar in one of my projects! I sent you a message about it.

-Thanks again

8

Download updated to fix a compilation error if used without SteamVR.

Sorry, i canā€™t reproduce that. You are sure you use the Unity 64 bit edition with this asset? You donā€™t have any log error?

9

is it possible instead of having the desktop rendered in a texture, to render different application frames opened in desktop, in different textures?

10

Any update on this for 5.4?

11

I found the problem. Iā€™m using a laptop and I guess it captures the desktop differently. Any ideas on how to get around this?

12

have a 4970k and an ASUS GeForce GTX 970 STRIX, and I experience zero stuttering in those games. All stock, no special configurations. Edit: No idea how much of a difference it makes but I have 32GB of RAM.

13

There is a Error , Can anyone help meā€¦

Its occurs when I Import asset there was 3 error 2 was of about //#define VDM_SteamVR and 3 one is this
Kindly help me I really like your asset

Assets/VR Desktop Mirror/Scripts/VdmDesktopManager.cs(386,16): error CS1501: No overload for method IssuePluginEvent' takes 2ā€™ arguments

14

How to use it Can Anyone help me?

15

@abrar-robotics Iā€™m not the developer but I got it working by importing the package as normal, then importing the SteamVR plugin from the asset store, then deleting the camera and using the prefab steam rig and steamvr into the empty scene. It then worked for me.

16

@Clodo Iā€™ve sent you a message relating to this plugin. I make a player called Whirligig and Iā€™d like to use this plugin to allow people to mirror their desktop to watch content like youtube etc. I think Iā€™ve got it working the way I want but still need to implement it and also have some other questions. Iā€™ve sent you a private message but I thought Iā€™d leave a message here as I canā€™t think of any other way of getting in touch :slight_smile:

Great work on this by the way. This should be an Asset store plugin rather than just given away free.

17

Hi to all, sorry for the delay.
I updated the .unitypackage for Unity 5.4.1f1. Download: https://github.com/Clodo76/vr-desktop-mirror/raw/master/Output/VrDesktopMirror.unitypackage
I released all the source-code (C# Unity and C++ DLL) on GitHub, under LGPL. https://github.com/Clodo76/vr-desktop-mirror
I added multiscreen support.
I added some property related to multiscreen and colorspace.

Previous version works only with Unity 64 bit, the new version have both 32/64 dll.

Please note:

  • Require at least Windows Vista. DONā€™T WORK with Windows 7. It use Windows API that are available only from Vista and above, nothing i can do about this.
  • I cannot detect if SteamVR/Vive asset is installed or not. See the first post if anyone have a compilation issue.

Honestly, the desktop mirror feature maybe, but this Asset aim to be a VR interface for Unity, but the Unity bug described in first post are too much trouble.

18

How to use the multiscreen function?
I imported the new package but I have no idea how I can use it

19

Is it possible to add a function that can not only show the desktop on the unity, it also show the specified window like folder or chrome on the unity? It just like the AR screen from Leap Motion.

0:35

20

You need multiple screens plugged in to your PC.

Itā€™s possible to ā€˜fakeā€™ it using a headless hdmi adaptor such as: https://www.megamac.com/products/newertech-hdmi-headless-video-accelerator-nwtadp4khead