Hi all,
What is the best practice of generating or building the same UI for different sources (It can be Oculus VR UI, can be a tablet UI or can be desktop UI)? The problem is each source uses different dependencies and Is it possible to exclude these dependencies to create the same UI for the different sources. If it doesn’t clear I can elaborate on it.
Thanks.
[Edit] Also found this link Best Practices for User Interfaces (UI) in VR with the XR Interaction Toolkit - Unity Learn
If i remember correctly the demos for Vive, Oculus, etc use custom classes you can drop onto your UI elements. Things like VRClickable and the such like. I assume your talking about those type of platform specific classes that tie your UI implementation into a specific XR platform.
If so, then my approach as been to first look at how to not use any of those classes at all. Period, consign them to the bin. Oculus framework code is in my oppinion garbage.
Unity i believe as tried to resolve this issue with their XR input code. You can begin to read about it here Unity - Manual: XR
I think that the idea behind Unity XR is that you can use Unity’s built in classes to manage inputs such as Hands, controllers etc. Which means you don’t have to worry so much about the specific platforms implementation of controllers etc.
I haven’t tried it in anger yet, so not 100% sure how it works, but thats where i’d start looking.
Regards UI, i currently stick a raycaster object on the hand controllers and have a custom raycast listener on the UIcanvas that triggers the existing events (akin to mouse events). Saves having to add specific classes to the UIelements (buttons, and so forthe). God knows why this isn’t the standard. I’m assuming theres a better way, but this is how i do it at the moment and its done just enough to trigger buttons.
Drop the below script on your EventSystem object. Add to pointerTransforms anything you want to be able to point on a UI with. i.e. could be a laserPointer like object or the vr hand instances.
using System.Collections;
using System.Collections.Generic;
using UnityEngine;
using UnityEngine.EventSystems;
using UnityEngine.UI;
// Reference Articles
// https://github.com/tenpn/unity3d-ui/blob/master/UnityEngine.UI/EventSystem/InputModules/PointerInputModule.cs
// https://unfragilecoding.blogspot.com/2019/01/how-to-make-all-gui-components.html
// https://ritchielozada.com/2016/01/01/part-8-creating-a-gaze-based-input-module-for-unity/
// https://medium.com/@naszrai.andras/virtual-reality-ui-with-unity-6e3d02f671e4
// https://answers.unity.com/questions/1248512/graphicraycaster-seems-to-ignore-toggle-elements.html
// https://docs.unity3d.com/Manual/OculusControllers.html
// https://answers.unity.com/questions/945299/how-to-trigger-a-button-click-from-script.html TOGGLE
public class VRHand_EventSystem_InputModule : PointerInputModule
{
public LayerMask layerMask;
public List<Transform> pointerTransforms = new List<Transform>();
private List<RaycastResult> lastResults =new List<RaycastResult>();
public LineRenderer debugLineRenderer;
/// <summary>
/// If the raycast appears off set, check for invisible colliders that will report an incorrect screen position.
/// </summary>
public override void Process()
{
Vector3 endPosition = Vector3.zero;
if (pointerTransforms.Count > 0)
{
// TODO multiple points cancel each other out. Fix
pointerTransforms.ForEach((transform) =>
{
endPosition = transform.position + (0.1f * transform.forward);
Ray ray = new Ray(transform.position, transform.forward);
RaycastHit hit;
//Debug.DrawRay(manipulator.gameObject.transform.position, manipulator.gameObject.transform.forward);
if (Physics.Raycast(ray, out hit, layerMask))
{
endPosition = new Vector3( hit.point.x, hit.point.y, hit.point.z);
// given the world position of the ray cast on a UI, calcult screen position and let EventSystem do everythyinhg else.
// TODO remove Camera.main
if (Camera.main != null) {
//castUIRay(Camera.main.WorldToScreenPoint(hit.point));
castUISelectableRay(Camera.main.WorldToScreenPoint(hit.point));
}
}
if (debugLineRenderer)
{
debugLineRenderer.SetPosition(0, transform.position + transform.forward * 0.1f);
debugLineRenderer.SetPosition(1, endPosition);
}
});
}
}
GameObject lastActivatedTarget;
GameObject target;
void castUISelectableRay(Vector3 point)
{
// Create Pointer Event
PointerEventData pointer = new PointerEventData(EventSystem.current);
pointer.position = new Vector2(point.x, point.y);
pointer.button = PointerEventData.InputButton.Left;
List<RaycastResult> raycastResults = new List<RaycastResult>();
EventSystem.current.RaycastAll(pointer, raycastResults);
raycastResults.Sort((x, y) => x.distance.CompareTo(y.distance));
if (raycastResults.Count > 0)
{
// Target is being activating -> fade in anim
if (target == raycastResults[0].gameObject && target != lastActivatedTarget)
{
if (target.GetComponent<Selectable>())
target.GetComponent<Selectable>().OnPointerEnter(pointer);
bool indexTrigger = false;
#if OVRINPUT_USE
indexTrigger = OVRInput.GetUp(OVRInput.Button.One, OVRInput.GetActiveController());
#endif
// if (Time.time >= endFocusTime && target != lastActivatedTarget) //Input.GetButtonUp("Magic_Leap_Trigger") ||
if (Input.GetButtonUp("Fire1") || OVRInput.Get(OVRInput.Button.One) || indexTrigger)
{
lastActivatedTarget = target;
if (target.GetComponent<ISubmitHandler>() != null)
target.GetComponent<ISubmitHandler>().OnSubmit(pointer);
else if (target.GetComponentInParent<ISubmitHandler>() != null)
target.GetComponentInParent<ISubmitHandler>().OnSubmit(pointer);
//else if (target.GetComponentInParent<Slider>() != null)
//{
// lastActivatedTarget = null;
// endFocusTime = Time.time + loadingTime;
// if (target.GetComponentInParent<Slider>().normalizedValue < 1f - sliderIncrement)
// target.GetComponentInParent<Slider>().normalizedValue += sliderIncrement;
// else if (target.GetComponentInParent<Slider>().normalizedValue != 1)
// target.GetComponentInParent<Slider>().normalizedValue = 1;
// else
// target.GetComponentInParent<Slider>().normalizedValue = 0;
//}
}
}
// Target activated -> fade out anim
else
{
if (target && target.GetComponent<Selectable>())
target.GetComponent<Selectable>().OnPointerExit(pointer);
if (target != raycastResults[0].gameObject)
{
target = raycastResults[0].gameObject;
}
}
}
// No target -> reset
else
{
lastActivatedTarget = null;
if (target && target.GetComponent<Selectable>())
target.GetComponent<Selectable>().OnPointerExit(pointer);
target = null;
}
}
/// a furst attempt.
void castUIRay(Vector3 point)
{
// Got from examples, seems to work ok.
PointerEventData pointerEventData = new PointerEventData(EventSystem.current);
// ******************************************
// Generate screen coords based on virtual point.
// Instead of mouse position pass in hit point of canvas collider convertered to camera screen space. Essentially a world coordinate as a mouse coordinate relative to camera.
// ******************************************
pointerEventData.position = new Vector2(point.x, point.y);
// ******************************************
// Get Raycast and sort results.
// ******************************************
List<RaycastResult> results = new List<RaycastResult>();
EventSystem.current.RaycastAll(pointerEventData, results);
results.Sort((x, y) => x.distance.CompareTo(y.distance));
// ******************************************
// Clear last selection
// ******************************************
if (lastResults.Count > 0)
{
lastResults.ForEach((result) =>
{
ExecuteEvents.ExecuteHierarchy(result.gameObject, pointerEventData, ExecuteEvents.deselectHandler);
});
}
// ******************************************
// Update New hit tartgets.
// For every result returned, output the name of the GameObject on the Canvas hit by the Ray
// ******************************************
foreach (RaycastResult result in results)
{
//Debug.Log("Hit " + result.gameObject.name);
ExecuteEvents.ExecuteHierarchy(result.gameObject, pointerEventData,ExecuteEvents.selectHandler);
// Rather than Execute evetns. just set the selected object to whatever is under the virtual mouse position.
EventSystem.current.SetSelectedGameObject(result.gameObject);
// Not sure you'd ever want to do this. but hey its possible.
//if (Input.GetButtonUp("Fire1"))
//{
// ExecuteEvents.ExecuteHierarchy(result.gameObject, pointerEventData, ExecuteEvents.pointerClickHandler);
//}
}
lastResults = results;
// ******************************************
// Map standard Input events to trigger click
// ******************************************
if (Input.GetButtonUp("Fire1") || Input.GetButtonUp("Magic_Leap_Trigger"))
{
GameObject target = EventSystem.current.currentSelectedGameObject;
//Debug.Log("Submit " + target);
if (target)
ExecuteEvents.ExecuteHierarchy(target, pointerEventData, ExecuteEvents.pointerClickHandler);
}
}
}
I should also point out that in the Asset Store there are some really good VR input Assets, that done really good grabbing of objects, and UI input management. If you can afford it, its worth considering using a well updated and cohesive system someones already put the time and effort into.
Absolutely, for HoloLens for example uses its own interactable libs for interactions but Oculus uses different sources.
I assume there is no template or tutorials for this. I need to dirt my hand 
As you said, regarding runtime, there is no guarantee for AR and VR systems at the same time but thank you for the code block and I will try it out for HoloLens at my first try.
Have you ever use the same UI for AR and VR systems?
Sort of, in that i was always irritated by the fact most UI solutions required you to drop specific scripts on components in the UI to detect interactions, and didn’t use the built in functionality available for mouse users. Hence the above script allows you to mimick mouse cursor events by transforming an in scene cursor into a mouse point position and trigger the appropriate events. It’s by no means complete but certainly works for me.
Also, a big issue i had with the first version of Hololens was when building a project to work for both Vive and Hololens. None of the packages had been wrapped in a USES_VIVE or USES_HOLOLENS define, so it was a real pain in the ass trying to seperate those packages in a single project. Can’t remember what my solution for that was, I think i setup a seperate project as a dll import to the vive and hololens specific projects.