How do you deal with complex input devices?

I’ve had a good read of the stuff in here, and it looks pretty good, but I don’t understand how it’s going to extend beyond ‘simple’ devices.

At a high level, my understanding is that you create a ControlScheme which is linked to a number of Device types.

Actions can then be associated to Device specific input.

Sound about right?

So for example, you associate a ButtonAction with a Gamepad input button, and that lets you check if the button is up, down, held, etc.

Superficially this seems very reasonable, but how is it going to work with more complex devices?

Let’s have a look at some concrete examples:

PS4 controller
The PS4 controller has a rotation associated with it (eg. Vector3Action). However, it is a Gamepad.
It also supports Rumble. Neither of these action types are exposed on the Gamepad type.

If the Gamepad device is modified to support these, does that mean you have to query your Gamepad device to see what actions are available on it somehow?

What if someone comes out with a new “3DScanning Gamepad” that yields a stream of point cloud data? Or a gamepad with a keyboard on it (PS4…) (obviously a contrived example, but bear with me…) does Gamepad get modified to support it? Doesn’t that mean that Gamepad becomes a MegaDevice with a million different Actions on it?

Or do you create a new virtual device that is some kind of weird thing where it is actually multiple ‘Devices’ associated with a ControlScheme (eg. keyboard + gamepad) when there is actually only one physical device?

Vive 3d controllers
So here we have both 3d world position and rotation for each input, and there can be multiple physical devices.

You want to do this:

    void Start() {
        // Bind the actions.
        look.Bind(playerInput.handle);
        leftHandPos.Bind(playerInput.handle);
        leftHandRotation.Bind(playerInput.handle);
        leftHandButton.Bind(playerInput.handle);
        rightHand.Bind(playerInput.handle);
        rightHandRotation.Bind(playerInput.handle)

...

You see the issue I’m sure.

How are leftHand, leftHandButton and rightHand, rightHandButton correctly bound to the right devices?

Or do we get a PositionRotationButtonVector3Input type for this?

More generally, when you have a multiple physical devices that each provide multiple complex input types (Vector3, Button, maybe even say a camera), how do you correctly associate all of the Actions into one group for each input device?

Swapping devices
Perhaps you may have seen this recent GGJ game, where the ‘ghost’ players take one controller each, while the other player where the headset: http://globalgamejam.org/2016/games/hide-spook-steam-vr

In this situation you have three players; each player must be associated with one device; a move controller or a VR headset. How is assignment of physical device to each player handled?

How do players indicate that they have swapped devices if they do?

(I could imagine a reinitialize command → ‘click with your device to select your player’ or something, but technically how would this be handled by the system? All the input bindings would have to be updated to reflect new devices)

Thanks for taking the time to look at the system in depth!

Ideally, you should not need to do queries. The point of control schemes is to encapsulate device differences in the control schemes so that your gameplay code doesn’t need to know about it. This may be an ideal that doesn’t work 100% in practice but the aim is to make it work as much as possible.

Our aim is to provide some standardized device classes to help you not have to worry about individual models and brands of devices of that type. The Gamepad device type is such a standardized device. These standardized devices should contain the controls that you can expect and rely on being present on that type of device.

Some devices may fit the criteria of a standardized device class but have additional controls on top. Those should have their own device types that inherit from the standardized type. E.g. PS4Controller inherits from Gamepad. This allows you to implement a control scheme for just gamepads if you don’t care about the extra controls, or to (additionally) implement a control scheme for PS4Controlller if you want to use the extra controls or in other ways do a customized control scheme tailored just for the PS4 controller.

Right now this inheritance support is not fully implemented. You should be able to do it, but we haven’t tested it, and the logic that determines which control scheme to pick doesn’t know what to do if there is ambiguity due to there being both a control scheme for Gamepad and for PS4Controller, so the result is currently undefined. We probably need to add a priority value to control schemes for handling these cases. (Picking the control scheme that uses the more specific class could make sense for some cases but given control schemes can have multiple devices, it’s not always that simple and a user-specified priority value would be safest.)

Does this approach of specific devices being able to inherit from standardized devices make sense to you and do you think it will solve most problems of the type you described? Or if not, do you have suggestions for other ways to handle it?

Yep, we don’t currently support more than one device of the same type in a control scheme. We’re aware of the problem and it’s on our todo-list to fix it. :slight_smile: We’ll need a way to give each device slot in the control scheme a name/identifier.

[quote=“doug_1, post:1, topic: 625469, username:doug_1”]
Swapping devices
**[/quote]
**[quote=“doug_1, post:1, topic: 625469, username:doug_1”]
Perhaps you may have seen this recent GGJ game, where the ‘ghost’ players take one controller each, while the other player where the headset: http://globalgamejam.org/2016/games/hide-spook-steam-vr

In this situation you have three players; each player must be associated with one device; a move controller or a VR headset. How is assignment of physical device to each player handled?

How do players indicate that they have swapped devices if they do?

(I could imagine a reinitialize command → ‘click with your device to select your player’ or something, but technically how would this be handled by the system? All the input bindings would have to be updated to reflect new devices)
[/quote]
In the system there is API for you to assign devices to PlayerHandles. We will provide some scripts for a few typical multi-player use cases (and a simple script like that is included with the demo) but this kind of thing is often game-specific enough that you may need to write your own logic for it.

How players indicate that they swapped devices will be up to the game developer. What info to show to the players and how, and which input actions to use to make it happen. if you had a device swapping game that used currently supported devices you’d be able to do it already with the current version of the prototype.

This seems like a quite reasonable approach to take.

I can see a few cases where exactly how that works is going to be interesting; for example:

HoloLens → Position, Orientation, Camrea / Depth Camera
Gear VR → Orientation, Button
Rift → Position, Orientation
Vive → Position, Orientation

What’s the ‘VR Headset’ generic device going to look like? Seems like it would be difficult to get away without several device specific types, and an input scheme for each one.

Certainly interesting though, great work!

I’m having a play with the prototype, and I’ll definitely feed any feedback back in here~

1 Like

What about having Interfaces; IKeyInput, INumericInput, IDigitalJoystickAxis, IAnalogJoystickAxis, ISpacialPoint, IRotationAxis etc… that the base classes implement?
Then new input devices could inherit from a basic InputDeviceBase abstract class and implement the interfaces they need.

When designing a control scheme (or multiple control schemes) for an action map, can you help me understand how you imagine you as a user would use this? How would the interfaces be used in the UI for creating the control scheme?