VR Magic Gestures AI - 3D Gesture Detection and Recognition

Community Showcases
1

VR Magic Gestures AI is a starter kit for detecting and recognizing 3D gestures using the power of neural networks.

Links:

2

The package is live! VR Magic Gestures AI | Behavior AI | Unity Asset Store

3

Version 1.03 brings the following improvements:

  • Added event delegates to WandManager class (gesture events now can be handled without modifying the code, and the class does not rely on the existence of the UI system)
  • Added optional mirroring for a recognized gesture (support for left-handed players)
  • Fixed the singleton base class (it triggered assertion when a singleton was in a prefab)
4

Hey @ravingbots ,

Do you think your plugin would be able to help with recognizing sword swings (i.e. when they start and end) and where they will end up striking an enemy?

Cheers,
Colton

5

Hi Colton,

The mechanics you described would require continuous gesture recognition, which is still on the road map. However, you should be able manage and customize the way how the recognition system works quite easily. You could continuously register the tracker position, add it to the gesture trail (prune it when it is too long), invoke the recognition method every frame (or in intervals), and clear the gesture trail every time a sword swing was successfully recognized. You could also add some constraints like minimum gesture length or minimum recognition interval. To summarize, the package does not have that but it should be easy to add.

Best,
Bartek

6

Hi @ravingbots ,
Ready-To-Use gestures (like ‘m’, ‘|’ gesture in the demo) can be added by developer?

Thanks

7

Hi @sunari ,

I will add this to the to-do list for the next update, but for now you should be able to train a new gesture easily.

Best,
Bartek

8

Hi ravingbots,

I got it, I will purchase when the next update is alive.
Can’t wait!

Thanks

9

I get a compile error –
Assets/RavingBots/Sources/MagicGestures/AI/GesturePreprocessor.cs(132,64): error CS0104: Vector3Int' is an ambiguous reference between RavingBots.MagicGestures.Utils.Geometry.Vector3Int’ and `UnityEngine.Vector3Int’

10

Hello,
The package was uploaded before Unity introduced its own int vectors (Vector3Int), so these classes were provided in the package. They are exactly the same and you can delete them (Vector3Int.cs in the package). After that you will need to change the int vector variables from upper case to lower case (replace “.X” with “.x”). This should solve the problem.

11

Great, is there a way

Great, the only problem I have now is using this with an Oculus, is this possible? I tried other products but none of them work except this one. It would be a big plus for my project if I can get this to work.

12

At the moment, the package communicates with SteamVR plugin, which should support both the Vive and Oculus. However, I haven’t tested it for Oculus, but it shouldn’t be difficult to adapt for it.

13

Could you tell me which script I need to edit for the controls? This way I could change it to whichever button I need.

14

Check out ControllerManager and TrackedController in the RavingBots.MagicGestures.Integration namespace. They handle controller events. The events are used by MagicWand class.

1 Like
15

Thank you!

16

and I’ve forgot about the WandManager - you should look there before MagicWand :slight_smile:

1 Like
17

I tried making this work for simple keyboard controls but the coding is fairly hard to understand as it references a lot of ambiguous objects.

18

Do you have any particular objects in mind? The package has several modules.

19

I just want the start action and stop action to be controlled by keys on my keyboard, but simply calling the methods doesn’t seem to work since they have some object “this” inputed

20

Maybe this will help:

  1. Open MagicWand.cs
  2. Replace all occurrences of SteamVR_TrackedController with MyCustom_TrackedController
  3. Create a new class MyCustom_TrackedController (MonoBehaviour) and implement the following events:
  • public event ClickedEventHandler MenuButtonClicked;
  • public event ClickedEventHandler TriggerClicked;
  • public event ClickedEventHandler TriggerUnclicked;
    (the events are referenced by the MagicWand in Awake())
  1. Use keyboard or mouse input to invoke the events in MyCustom_TrackedController
  2. Use keyboard or mouse input to move the transform in MyCustom_TrackedController
  3. Remember to replace all SteamVR_TrackedController with MyCustom_TrackedController components in the scene.

Now your new class should emulate a VR controller.