Welcome to Spatial CV and Edge AI Revolution !
Documentation | Video Tutorials | Github | Discord
What is OAK For Unity?
OAK For Unity is native plugin for Windows, Linux and MacOS (Android support ongoing) to enable OAK devices and capabilities inside Unity.
Main goal is bring the power of OAK devices, CV and Edge AI to Unity community to build the next generation of Spatial AI / Edge AI applications.
- OAK For Unity Creators: Build interactive videogames and experiences with pretrained models and high-level API
-
OAK For Unity Developers: Productivity tools for Unity developers: VR workflows, MoCap tools (body pose and face mesh)
-
OAK For CV/AI/Robotics Developers:
-
Unity Virtual OAK camera
-
Define custom pipelines (as much as possible inside Unity with visual tools)
-
Robotics Simulation and Digital Twins
-
Synthetic dataset generation with automatic labelling
-
Deep RL with ML-Agents
What is OAK?
OAK is real swiss army knife for CV/AI.
OAK devices are Depth CV (Computer Vision) + AI (Deep Learning) cameras from Luxonis powered by Intel MyriadX VPU (Vision Processing Unit) at unbeatable price tag (<200$)
Check out OAK devices, documentation and specs on Luxonis website.
Application examples:
Are you building interactive experiences/installation/games where users can interact with their face, head, body, eyes-gaze and/or hands?
Are you building health/sports applications monitoring users body pose?
Are you building multiplayer VR experiences? Are you looking to speedup VR development?
Are you looking/building an affordable body / face MoCap solution?
Are you building next generation of streaming application using advanced background segmentation? Or depth/3D streaming to LookingGlass holographic display?
Are you building AR filters thanks to face detection and face mesh?
Are you building monitoring applications needs object detection?
Are you building application needs OCR (Optical Character Recognition)
Are you building robotic digital twin using CV/AI?
…. Endless application list here …
Features and Roadmap:
v1.0.0 Initial Release to AssetStore (free):
-
No code approach. Just drag and drop some prefabs.
-
Access to OAK device:
-
RGB, mono images and depth (point cloud viz currently ongoing dev).
-
About point cloud: Add support for external libraries: PCL, …
-
Access to IMU if available and retrieve system information
-
Record and Replay capability
-
OAK Device Manager and multiple OAK devices support
-
OAK For Unity Creators: High-Level API - Unlock “Myriad” applications with predefined and ready-to-use pretrained models (with depth support if you run OAK-D family)
-
Face Detector
-
3D Head Pose Estimation
-
Face emotion
-
Body Pose (MoveNet)
-
Hand Tracking and Gesture recognition (Blaze Hands)
-
Object detector (Tiny Yolo v3/4/5 and MobileNet SSD)
-
DeepLabv3 segmentation. Background removal.
-
Depth (only OAK-D family) - point cloud visualization
-
Example how to integrate your own pipeline
-
Integration with some paid assets (demos)
-
OAK For CV/AI/Robotics: Unity Virtual OAK Camera
Next versions:
- Improve Hand track
- Face mesh and Face animation tool
- Humanoid support with Body pose
- Eye-gaze
- OCR
- More integration with paid assets
- OpenCV and PCL support inside Unity
- More advanced demos: combining different pretrained models
Do you miss some models / use cases / integration with some specific paid asset? Please let us know.
- Android support. Integration with AR.
- End-to-end workflows for synthetic dataset generation, training and deploy
- Integration with Unity Robotics Hub (ROS) / SystemGraph / SensorSDK
- Integration with Unity Simulation and Unity Simulation Pro
- Intergration with Unity ML-Agents
- Define custom pipelines inside Unity (visual tool)
2022-01-22: New demo menu scene added. Now easier navigate through the demos
Some examples:
Old video showing some possibilites with ML pretrained models:
Body Pose:
Hand Tracking:
Synthetic Datasets:
Record and replay feature for demo scenes (replay working even without OAK device - automatic fallback when there is no device)
