We just released a framework on the Asset Store that will let you integrate the Kinect™ Facetracking technology directly in Unity.
Using the package you will be able to:
track the user head position and rotation (x,y,z coordinates relative to the device);
track a set of 6 (facial) animation units (read more);
receive the rbg and depth camera streams directly from the device.
To demonstrate the technology we used the facetracking data to animate a high-poly mask of an Oni in real-time. The complete source code for the kinect™ integration is also included in the package.
Wow, this looks pretty impressive. I don’t have a Kinect, yet - let alone have it set up to work with my Mac … but I certainly have bookmarked this and will buy as soon as I have a game where I could use it.
One thing I noticed with your YouTube video: It doesn’t have a link to the asset store or this forum posting. You might consider adding that so people who run into your demo video can easily navigate to the forum / asset store
Looks pretty awsome! But I don’t have Kinect…
So is it possible to make a webcam/mobile cam (front and back) version? I think you can reach out to more users this way.
It is a good point, yes, and we are considering exploring this possibility.
The motivation behind using Kinect though, is that the facetracking quality is much better as it uses both the depth and skeleton tracking information.
When I saw your post I thought you guys are the same guys…
EDIT: Apparently, Face tracking has become popular over the past few years… there is even an open source one called “FaceTrackNoIR” : http://facetracknoir.sourceforge.net/
So I think webcam isn’t impossible. The problem is the mobile, which is what I am most interested in - even with just rudimentary head tracking I think its good enough and hopefully more sophiscated face tracking would be avaliable later for mobile.
EDIT 2 : Looks like even on mobile it isn’t impossible - its already been done!
IIRC, there was also another one, which was more specific (I may have created that) - but since the Feedback site went to the new system, I got disconnected from all the ideas I had voted on / created … so it’s kind of difficult to find that one (if it exists at all - I just have a rather faint memory on that).
FaceAPI was a strong contender when we were researching the technology out there.
For us, the drawback was the licensing terms (each person buying our package would need to go to FaceAPI and get a license on their own terms). Of course there is a free license but that only allows to detect the head position, in a non-commercial application, and the whole source code must be released under GPL3 (which by itself would be against the EULA Unity’s asset store is using).
All in all, owning a Kinect seemed a much more reasonable alternative (from our point of view).
We are also preparing to offer other NUI mechanims/features on the asset store which would also require a Kinect (or similar tracking technology).
OpenCV might also be a worthy candidate for those whose requirements are only to track the user’s head position.
A quick update: some users reported bugs and crashes while using the kit. We just submitted a new version that fixes those problems to the asset store. It is currently under review and should be soon available.
You are correct, this uses the Kinect SDK and currently works on windows 7 and 8. It has been fully tested with both the kinect xbox 360 and kinect for pc. Note though that the installation of the SDK is necessary to use the kinect for xbox, whereas the redistributables are enough for the kinect for pc.
The accuracy is pretty much the same the one in the kinect samples, which you can try easily at http://www.microsoft.com/en-us/kinectforwindows/.
It is their detection algorithms, although we did use a little bit of magic to animate the mask more gracefully.
