Face Landmark Detection

[marsinator]

Can someone help me with detecting face landmarks in Unity AR?

[andyb-unity]

The best place to start is our Samples project: GitHub - Unity-Technologies/arfoundation-samples: Example content for Unity projects based on AR Foundation

[marsinator]

No problem with detecting faces, I struggle to detect face landmarks

[andyb-unity]

Keep reading the link I posted above?

Face Mesh
This sample instantiates and updates a mesh representing the detected face. Information about the device support (e.g., number of faces that can be simultaneously tracked) is displayed on the screen.

This sample uses the front-facing (i.e., selfie) camera.

Face Regions (ARCore)
“Face regions” are an ARCore-specific feature which provides pose information for specific “regions” on the detected face, e.g., left eyebrow. In this example, axes are drawn at each face region. See the ARCoreFaceRegionManager.cs.

This sample uses the front-facing (i.e., selfie) camera.

Blend Shapes (ARKit)
“Blend shapes” are an ARKit-specific feature which provides information about various facial features on a scale of 0…1. For instance, “wink” and “frown”. In this sample, blend shapes are used to puppet a cartoon face which is displayed over the detected face. See the ARKitBlendShapeVisualizer.cs.

This sample uses the front-facing (i.e., selfie) camera.

Eye Lasers, Eye Poses, and Fixation Point (ARKit)
These samples demonstrate eye and fixation point tracking. Eye tracking produces a pose (position and rotation) for each eye in the detected face, and the “fixation point” is the point the face is looking at (i.e., fixated upon). EyeLasers uses the eye pose to draw laser beams emitted from the detected face.

This sample uses the front-facing (i.e., selfie) camera and requires an iOS device with a TrueDepth camera.

[marsinator]

I read this, but i want to detect the mouth region. More specific the landmarks that are in this region in order to determine if the mouth is open or closed and so on…

[andyb-unity]

These samples demonstrate all the face tracking techniques available in ARCore and ARKit. On the ARKit side, mouth tracking is straightforward using Blend Shapes.

On ARCore, you would have to do some processing yourself based on the vertices of the Face Mesh, as ARCore’s Augmented Faces feature doesn’t specifically handle mouth tracking for you: Введение в дополненные лица  |  ARCore  |  Google for Developers

[marsinator]

I need to do it on Android so i have to use ARCore. I got the OpenCV and the DlibFaceLandmarkDetector Assets but have problems to get it to work.

But about the face mesh. Does every vertex have an specific index? That way i would just have to loop through the vertices list.
Do you know which vertices specify the mouth region?

[andyb-unity]

I am not certain, but I would wager that the vertices are always reported in the same order. I have not done this experiment but it seems possible that you could identify the mouth vertices.

I definitely can’t help you with OpenCV, but good luck on your app!

[marsinator]

Do you know how to draw for example a circle at a specific vertex and display this on the device?