We are happy to announce the release the SpeechBlend for Unity!
SpeechBlend is a machine learning-based plugin for producing realistic and real-time lip syncing from any audio source.
SpeechBlend uses the viseme (mouth shape) blendshapes from character models such as DAZ or iClone characters (or any character model containing a similar set of viseme blendshapes) to perform real-time lip syncing.
This is a list of the supported viseme templates (each box represents a sound corresponding to a mouth shape):
Any character with all or some of these viseme shapes will be compatible.
SpeechBlend works by analyzing the audio signals from an audio source and predicting what mouth shape would most likely approximate that particular sound.
To set up SpeechBlend:
- Add the SpeechBlend component to a character
- Point the input audio source to the character’s head audio source
- Select the head mesh containing the viseme blendshapes
- Assign the correct blendshapes from the drop-down menu or click the Auto-Detect button if the shapes are named correctly.
- Start lip syncing!
SpeechBlend has a variety of options to adjust the performance and look of the lip syncing.
There is also a “Jaw-Joint Only” mode if no viseme blendshapes are available. This will simply open/close the jaw-joint in response to the audio signal amplitude for a less realistic result.
Find SpeechBlend in the asset store here
Thanks for reading and I’m happy to answer any questions!