Home!
About
 
 
Auto Lip Syncing with Machine Learning
 Date: May 29, 2020 → July 7, 2020
 Type: Personal Project, Machine Learning and Computer Science
 
Basic Explanation  

In animation, it takes several hours to create only a couple seconds of visuals. As an animator myself, I often become overworked with the harder smaller parts of animation that just end up costing more time such as lip syncing an animated character. These are the visemes I created and used to animate lip syncing.

"Ahh" Sound:
"Eee" Sound:
"Sss" Sound:
"L/Th" Sound:
"Oh" Sound:
"F/V" Sound:
"B/Mmm/P" Sound:

In my project, rather than using audio, I decided to focus on the video of my mouth speaking. To do this, I wrote software to locate where my mouth was in frame then passed it through a machine learning model to predict what mouth shape I was making. Overall, the computer does only a fair job at creating a convincing lip sync. (The bar graph shows how confident the computer was at it's answer. The red bar denotes which one the computer picked):