Deep Facial Expression Modeling and 3D Motion Retargeting from 2D Images

Abstract

Facial expression modeling and motion retargeting, which involves estimating the 3D motion of a human face from a 2D image and transferring it to a 3D character, is an important problem in both computer graphics and computer vision. Traditional methods fit a 3D morphable model (3DMM) to the face, which requires an additional face detection step, does not ensure perceptual validity of the retargeted expression, and has limited modeling capacity (hence fails to generalize well to in-the-wild data). In this thesis, I present five deep learning based approaches to overcome these limitations: (1) a supervised network to jointly predict the bounding box locations and 3DMM parameters for multiple faces in a 2D image, (2) a self-supervised framework to jointly learn a personalized face model per user and per-frame facial motion parameters from in-the-wild videos of user expressions, (3) a multimodal approach that leverages both audio and video information to create a 4D facial avatar using dynamic neural radiance fields, (4) a semi-supervised multi-stage system that leverages a database of hand-animated character expressions to predict a character's rig parameters from a user's facial expressions, and (5) an unsupervised cycle-consistent generative adversarial network to directly predict the character's 3D geometry with retargeted expression. Experimental results have shown that these approaches outperform state-of-the-art methods in terms of retargeting accuracy. Applications of these approaches include avatar animation for visual storytelling or virtual conversation, motion capture films, and social AR/VR experience.