An Integrated Framework for Face Modeling , Facial Motion

This dissertation presents an integrated framework for face modeling, facial motion analysis, and facial motion synthesis. This framework systematically addresses three closely related research issues: (1) selecting a quantitative representation of facial deformation for face modeling and animation; (2) automatic facial motion analysis based on the same visual representation; and (3) speech-to-facial-coarticulation modeling. The framework provides a guideline for methodically building a face modeling and animation system. The systematicness of the framework is reflected by the links among its components , whose details are presented. Based on this framework, a face modeling and animation system, called the iFACE system, is developed. The system provides functionalities for customizing a generic face model for an individual, text-driven face animation, off-line speech-driven face animation, and real-time speech-driven face animation. iv ACKNOWLEDGMENTS I would like to take this opportunity to express my appreciation to all the people who have guided, supported, and encouraged me. text-to-visual-speech project. I worked for Dr. Shum as a summer intern on the facial motion modeling and tracking project in 2000. I really enjoyed the inspiring discussions with Dr. Zhong and Dr. Shum and their invaluable suggestions. My continuous endeavors on the above directions produce fruitful results that become important components of this thesis. I especially thank Professor Stephen E. Levinson. During my thesis work on speech-driven face animation, Professor Levinson is always ready to provide his generous and stimulating suggestions based on his expertise in speech processing and recognition. My lab colleagues have also provided critical help. I especially thank thank Zhen Wen, my closest partner, who has been particularly cooperative and supportive. I thank other former and current graduate students in our group for the discussions and support. Thanks also go to the Dr. Russell L. Storms and Dr. Larry Shattuck, who kindly provided their face data for this research. Many individuals provided additional valuable technical assistance. The constant, prompt help of our system administrators, v Kaczmarski, made the research environment much more comfortable. The secretaries, are always cheerful and helpful. Finally, there are others who have influenced this research indirectly, but fundamentally, through their influence on my life. They are my parents, sister, and brother-in-law, whose love, patience, and encouragement made this research possible.