Multimodal Interaction in Collaborative Virtual Environments

Human interfaces for computer graphics systems are now evolving towards a total multi-modal approach. Information gathered using visual, audio and motion capture systems are now becoming increasingly important within user-controlled virtual environments. This paper discusses real-time interaction through the visual analysis of human face feature. The underlying approach to recognize and analyze the facial movements of a real performance is described in detail. The output of the program is directly compatible with MPEG-4 standard parameters and therefore enhances the ability to use the available data in any other MPEG-4 compatible application. The real-time facial analysis system gives the user the ability to control the graphics system by means of facial expressions. This is used primarily with real-time facial animation systems, where the synthetic actor reproduces the animator’s expression. The MPEG4 standard mainly focuses on networking capabilities and it therefore offers interesting possibilities for teleconferencing, as the requirements for the network bandwidth are quite low.