Audio and Visual Processing to Enhance User Interfaces

SUMMARY This report details the work carried out between the months of October 1996 and June 1997, and the results so far achieved. Numerous speech and person recognition experiments have been performed using both speech and visual lip features. The discriminatory properties of audio and visual features are examined, along with the performance of two classiiers, namely VQ and DTW. The eeect of varying the amount of training data is observed and both static and dynamic features are investigated. The results for person recognition show that both the audio and visual features contain valuable speaker speciic information. For speech recognition, the audio signal, as might be expected produced easily the better results, when compared with the visual features. It was noted that for both speech and person recognition, increasing the amount of representative training data improved the recognition results. Also, the DTW type classiier performed marginally better than the VQ classiier for recognition, with only a few (unexplained) exceptions. It was observed that the audio dynamic features contained valuable speech and speaker idiosyncratic information, while the visual dynamic features, for both speech and person recognition contained very little discriminatory information. Possible avenues for future research related to enhancing a user interface are postulated.