Lessons Learned from Building a Terabyte Digital Video Library

T he Informedia Project at Carnegie Mellon University has created a terabyte digital video library in which automatically derived descriptors for the video are used for indexing , segmenting, and accessing the library contents. Digital video presented a number of interesting challenges for library creation and deployment: the way it embeds information, its voluminous file size, and its temporal characteristics. In the Informedia Project, we addressed these challenges by • automatically extracting information from digitized video, • creating interfaces that allowed users to search for and retrieve videos based on extracted information , and • validating the system through user testbeds. We met these objectives during the course of the project , learning many lessons along the way. Our library consisted of two types of video: news video (from the Cable News Network) and documentary video (from the British Open University, QED Communications, the Discovery Channel, and a number of US government agencies, including NASA, the National Park Service, and the US Geological Survey). As of May 1998, the library contained more than 1,000 hours of news and 400 hours of documentary video, with additional video being added daily. The news video often included tags which marked story boundaries within longer broadcasts. At first, we added these story boundaries manually for the documentary video; in our subsequent experiments we looked at generating story boundaries automatically. The stories, or video segments, averaged a few minutes in length, so that the entire library contained more than 40,000 video segments. We used artificial intelligence techniques to create metadata, the data that describes video content. We found that all the AI techniques we used were applicable to both the news corpus and the documentary corpus. These techniques included speech recognition, image processing, and information retrieval. We learned that by integrating across these three areas we were able to compensate for limitations in accuracy, coverage, and communicative power. We analyzed the audio component of the video with the CMU Sphinx speech recognition system. 1 This created a complete transcript for text-based retrieval from the speech and aligned existing imperfect transcripts to the video. The tightly synchronized transcript was subsequently used in the library interface for quickly locating regions of interest within relevant video segments. In creating transcripts, CMU Sphinx's word error rate is inversely proportional to the amount of processing time devoted to the task. Processing time varies from real time, which offers relatively poor …