CONTENT-AWARE APPROACHES FOR DIGITAL VIDEO ADAPTATION, SUMMARIZATION AND COMPRESSION By TAORAN LU A DISSERTATION PRESENTED TO THE GRADUATE SCHOOL OF THE UNIVERSITY OF FLORIDA IN PARTIAL FULFILLMENT OF THE REQUIREMENTS FOR THE DEGREE OF DOCTOR OF PHILOSOPHY

of Dissertation Presented to the Graduate School of the University of Florida in Partial Fulfillment of the Requirements for the Degree of Doctor of Philosophy CONTENT-AWARE APPROACHES FOR DIGITAL VIDEO ADAPTATION, SUMMARIZATION AND COMPRESSION By Taoran Lu December 2010 Chair: Dapeng Oliver Wu Major: Electrical and Computer Engineering In this dissertation, we mainly present our work on three challenging problems of digital video applications: video compression, summarization and adaptation. Unlike conventional techniques, we focus on the video content modeling and investigate how the characteristics of human attention will help solving these problems. We denote these approaches “content-aware” approaches and present our innovations in the main body of this dissertation. The first problem is content-aware video adaptation. We employ saliency analysis, which generates a saliency map to indicate the relative importance of pixels within a frame for human attention modeling. We also propose a nonlinear saliency map fusing approach that considers human perceptual characteristics. To effectively map the important content from source to the target display, we propose to have both intra-frame visual considerations and inter-frame visual considerations, where intra-considerations focus on measuring the information loss within a frame, and inter-considerations emphasis the visual smoothness between frames. The mapping problem is formulated as a shortest path problem and is solved with dynamic programming. The second problem is content-aware video summarization. We introduce an automatic video summarization approach that includes unsupervised learning of original video-audio concept primitives and hierarchical (both frame and shot levels) skimming. For video concept mining, we propose a novel model using bag-of-word (BoW) shot