Numerous technical, cognitive, social, and organizational constraints and biases can reduce the quality of usability data, preventing optimal responses to a system’s usability deficiencies. Detecting and appropriately responding to a system’s usability deficiencies requires powerful collection methods and tools, skilled analysts, and successful interaction amongst usability specialists, develop...

This paper first looks at current methods of evaluating automated content-based media analysis systems. Several key deficiencies are identified, particularly with regard to test set creation and metric design. A new framework is proposed that better reflects real-world conditions and end-user requirements. This is based on the author’s experience as a professional filmmaker and researcher in th...