Detecting High Level Story Elements from Low Level Data

The problem addressed here is yet another artificial intelligence problem that is easily solved by young children yet challenges even sophisticated computer programs. This thesis’s canonical example is a scene featuring two entities drinking. In one scene, a cat drinks from a faucet. In the other, a human drinks from a glass. Even young humans can identify that the two images are similar in that they both involve drinking. However, low-level analysis of the scene will find many more differences than similarities in the case cited above. In my research examines ways to detect high-level story elements such as drinking from low-level data such as that which might be produced from analyzing pictures and videos directly. I present a system that accepts as input a collection of high-level events represented in transition space. I analyze, then select the affinity propagation clustering algorithm to group the events using only their low-level representations. To this end, I present a novel algorithm for determining how similar any two points in transition space are. Due to the lack of vision systems capable of providing a varied dataset, I create a system which translates English language descriptions of high-level events and produces a specially formatted transition space file. To support my hypotheses, I presents the results of two experiments using the system described in this thesis. The first experiment uses English language files and the second uses data produced from a set of experimental videos. Using the English language files the system was able to detect groups corresponding to flying and walking among others out of a total set of 16 events. Thesis Supervisor: Patrick H. Winston Title: Ford Professor of Engineering