Visual Routines and Visual Search: A Real-Time Implementation and an Automata-Theoretic Analysis

I describe a real-time implementation of Ull-man's visual routine processor (VRP) theory of intermediate vision for visual search. The system performs serial self-terminating visual search and computes 2D spatial relations of objects from live color video using low cost hardware. I present a formal model of a VRP with unbounded resources and quantify the amount of external control structure required to solve Horn clauses using the VRP. In discussing the effect of resource limitations I show that contemporary models of biological visual attention are unable to solve surprisingly simple queries. I also describe a novel logic programming system that finds satisfying variable assignments for Horn clause queries using the VRP. The system contains no internal database: all logic variables are directly grounded in the world using VRP queries. Finally, I briefly discuss experiments with natural language interpretation and motor control using the VRP. Experiments on real data are given. 1 1 Introduction Shimon Ullman proposed the visual routines theory of intermediate vision as a way of explaining how the human visual system might solve certain visual tasks (such as computing spatial relations) that seem to require serial processing [Ullman, 1984]. At a gross level, the theory proposes that the visual system contains a set of registers that can contain different types of visual data, a means of focusing visual attention on task-relevant portions of the image, and a set of primitive "instructions," such as coloring and line drawing, that can be combined like instructions in a computer program to compute useful Bryson kindly read drafts of this paper and provided useful comments. properties of an image. These resources are collectively referred to as the visual routine processor or VRP. The visual routines theory have received increasing attention from the AI community in recent years Much of this attention has come from the reactive reasoning and planning community, in part because Agre and Chapman's implementation of the visual routines model provides an alternative interface between reasoning and perception, which is both easier to implement and more biologically plausible than standard database-like interfaces. Despite the level of interest, there has yet to be a VRP implementation that runs on real camera images. To date, VRP systems have run either off of hand-drawn bitmaps [Romanycia, 1987] or have been directly interfaced to the world model of a world simulator, thus bypassing low-level vision entirely [Agre and Chapman, 1987][Chapman, 1990][R eece and Shafer, 1991]. …