Coupling Robot Perception and Online Simulation for Grounding Conversational Semantics

How can we build robots that engage in fluid spoken conversations with people, moving beyond canned responses and towards actual understanding? Many difficult questions arise regarding the nature of word meanings, and how those meanings are grounded in the world of the robot. We introduce an architecture that provides the basis for grounding word meanings in terms of robot perception, action, and memory. The robot’s perceptual system drives an online simulator that maintains a virtual version of the physical environment in synchronization with the robot’s noisy and changing perceptual input. The simulator serves as a “mental model” that enables object permanence and virtual shifts of perspective. This architecture provides a rich set of data structures and procedures that serve as a basis set for grounding lexical semantics, a step towards situated, conversational robots.