Affordances Provide a Fundamental Categorization Principle for Visual Scenes

How do we know that a kitchen is a kitchen by looking? Relatively little is known about how we conceptualize and categorize different visual environments. Traditional models of visual perception posit that scene categorization is achieved through the recognition of a scene's objects, yet these models cannot account for mounting evidence that human observers are relatively insensitive to the local details in an image. Psychologists have long theorized that the affordances, or the actionable possibilities of a stimulus are pivotal to its perception. To what extent are scene categories created from similar affordances? Using a large-­‐scale experiment using hundreds of scene categories, we show that the activities afforded by a visual scene provide a fundamental categorization principle. Affordance-­‐based similarity explained the majority of the structure in human scene categorization patterns, outperforming alternative similarities based on objects or visual features. When all these models are combined, affordances provide the majority of the predictive power in the combined model, and nearly half of the total explained variance is captured only by affordances. These results challenge many existing models of high-­‐level visual perception, and provide immediately testable hypotheses for the functional organization of the human perceptual system. Significance Statement How do we know that a kitchen is a kitchen by looking? Models of visual perception assume that scene identification is facilitated through object recognition. However, these models fail to account for observers' relative insensitivity to local image details. We explore an alternative view that posits that a scene's identity is determined by the possibilities for actions that a scene affords (its affordances). In a large-­‐scale experiment using hundreds of scene categories, we found that human scene similarity ratings were more closely related to affordance-­‐based similarity than to object or visual feature-­‐based models. Combining models revealed that nearly half of the explained variance was captured only by affordances. This work demonstrates that affordances provide a fundamental grouping principle for scenes.