Amnesic Probing: Behavioral Explanation with Amnesic Counterfactuals

Abstract A growing body of work makes use probing in order to investigate the working neural models, often considered black boxes. Recently, an ongoing debate emerged surrounding limitations paradigm. In this work, we point out inability infer behavioral conclusions from results, and offer alternative method that focuses on how information is being used, rather than what encoded. Our method, Amnesic Probing, follows intuition utility a property for given task can be assessed by measuring influence causal intervention removes it representation. Equipped with new analysis tool, ask questions were not possible before, example, part-of-speech important word prediction? We perform series analyses BERT answer these types questions. findings demonstrate conventional performance correlated importance, call increased scrutiny claims draw or results.1