Visuo-Tactile Servoing for Exploration and Manipulation

We present a hierarchical control framework to allow for robust manipulation and exploration of objects with unknown shape, mass, and friction. The framework is divided into 3 layers: joint-level, tactile servoing, and visual servoing. While the middle layer provides “blind” surface exploration skills, maintaining desired contact patterns, the visual layer monitors and controls the actual object pose providing high-level finger-tip motion commands that are merged with the tactile-servoing control commands. We illustrate the efficiency of the framework using a series of manipulation actions performed with two KUKA LWR arms equipped with tactile sensor arrays as “sensitive fingertips”.