Asking for Help with the Right Question by Predicting Human Visual Performance

In this paper, we consider robotic surveillance tasks that involve visual perception. The robot has a limited access to a remote operator to ask for help. However, humans may not be able to accomplish the visual task in many scenarios, depending on the sensory input. In this paper, we propose a machine learning-based approach that allows the robot to probabilistically predict human visual performance for any visual input. Based on this prediction, we then present a methodology that allows the robot to properly optimize its field decisions in terms of when to ask for help, when to sense more, and when to rely on itself. The proposed approach enables the robot to ask the right questions, only querying the operator with the sensory inputs for which humans have a high chance of success. Furthermore, it allows it to autonomously locate the areas that need more sensing. We test the proposed predictor on a large validation set and show Normalized Mean Square Error of 0.0199, as well as a reduction of about an order of magnitude in error as compared to the state-of-the-art. We then run a number of robotic surveillance experiments on our campus as well as a larger-scale evaluation with real data/human feedback in a simulation environment. The results showcase the efficacy of our approach, indicating a considerable increase in the success rate of human queries (a few folds in several cases) and the overall performance (30%-41% increase in success rate).