Benchmarks for Robotic Soccer Vision

Robotic soccer vision has been a major research problem in RoboCup and, even though many progresses have been made so that, for example, games now can run without many constraints on the lighting conditions, the problem has not been completely solved and on-site camera calibration is always a major activity for RoboCup soccer teams. While different robotic soccer vision and object perception techniques continue to appear in the RoboCup Soccer League, there is a lack of quantitative evaluation of existing methods. Since we believe that a quantitative evaluation of soccer vision algorithms will led to significant advances in the performance on perception and on the entire soccer task, in this paper we propose a benchmarking methodology for evaluating robotic soccer vision systems. We discuss the main issues of a successful benchmarking methodology: (i) a large and complete data base or data sets with ground truth; (ii) a public repository with data sets, algorithms and implementations that can be dynamically updated and (iii) evaluation metrics, error functions and comparison results.