Introducing ReQuEST: an Open Platform for Reproducible and Quality-Efficient Systems-ML Tournaments

Co-designing efficient machine learning based systems across the whole hardware/software stack to trade off speed, accuracy, energy and costs is becoming extremely complex and time consuming. Researchers often struggle to evaluate and compare different published works across rapidly evolving software frameworks, heterogeneous hardware platforms, compilers, libraries, algorithms, data sets, models, and environments. We 1 present our community effort to develop an open co-design tournament platform with an online public scoreboard. It will gradually incorporate best research practices while providing a common way for multidisciplinary researchers to optimize and compare the quality vs. efficiency Pareto optimality of various workloads on diverse and complete hardware/software systems. We want to leverage the open-source Collective Knowledge framework and the ACM artifact evaluation methodology to validate and share the complete machine learning system implementations in a standardized, portable, and reproducible fashion. We plan to hold regular multi-objective optimization and co-design tournaments for emerging workloads such as deep learning, starting with ASPLOS’18 (ACM conference on Architectural Support for Programming Languages and Operating Systems the premier forum for multidisciplinary systems research spanning computer architecture and hardware, programming languages and compilers, operating systems and networking) to build a public repository of the most efficient algorithms and systems which can be easily reused and built upon. We will also use the feedback from participants to continue improving our platform and common co-design methodology. 1ReQuEST organizers (A-Z): Luis Ceze, University of Washington (USA), Natalie Enright Jerger (University of Toronto, Canada), Babak Falsafi (EPFL, Switzerland), Grigori Fursin, (cTuning foundation, France), Anton Lokhmotov, (dividiti, UK), Thierry Moreau, (University of Washington, USA), Adrian Sampson, (Cornell University, USA) Phillip Stanley Marbell, (University of Cambridge, UK)