BANANAS: Bayesian Optimization with Neural Architectures for Neural Architecture Search

Over the past half-decade, many methods have been considered for neural architecture search (NAS). Bayesian optimization (BO), which has long had success in hyperparameter optimization, recently emerged as a very promising strategy NAS when it is coupled with predictor. Recent work proposed different instantiations of this framework, example, using networks or graph convolutional predictive model within BO. However, analyses these papers often focus on full-fledged algorithm, so difficult to tell individual components framework lead best performance. In work, we give thorough analysis "BO + predictor framework" by identifying five main components: encoding, predictor, uncertainty calibration method, acquisition function, and function optimization. We test several each component also develop novel path-based encoding scheme architectures, show theoretically empirically scales better than other encodings. Using all our analyses, final algorithm called BANANAS, achieves state-of-the-art performance spaces. adhere research checklist (Lindauer Hutter 2019) facilitate practices, code available at https://github.com/naszilla/naszilla.