The Universal Approximation Property

Abstract The universal approximation property of various machine learning models is currently only understood on a case-by-case basis, limiting the rapid development new theoretically justified neural network architectures and blurring our understanding current models’ potential. This paper works towards overcoming these challenges by presenting characterization, representation, construction method, an existence result, each which applies to any approximator most function spaces practical interest. Our characterization result used describe activation functions allow feed-forward architecture maintain its capabilities when multiple constraints are imposed final layers remaining sparsely connected. These include rescaled shifted Leaky ReLU but not function. representation exhibit simple modification architecture, can approximate continuous with non-pathological growth, uniformly entire Euclidean input space. improves known architecture.