Why Neural Networks Are Computationally Efficient Approximators: An Explanation

Many real-life dependencies can be reasonably accurately described by linear functions. If we want a more accurate description, we need to take non-linear terms into account. To take nonlinear terms into account, we can either explicitly add quadratic terms to the regression equation, or, alternatively, we can use a neural network with a non-linear activation function. At first glance, regression algorithms would work faster, but in practice, often, a neural network approximation turns out to be a more computationally efficient one. In this paper, we provide a reasonable explanation for this empirical fact. 1 Formulation of the Problem Practical need to find dependencies. In practice, it often occurs that we know (or conjecture) that a quantity y depends on quantities x1, . . . , xn, but we do not know the exact form of this dependence. In such situations, we must experimentally determine this dependence y = f(x1, . . . , xn). For that, in several (S) situations s = 1, . . . , N , we measure the values of both the dependent variable y and of the independent variables xi. Then, we use the results ( x (s) 1 , . . . , x (s) n , y ) of these measurements to find a function f(x1, . . . , xn) which is consistent with all these measurement results, i.e., for which y ≈ f ( x (s) 1 , . . . , x (s) n ) for all s from 1 to S. (The equality is usually approximate since the measurements are approximate and the value y is often only approximately determined by the values of the variables x1, . . . , xn.)