Optimal Universal Parallel Computer, Neural Networks, and Kolmogorov’s Theorem

How can we design the fastest parallel computer architecture that is universal in the sense that it will be able to perform arbitrary computations? To make computations faster, we must divide them into the simplest possible (and thus, fastest possible) processing elements working in parallel. The simplest possible operation with real numbers x1, ..., xn is computing their linear combination. However, if we only have these linear processing elements, we will only be able to compute linear functions. So, to make the computer universal, we also need some nonlinear processing elements that compute nonlinear functions y = f(x1, ..., xn). In general, the greater the number n of inputs, the more time it takes to process them and compute n. So, the simplest nonlinear processing element computes a function of one variable f(x). To get a general computation, we must combine the resulting processing elements. The resulting time of parallel computation increases with the number of layers. So, the fewer layers, the faster the computations. We show that: • with one or two layers, we do not get a universal computer; • for an appropriate three-layer architecture, we get a neural architecture that enables us to approximate an arbitrary non-linear function; we also prove that three layers are not sufficient to exactly represent all non-linear functions; • finally, for four layers, Kolmogorov’s superposition theorem enables us to exactly represent all non-linear functions. We also discuss whether these results remain valid if, in addition to computation time (as described by the number of layers), we take communication time into consideration. 1. AN INFORMAL INTRODUCTION: WHAT IS A UNIVERSAL OPTIMAL PARALLEL COMPUTER Why parallel. Many real-life computational problems (weather prediction, etc) take too long to compute. A well known way to speed up computations is to perform them in parallel by dividing them into smaller and thus faster subtasks that can be performed simultaneously rather than sequentially. The simpler these subtasks, the faster they are performed and therefore, the faster the resulting computation. The efficiency of parallel computations essentially depends on the architecture of the corresponding computer. In this paper, we want to decide what architecture is the most suitable. Universal. In principle, we can choose a specific problem (e.g., weather prediction), and find an architecture that is best suited for this particular problem. However, finding such an architecture is in itself a difficult task. Moreover, there are many different computational