Improved Rates and Asymptotic Normality for Nonparametric Neural Network Estimators

Barron (1993) obtained a deterministic approximation rate (in L2-norm) of r-l12. for a class of single hidden layer feedforward artificial neural networks (ANN) with r hidden units and sigmoid activation functions when the target function satisfies certain smoothness conditions. Hornik, Stinchcombe, White, and Auer (HSWA, 1994) extended Barron's result to a class of ANNs with possibly non-sigmoid activation approximating the target function and its derivatives simultaneously. Recently Makovoz (1996) obtained an improved degree of approximation rate r-(I+I/dYl for Barron's ANNs with sigmoid activation function where d is the dimension of the domain of the target function. When applying Barron's ANNs with sigmoid activation functions to nonparametrically estimate a rcgrcssion function (tl:lC trogc;t), B(UIUIl (1994) ubtalned a roOt mean square convergence rate of Op([n/logn ]-1/4) for a minimum complexity regression estimator with i.i.d. observations, where n is the sample size (number of training examples). Unfortunately, this rate is not fast enough to establish root-n asymptotic normality for plug-in estimates of functionals of the regression function, according to a recent result obtained by Chen and Shen (1996). In this paper, we first obtain an improved approximation rate (in Sobolev norm) of r-l/2-al(d+l), O<a~l, forHSWA's ANNs with possibly non-sigmoid activation functions, where a is related to the choice of activation function. We then obtain a root mean square convergence rate of Op ([n/log(n)]-(I+2a/(d+l»)114(I+ai(d+l»])=Op (n-l/4) for general nonparametric ANN sieve extremum esti-mators, by letting the number of hidden units Tn increase with the sample size n on the order of (r n)2(1 +a/(d+1» lo2(rn)=O (n). Our rate!: are v:llici for i i d 9~ w~1l 9~ for uniform mixing and abgolutely regular (/3-mixing) stationary time series data. Among other things, this rate provides theoretical justification for the popularity of ANN models in fitting multivariate financial data, since many nonlinear financial time series are plausibly modeled as {3-mixing processes. In addition, the rate is fast enough to deliver root-n asymptotic normality for plug-in estimates of smooth functionals using general ANN sieve estimators. As interesting applications to nonlinear time series, we establish rates for ANN sieve estima-tors of three different target functions: a multivariate conditional mean function, a joint density, and a conditional density. We also obtain root-n asymptotic normality results for semipararnetric models and average derivative statistics.-3