Adaptive Natural Gradient Learning Based on Riemannian Metric of Score Matching

The natural gradient is a powerful method to improve the transient dynamics of learning by considering the geometric structure of the parameter space. Many natural gradient methods have been developed with regards to Kullback-Leibler (KL) divergence and its Fisher metric, but the framework of natural gradient can be essentially extended to other divergences. In this study, we focus on score matching, which is an alternative to maximum likelihood learning for unnormalized statistical models, and introduce its Riemannian metric. By using the score matching metric, we derive an adaptive natural gradient algorithm that does not require computationally demanding inversion of the metric. Experimental results in a multi-layer neural network model demonstrate that the proposed method avoids the plateau phenomenon and accelerates the convergence of learning compared to the conventional stochastic gradient descent method. 1 SCORE MATCHING AND ITS RIEMANNIAN METRIC Score matching has been developed for training unnormalized statistical models and applied to various kinds of practical applications such as signal processing (Hyvärinen, 2005) and representation learning for visual and acoustic data (Köster & Hyvärinen, 2010). We can also train single-layer models (Swersky et al., 2011; Vincent, 2011) and a two-layer model with the analytically intractable normalization constants (Köster & Hyvärinen, 2010), which are hard to train by maximum likelihood learning. The objective function of score matching is given by the squared distance between derivatives of the log-density, DSM [q : p] = ∫ dxq(x) ∑ i |∂i log q(x) − ∂i log p(x)|, where we denote the derivative with respect to the i-th probability variable as a partial derivative symbol ∂i = ∂ ∂xi . In this paper, we refer to this objective function as score matching (SM) divergence. In general, we can derive the Riemannian structure from any divergence (Eguchi, 1983; Amari, 2016). Let us consider a parametric probability distribution p(x; ξ). When we estimate the parameter ξ with a divergence D[q : p], its parameter space has the Riemannian metric matrix G defined by D[p(x; ξ) : p(x; ξ + dξ)] = ∑ i,j Gijdξidξj . The metric matrix G can be obtained by the second derivative, Gij = ∂ 2 ∂ξ′ i∂ξ ′ j D[p(x; ξ) : p(x; ξ′)] ∣∣ ξ′=ξ . In particular, when we consider the SM divergence, its metric becomes the following positive semi-definite matrix,