Why Vector-jacobian Products Are Cheap but Preconditioning Is Generally Costly

All methods for solving linear or nonlinear systems of equations require the evaluation of residual vectors, most algorithms are based on Jacobian-vector products and many schemes involve vector-Jacobian products. Moreover, while only exact Newton methods form and factor the Jacobian explicitly, some information about the size of its elements or the structure of its spectrum appears to be indespensible for preconditioning purposes. In this paper we will examine what statements can be made about the relative costs of obtaining these various quantities for a linear or nonlinear mapping between Euclidean spaces. Our only assumption is that this vector function is evaluated by a computer program as a composition of arithmetic operations and univariate algebraic or transcendental functions, like the square root and the exponential. Complete Jacobians are in general an order of magnitude more expensive to obtain than the underlying residual. The complexity ratio between these two mathematical objects depends on the sparsity of the Jacobian. Analytical values for Jacobian-vector products are never much more expensive than residuals, which is clearly the case for their approximation by divided diierences. In fact, sometimes such directional derivatives are signiicantly cheaper, especially if evaluated as a bundle. These observations are relevant for Krylov or block-Krylov methods, irrespective of whether the problem is linear or not. Iterative equation solvers that consistency reduce some xed norm of the residual over successive steps tend to involve vector-Jacobian products. Because evaluating these row-vectors is generally assumed to be quite expensive or at least troublesome for the user considerable eeort has gone into the development of transpose-free variants that require only Jacobian-vector products. The good news part of this talk is that vector-Jacobian products can in principle be obtained at the same operations count as Jacobian-vector products. It is commonly understood that all iterative solvers can be painfully slow if the equations are poorly scaled. Most strategies to detect and remedy that calamity are based on guessing or collecting information about some entries of the Jacobian, speciically its diagonal. It has been shown that evaluating just the trace of the Jacobian is in general almost as expensive as evaluating the complete Jacobian. Obviously, this bad news suggests that preconditioning information cannot be obtained automatically, but must be gleaned from \physical" properties of the underlying system, possibly before its discretization or through a multi-level approach.