Generalized Algorithms for Least Squares Optimization for Nonlinear Observation Models and Newton’s Method

Many problems in science and engineering must solve nonlinear necessary conditions for defining meaningful solutions. For example, a standard problem in optimization involves solving for the roots of nonlinear functions defined by f(x) = 0, where x is the unknown variable. Classically one develops a first-order Taylor series model which provides the necessary condition that must be iteratively solved. The success of this approach depends on the validity of the assumption that the correction terms are small. Two classes of problems arise: (1) nonsquare systems that lead to least-squares solutions, and (2) square systems that are often handled by Newton-like methods. Both problem classes are sensitive to the accuracy of the starting guess, which impacts the number of iteration cycles required for achieving desired accuracy goals. As problems become more nonlinear, both approaches are generalized by permitting firstthrough fourthorder approximations. Computational differentiation tools are used form computing the partial derivatives. Two solution approaches are presented: (1) a Legendre transformation, and (2) a generalized linear algebra approach for handling tensor equations. Several numerical examples are presented to demonstrate generalized multilinear algebra solution algorithms for Least-Squares and Newton-Raphson Methods. Accelerated convergence rates are demonstrated for scalar and vector root-solving problems. The integration of generalized algorithms and automatic differentiation is expected to have broad potential for impacting the design and use of mathematical programming tools for knowledge discovery applications in science and engineering.