Efficient First Order Superlinear Algorithms

Substantial number of problems in artificial intelligence requires optimization. Increasing complexity of the problems imposes several challenges on optimization algorithms. The algorithms must be fast, computationally efficient, and scalable. Balance between convergence speed and computational complexity is of central importance. Typical example is the task of training neural networks. Superlinear algorithms are highly regarded for their speed-complexity ratio. With superlinear convergence rates and linear computational complexity they are often the primary choice. Two first order superlinear algorithms are introduced. The algorithms are computationally efficient, convergent, and theoretically justified. They are applied to several neural network training tasks, practically evaluated, and compared to the relevant first order optimization techniques. Results indicate superior performance.