A Consensus-Based Global Optimization Method with Adaptive Momentum Estimation
نویسندگان
چکیده
Objective functions in large-scale machine-learning and artificial intelligence applications often live high dimensions with strong non-convexity massive local minima. First-order methods, such as the stochastic gradient method Adam, are used to find global Recently, consensus-based optimization (CBO) has been introduced one of gradient-free methods its convergence is proven dimension-dependent parameters, which may suffer from curse dimensionality. By replacing isotropic geometric Brownian motion component-wise one, latest improvement CBO guaranteed converge minimizer dimension-independent although initial data need be well-chosen. In this paper, based on we propose a adaptive momentum estimation (Adam-CBO). Advantages Adam-CBO include: (1) capable finding minima non-convex objective success rates low costs; (2) can handle non-differentiable activation thus approximate low-regularity better accuracy. The former verified by approximating $1000$ dimensional Rastrigin function $100\%$ rate at cost only growing linearly respect latter confirmed solving machine learning task for partial differential equations solutions where provides results than state-of-the-art Adam. A linear stability analysis provided understand asymptotic behavior method.
منابع مشابه
A novel three-stage distance-based consensus ranking method
In this study, we propose a three-stage weighted sum method for identifying the group ranks of alternatives. In the first stage, a rank matrix, similar to the cross-efficiency matrix, is obtained by computing the individual rank position of each alternative based on importance weights. In the second stage, a secondary goal is defined to limit the vector of weights since the vector of weights ob...
متن کاملA Model Reference Adaptive Search Method for Stochastic Global Optimization
We propose a new method called Stochastic Model Reference Adaptive Search (SMRAS) for finding a global optimal solution to a stochastic optimization problem in situations where the objective functions cannot be evaluated exactly, but can be estimated with some noise (or uncertainty), e.g., via simulation. SMRAS is a generalization of the recently proposed Model Reference Adaptive Search (MRAS) ...
متن کاملA Model Reference Adaptive Search Method for Global Optimization
We introduce a new randomized method called Model Reference Adaptive Search (MRAS) for solving global optimization problems. The method works with a parameterized probabilistic model on the solution space and generates at each iteration a group of candidate solutions. These candidate solutions are then used to update the parameters associated with the probabilistic model in such a way that the ...
متن کاملGlobal optimization of mixed-integer polynomial programming problems: A new method based on Grobner Bases theory
Mixed-integer polynomial programming (MIPP) problems are one class of mixed-integer nonlinear programming (MINLP) problems where objective function and constraints are restricted to the polynomial functions. Although the MINLP problem is NP-hard, in special cases such as MIPP problems, an efficient algorithm can be extended to solve it. In this research, we propose an algorit...
متن کاملA Robust Adaptive Observer-Based Time Varying Fault Estimation
This paper presents a new observer design methodology for a time varying actuator fault estimation. A new linear matrix inequality (LMI) design algorithm is developed to tackle the limitations (e.g. equality constraint and robustness problems) of the well known so called fast adaptive fault estimation observer (FAFE). The FAFE is capable of estimating a wide range of time-varying actuator fault...
متن کاملذخیره در منابع من
با ذخیره ی این منبع در منابع من، دسترسی به آن را برای استفاده های بعدی آسان تر کنید
ژورنال
عنوان ژورنال: Communications in Computational Physics
سال: 2022
ISSN: ['1991-7120', '1815-2406']
DOI: https://doi.org/10.4208/cicp.oa-2021-0144