The Primal-Dual hybrid gradient (PDHG) method is a powerful optimization scheme that breaks complex problems into simple sub-steps. Unfortunately, PDHG methods require the user to choose stepsize parameters, and the speed of convergence is highly sensitive to this choice. We introduce new adaptive PDHG schemes that automatically tune the stepsize parameters for fast convergence without user inp...

We propose a stepsize adaptation scheme for stochastic gradient descent. It operates directly with the loss function and rescales the gradient in order to make fixed predicted progress on the loss. We demonstrate its capabilities by strongly improving the performance of Adam and Momentum optimizers. The enhanced optimizers with default hyperparameters consistently outperform their constant step...

In this work, we present a new proposal for the second-order Adaptive Sigma Delta Modulation (ASDM). The new proposed Adaptation scheme is based on using Operational Transconductance Amplifier (OTAs) as an Integrator and as an Amplifier to adapt the quantizer stepsize to control the voltage gain by feedback the quantizer output through adaptation scheme. The stepsize is changing up or down by t...

It is widely accepted that the stepsize of great significance to gradient method. An efficient method with approximately optimal stepsizes mainly based on regularization models proposed for unconstrained optimization. More specifically, if objective function not close a quadratic line segment between current and latest iterates, model exploited carefully generate stepsize. Otherwise, approximat...