• Less time than an identical iteration of Algorithm 1 if q(t−1) ≤ τi and x i = 0 (the update is skipped) and rr is not updated. Specifically, StingyCD requires O(1) time, while CD requires O(NNZ (Ai)) time. • The same amount of time (up to an O(1) term) as a CD iteration if the update is not skipped and rr is not updated. In particular, both algorithms require the same number of O(NNZ (Ai)) op...

An adaptive back-propagation algorithm is studied and compared with gradient descent (standard back-propagation) for on-line learning in two-layer neural networks with an arbitrary number of hidden units. Within a statistical mechanics framework , both numerical studies and a rigorous analysis show that the adaptive back-propagation method results in faster training by breaking the symmetry bet...

Stochastic gradient descent (SGD) is a ubiquitous algorithm for a variety of machine learning problems. Researchers and industry have developed several techniques to optimize SGD's runtime performance, including asynchronous execution and reduced precision. Our main result is a martingale-based analysis that enables us to capture the rich noise models that may arise from such techniques. Specif...

In this paper it is shown that multi GDBF algorithm exhibits much faster convergence as compared to the single GDBF algorithm. The multi GDBF algorithm require less iterations when compared to the single GDBF algorithm for the search point to closely approach the local maximum point taking into consideration the gradient descent bit flipping (GDBF) algorithms exhibiting better decoding performa...

Several recent empirical studies demonstrate that important machine learning tasks such as training deep neural networks, exhibit a low-rank structure, where most of the variation in loss function occurs only few directions input space. In this paper, we leverage structure to reduce high computational cost canonical gradient-based methods gradient descent (GD). Our proposed Low-Rank Gradient De...

Considering an under supervised 3D space where a group of mobile devices with limited sensing and communicating capabilities are deployed, this paper aims at proposing a decentralized self-deployment algorithm for agents to get maximum connected coverage topology. The problem is modeled as maximization which is solved completely distributed. In fact each agent tries to maximize its sensing volu...

In recent years it has been made more and more clear that the critical issue in gradient methods is the choice of the step length, whereas using the gradient as search direction may lead to very effective algorithms, whose surprising behaviour has been only partially explained, mostly in terms of the spectrum of the Hessian matrix. On the other hand, the convergence of the classical Cauchy stee...