We introduce a class of variational actor-critic algorithms based on formulation over both the value function and policy. The objective consists two parts: one for maximizing other minimizing Bellman residual. Besides vanilla gradient descent with policy updates, we propose variants, clipping method flipping method, in order to speed up convergence. also prove that, when prefactor residual is s...

This paper discusses convergence issues when training adaptive critic designs (ACD) to control dynamic systems expressed as Markov sequences. We critically review two published convergence results of critic-based training and propose to shift emphasis towards more practically valuable convergence proofs. We show a possible way to prove convergence of ACD training.

A variety of methods for developing quasi-optimal intelligent control systems using reinforcement learning techniques based on adaptive critics have appeared in recent years. This paper reviews the family of approximate dynamic programming techniques based on adaptive critic methods and introduces a new hybrid critic training method.

An intelligent robot is a remarkably useful combination of a manipulator, sensors and controls. The use of these machines in factory automation can improve productivity, increase product quality and improve competitiveness. This paper presents a discussion of recent and future technical and economic trends. During the past twenty years the use of industrial robots that are equipped not only wit...

This paper presents a model-based actorcritic algorithm in continuous time and space. Two function approximators are used: one learns the policy (the actor) and the other learns the state-value function (the critic). The critic learns with the TD(λ) algorithm and the actor by gradient ascent on the Hamiltonian. A similar algorithm had been proposed by Doya, but this one is more general. This al...