A Reinforcement Learning Approach for Goal-directed Locomotion of a Complex Snake-Like Robot

Snake-like robots conventionally use undulation movements for locomotion [1], [2]. In contrast, a nonstandard snake-like robot having four screw-drive units which are connected serially by three active joints [3] uses propulsion by rotating its screw-drive units to move. Employing this screw-drive mechanism, undulation is not required for its locomotion. This helps it to move in narrow spaces. However, generating its goal-directed locomotion is a challenging control problem. In this work, we tackle the problem by using a reinforcement learning approach, called Policy Improvement using Path Integrals (PI2) [4]. PI2 is numerically simple and has an ability to deal with high dimensional systems. Here, it is used as a model-free learning mechanism to find a proper combination of seven locomotion control parameters (three yaw joint angles and four screw unit angular velocities) of the robot for moving towards a given goal. The learning process is achieved using simulation and the learned parameters are successfully transferred to the real robot. Experiments show that the robot with different body configurations (like, straight-line, zigzag, arc, etc.) can effectively move toward any given goal (see supplementary video at http://www.manoonpong.com /BCCN2014/SnakeRobot.wmv). In this way, a large repertoire of robot behaviors are obtained and used as motor primitives for generating other new behaviors online. By selecting different primitives and properly chaining them along with parameter interpolation and/or sensory feedback techniques, the robot can successfully handle complex tasks like, reaching a single goal or multiple goals while avoiding obstacles and compensating to a change of its body shape. Taken together this study suggests how the PI2 reinforcement learning approach can be used to solve coordination problem of many degrees-of-freedom systems, like the nonstandard snake-like robot, and to generate goal-directed locomotion in a complex environment.