Learning Othello using Cooperative and Competitive Neuroevolution

From early days in computing, making computers play games like chess and Othello with a high level of skill has been a challenging and, lately, rewarding task. As computing power becomes increasingly more powerful, more and more complex learning techniques are employed to allow computers to learn different tasks. Games, however, remain a challenging and exciting domain for testing new techniques and comparing existing ones due to the clearly defined and easily enforced rules, complexity of games and often being fully observable and deterministic. In this thesis we will focus on the game Othello. Othello (also known as Reversi) is an old boardgame being played all over the world by new players and grandmasters alike. Othello is known for being a game which is very easy to learn but hard to master. Due to this nature of the game it is excellent for comparing existing techniques and testing new ones. Nowadays software exists which plays better Othello than the current human world champion. This software is capable of providing such a high level of play by using hard coded knowledge of the game (opening book), look ahead (mid game) and brute force calculations (end game). The goal of this research is to compare techniques in creating a player without the use of any such a priori knowledge. We intend to compare several neuroevolution techniques to random moving players and the more common reinforcement learning techniques of temporal difference learning. Research done for this thesis can be divided into two sections: Comparison between the three neuroevolution techniques, and a comparison between cooperative and competitive learning. For part I three different neuroevolution techniques are compared: SANE, ESP and NEAT. All three use a neural network as function approximator, which is evolved using one of the three techniques. A comparison is done against random moving opponents as well as deterministic (and more skilled) opponents. NEAT emerged as best at learning how to play Othello. Part II is research done to explore the usability of different tournament types for evolving in a competitive way rather than cooperative. Using competitive learning rather than cooperative results in less games needed for evaluation of the same number of players, however information is lost as players pair against other, possibly unskilled, players. ESP is used as neuroevolution-technique. First a standard group tournament is used to test the capabilities of basic tournament training in Othello. Training is done using only learning players in the tournament as well as random moving players and deterministic players. The latter two are added to provide more knowledge into the tournaments. As basic tournaments resulted in less skillful players than was the case with cooperative learning, a more sophisticated tournament type was used: Swiss pairing. Swiss pairing does result in better learning, although still less than with cooperative. Tournament training results in less skilled players than is the case with cooperative learning.