Multi-agent reinforcement learning approaches for distributed job shop scheduling problems

Decentralized decision-making has become an active research topic in artificial intelligence. In a distributed system, a number of individually acting agents coexist. If they strive to accomplish a common goal, i.e. if the multi-agent system is a cooperative one, then the establishment of coordinated cooperation between the agents is of utmost importance. With this in mind, our focus is on multi-agent reinforcement learning methods which allow for automatically acquiring cooperative policies based solely on a specification of the desired joint behavior of the whole system. Research in distributed systems has pointed out that the decentralization of the control of the system and of the observation of the system among independent agents has a significant impact on the complexity of solving a given problem. Therefore, we address the intricacy of learning and acting in multi-agent systems by the following complementary approaches. Many practical problems exhibit some structure whose exploitation may ease the task of finding solutions. For this reason, we are going to identify a subclass of general decentralized decision-making problems that features regularities in the way the agents interact with one another. We will show that the complexity of optimally solving a problem instance from this class is provably lower than solving a general one. Even though a lower complexity class may be entered by sticking to certain subclasses of a general multi-agent problem, the computational complexity may be still so high that optimally solving it is infeasible. This holds, in particular, when intending to tackle problems of larger size that are of relevance for practical problems. Given these facts, our goal will be not to develop optimal solution algorithms that are applicable to small problems only, but to look for techniques capable of quickly obtaining approximate solutions in the vicinity of the optimum. To this end, we will develop and utilize various model-free reinforcement learning approaches. In contrast to offline planning algorithms which aim at finding optimal solutions in a modelbased manner, reinforcement learning allows for employing independently learning agents and, hence, for a full decentralization of the problem. As a matter of fact, many large-scale applications are well-suited to be formulated in terms of spatially or functionally distributed entities. Thus, multi-agent approaches are of high relevance to various real-world problems. Job-shop scheduling is one such application stemming from the field of factory optimization and manufacturing control. It is our particular goal to interpret job-shop scheduling problems as distributed sequential decision-making problems, to employ the multi-agent reinforcement learning algorithms we will propose for solving such problems, and, moreover, to evaluate the performance of our learning approaches in the scope of various established scheduling benchmark problems.