A Q-learning System for Container Marshalling with Group-Based Learning Model at Container Yard Terminals

This paper addresses scheduling problems on the material handling operation at marine container-yard terminals. The layout, removal order and removal distination of containers are simultaneously optimized in order to reduce the waiting time for a vessel. The schedule of container-movements is derived by autonomous learning method based on a new learning model considering container-groups and corresponding Q-Learning algorithm. In the proposed method, the layout and movements of containers are described based on the Markov Decision Process (MDP), and a state is represented by a container-layout with a selection of a container to be removed or a selection of destination on where the removed container are placed. Then, a state transition arises from a container-movement, a selection of containerdestination, or a selectionh of container to be removed. Only the container-movement takes a cost, and a series of containermovements with selections of destination and order of containers is evaluated by a total amount of costs. As a consequent, the total amount of costs reflects the number of container-movements that is required to achieve desired container-layout. After adequate autonomous learning, the optimum schedule for material handling operation can be obtained by selecting a series of containermovements that has the best evaluation. In the problem, the number of container-arrangements increases by the exponential rate with increase of total count of containers. Therefore, conventional methods have great difficulties to determine desirable movements of containers in order to reduce the run time for shipping.