Multi-Time-Scale Markov Decision Processes for Organizational Decision-Making

Decision-makers in organizations and other hierarchical systems interact within and across multiple organizational levels and take interdependent actions over time. The challenge is to identify incentive mechanisms that align agents’ interests and to provide these agents with guidance for their decision processes. To this end, we developed a multiscale decision-making model that combines game theory with multi-time-scale Markov decision processes to model agents’ multi-level, multi-period interactions. For a two-level, multi-period, multi-agent problem, we derived closed-form solutions that determine optimal incentive levels, provide agents with decision rules, and identify information needs. We show the difference between a single-timescale model, where agents make decisions with the same frequency, and a multi-time-scale model, where superior agents make decisions less frequently than their subordinates. Our results show that in the single-time-scale case, agents can make optimal decisions with information merely from the current and subsequent period. In the multi-time-scale case, data from the entire decision horizon are necessary. This paper contributes to multiscale decision theory by generalizing solutions for multi-period interactions that were previously limited to a few time periods. We apply our model to a management problem with an account manager and multiple customer representatives. Subject classification: Multiscale decision theory, Markov decision processes, non-cooperative game theory, management application