MDPs for Adjustable Autonomy in Real-World Multi-Agent Environments

Research on adjustable autonomy (AA) is critical if we are to deploy multiagent systems in support of important human activities. Through AA, an agent can dynamically vary its level of autonomy — harnessing human abilities when needed, but also limiting such interaction. While most previous AA work has focused on individual agent-human interactions, this paper focuses on agent teams embedded in human organizations in the context of real-world applications. The need for agent teamwork and coordination in such environments introduces novel AA challenges. In particular, transferring control to human users becomes more difficult, as a lack of human response can cause agent team miscoordination, yet not transferring control causes agents to take enormous risks. Furthermore, despite appropriate individual agent decisions, the agent teams may reach decisions that are completely unacceptable to the human team. We address these challenges by pursuing a two-part decisiontheoretic approach. First, to avoid team miscoordination due to transfer of control decisions, an agent: (i) considers the cost of potential miscoordination with teammates; (ii) does not rigidly commit to a transfer of control decision; (iii) if forced into a risky autonomous action to avoid miscoordination, considers changes in the team’s coordination that mitigate the risk. Second, to ensure effective team decisions, not only individual agents, but also subteams and teams can dynamically adjust their own autonomy. We implement these ideas using Markov Decision Processes, providing a decision-theoretic basis for reasoning about costs and uncertainty of individual and team actions. This approach is central to our deployed multi-agent system, called Electric Elves, that assists our research group in rescheduling meetings, choosing presenters, tracking people’s locations and ordering meals. Introduction Recent exciting, ambitious applications in agent technology involve agents acting individually or in teams in support of critical activities of individual humans or even entire human organizations, in arenas such as intelligent homes (Lesser et al. 1999), “routine” organizational coordination (Pynadath et al. 2000), electronic commerce (Collins et al. 2000), and long-term space missions (Kortenkamp et al. 1999; Dorais et al. 1998). These new applications have raised interest in the development of agents with adjustable autonomy, i.e., agents that dynamically adjust their own level of autonomy based on their situation (Call for Papers 1999). An agent may act with full autonomy, or it may act with reduced or no autonomy, instead transferring decision-making control to a person. The agent must decide to transfer such control at appropriate times, without overly burdening people, while simultaneously harnessing their possibly superior skills or knowledge. Our research aims at a decision-theoretic approach to adjustable autonomy (henceforth AA) in rich environments with agent teams embedded within large-scale human organizations. In such environments, not only do individual agents interact with individual people, but they also coordinate with each other and act jointly in teams. The required teamwork and coordination give rise to novel AA challenges that previous work does not address. Whereas most existing research focuses on the interaction between an individual agent and an individual person, we focus on two key novel challenges in AA: the AA coordination challenge and the AA team-decision challenge. The AA coordination challenge arises in transferring decision-making control. In AA, the problem of when an agent should transfer decision-making control to a human (or vice versa) is already a well-known central problem. In a team setting, the novel challenge is that an agent must transfer control while also avoiding miscoordination with its teammates and while ensuring effective team performance. Techniques from previous AA research on transferring decision-making control fail to address this challenge. For instance, in one existing method, an agent avoids an autonomous decision if it has high uncertainty about the correctness of its decision, and it relies on human input instead (Gunderson & Martin 1999). Applying such a technique in a team setting, an agent may transfer control to a human to reduce decision-making uncertainty, but if the human fails to respond, then the agent may be unable to communicate with its teammates, resulting in costly miscoordination. On the other hand, while an agent’s autonomous decision may avoid miscoordination, it may also be erroneous and jeopardize the overall team activity. Thus, we address the AA coordination challenge by applying decisiontheoretic means to optimally balance possible miscoordination through inaction against possibly erroneous actions. The second AA team-decision challenge arises due to the multiple levels of decision making in teams, i.e., an individual agent’s decision may typically lead to negotiations and decisions within a subteam, and inputs from subteams may lead to decisions at the team level. Since individual human users interact with only individual agents, there is a key difficulty in ensuring effective team decisions. Unfortunately, despite responsible decisions by individual agents, the agent team’s collective decision may still be highly undesirable. Thus, AA must consider the impact of an individual decision at multiple levels of team decision making. Our research in AA addresses the above challenges in the context of a real-world, multi-agent system called Electric Elves (or E-Elves) (Pynadath et al. 2000), supporting everyday human collaboration for several months at USC/ISI. E-Elves assists a group of 12 researchers and one project assistant in their daily activities and provides a unique, exciting opportunity to test our ideas in a deployed team of intelligent agents1. To address the AA coordination challenge, agents in E-Elves explicitly reason about team coordination. In particular, they follow a novel three-step approach: (i) Before transferring decision-making control, an agent explicitly weighs the cost of waiting for user input and any potential team miscoordination against the cost of erroneous autonomous action; (ii) If transferring control is the best option, then an agent does not rigidly commit to this decision (as is often the case in previous work), but rather it flexibly reevaluates, sometimes reversing its decision to ensure team coordination (e.g., if an agent transfers control to a user, but the user fails to act, then the agent may act autonomously to avoid team miscoordination); (iii) Unfortunately, while an agent may appropriately regain control over a decision to avoid miscoordination, it may face significant uncertainty or cost over that decision, so rather than force a risky decision in such situations, an agent may elect to change its coordination arrangements, postponing or reordering activities, to potentially buy time to lower decision cost/uncertainty. Since each such coordination decision and change incurs various costs, agents can look ahead over the different possible sequences of changes in coordination and select the one that maximizes team benefits. With respect to the AA team-decision challenge, two classes of approaches are possible. One approach is to ensure perfect individual agent decisions, such that the (sub)team reaches the desired decision. A second approach is to simplify the individual agent decision, but to introduce E-Elves is a joint project among several research groups at USC/ISI. In this paper, we describe only the role of our own group within the overall project AA at the team level; thus, the agent team may consult a user team if the collective decision involves significant uncertainty, cost, etc. E-Elves relies on the second approach, introducing AA at multiple levels of team decision-making. A key novelty in AA at team-level decision-making is that the transfer-of-control reasoning focuses on collective team features (e.g., majority of team members) rather than specific individual features. We have implemented the above decision-theoretic approach to AA in E-Elves using Markov Decision Processes (MDPs) (Puterman 1994). MDPs allow explicit representation of individual and team decision costs, as well as explicit representation and reasoning about uncertainty of an agent’s state of knowledge or user responses. MDPs also provide a mechanism for decision-theoretic planning that allows agents to choose an optimal policy of actions representing possible coordination changes.