Continually Learning Optimal Web Service Compositions∗

Open service-oriented systems which autonomously and continually satisfy users’ service requests to optimal levels are an appropriate response to the need for increased automation of information systems. Given a service request, an open service-oriented system interprets the functional and nonfunctional requirements laid out in the request and identifies the optimal WS composition— that is, identifies web services (WS) whose coordinated execution optimally satisfies the various requirements laid out in the service request. The pool of services in such a system has three important characteristics: first, many WS are usually capable of executing the same functional requirements, so that it is necessary to select among competing WS based on their nonfunctional (i.e., quality) characteristics; second, new services may become available and others unavailable, so that there is no guarantee that a composition optimal at some point in time subsequently remains such—some of the needed services may be unavailable, or some new services may be available and more appropriate for a given request; finally, there is no guarantee that a service will execute as its provider advertises. Consequenty, when producing service compositions it is relevant to: (1) revise WS compositions as new WS appear and other WS become unavailable; (2) use multiple criteria, including nonfunctional ones to choose among competing WS; (3) base WS comparisons on observed, instead of advertised performance; and (4) allow for uncertainty in the outcome of WS executions. To address issues (1)–(4), we propose the Multi-Criteria Randomized Reinforcement Learning (MCRRL) approach to WS composition. MCRRL learns and revises WS compositions using a novel multicriteria-driven (including various quality of service parameters, deadline, reputation, cost, and user preferences) reinforcement learning algorithm, which integrates the exploitation of acquired data about individual services’ past performance with optimal, undirected, continual exploration of new compositions which involve services whose behavior has not been observed. The MCRRL enables adaptability to actual performance of WS and changes in their availability. The reported experiments indicate the algorithm behaves as expected and outperforms two standard approaches. ∗Preliminary versions of this paper [68] appear in Proceedings of the IEEE 2007 International Conference on Web Services (ICWS’07) [68] and Proceedings of the 2007 International Conference on Autonomous Agents and Multiagent Systems (AAMAS’07) [69].