Performance linked Workflow Composition for Video Processing – An Ecological Inspiration

Workflow management within DIET applied to multi-application scheduling Raphael Bolze Ecole Normale Superiere de Lyon The aim of my talk is to present the workflow management system implemented within DIET. DIET is a Network Enabled System built upon a set of distributed schedulers. I will explain the different solutions developed within DIET and how it is applied to the scope of workflow scheduling and execution. More precisely, I will present this feature applied to three bioinformatic applications that validate this work. I will also develop the ideas that are currently explored to deal with multi-workflow application scheduling within DIET. Performance linked Workflow Composition for Video Processing – An Ecological Inspiration Jessica Chen-Burger University of Edinburgh Taiwan has set up a sensor grid to cover 3 of their national parks. These wireless sensors have autonomously collected a large amount of interesting data during the last 4 years. However, little analysis has been done on them. We look into the video and imagery data and seek fully and semi-automation to enable efficient and effective analysis and organisation methods. In this talk, we will discuss the characteristics of the domain and problems that we face. A tentative performance linked workflow method will also be proposed to address the above mentioned problems. Time and Space Optimizations for Executing Scientific Workflows in Distributed Environments Ewa Deelman, USC Information Sciences Institute This talk addresses the problem of efficiently executing scientific workflows in today’s distributed execution environments. We examine the issues in the context of the Pegasus system, which maps high-level workflow descriptions onto distributed resources. The talk is based on our experiences in generating and managing workflows with thousands of tasks that execute on the order of CPU years and process approximately 10 TB of data. We will describe the challenges and opportunities in optimizing the overall workflow runtime and the amount of data storage needed to execute the workflows. We will explore the challenges in the context of several applications, including: astronomy, gravitational-wave physics, and earthquake science. What is the impact of a priori knowledge? Stephen Jarvis University of Warwick This talk reports on recent research that has involved the construction of a predictive performance model for a key biomedical imaging application found as part of the UK eScience IXI (Information eXtraction from Images) project. This code represents a significant challenge for our existing performance prediction tools as it has internal structures that exhibit highly variable runtimes depending on qualities in the input data provided. Since the runtime can vary by more than an order of magnitude, it has been difficult to apply meaningful quality of service criteria to workflows that use this code. Nevertheless, the model developed is used in the context of an interactive workflow scheduling system which provides rapid feedback to the users, allowing them to tailor their workloads to available resources, or to allocate extra resources to scheduled workloads. Optimising the OGSA-DAI Enactment Model Konstantinos Karasavvas National e-Science Centre We are going to introduce the OGSA-DAI workflow and discuss some key differences from traditional workflow systems. We will then describe the changes to the enactment model made for the new version and talk about some implementation details. Finally, at a higher-level we will discuss some scenarios that workflow restructuring would improve performance and some thoughts on automating that procedure. Workflow, Planning and Performance A. Stephen McGough, Imperial College London This talk discusses the pipeline required to take the user’s goal description and map this into a formal workflow description, along the components that will be required, through to selecting where each component of the workflow will be deployed. In order for the planning stage to make appropriate decisions it needs to be aware of information collected from many sources: details of the components to be used provided by the component developer; details of the resources that are available to deploy components onto; performance information which can be used to steer the selection of resources; knowledge about the composition of the components along with domain specific information about the area in which the user is working. Current workflow systems tend to limit themselves to information about the implementation of the components the syntax. However, in this talk we propose the collection of higher level information describing the semantics of components and workflows. This extra information allows the system to reason about the workflow, thus allowing the planner to manipulate the workflow in order to improve the overall performance. Extending grid-based workflow tools with patterns/operators Omer Rana Cardiff University Please follow the link to http://users.cs.cf.ac.uk/David.W.Walker/WOSE/OmerRana.pdf Workflow Scheduling Optimisation: The Case for Revisiting DAG Scheduling Rizos Sakellariou, University of Manchester In many typical cases, a workflow can be represented by a Directed Acyclic Graph (DAG). DAGs have been extensively considered in the context of studying the DAG scheduling problem in homogeneous parallel systems, but less so in the context of heterogeneous systems, which can be regarded as a reasonable abstraction of existing Grid systems. Yet, the systematic study of the properties of workflows and the application of solid scheduling techniques can provide significant improvements in the execution time of a workflow. This talk will start by illustrating the performance benefits that can arise from the application of solid scheduling techniques to workflow execution. It will then consider the workflow (DAG) scheduling problem along the following dimensions: 1. How to build efficient schedules that exploit the structure of the (workflow – DAG) graph. [1] 2. How to deal with run-time changes in the execution time of individual components of the graph. [2] 3. How to maintain Quality of Service in the schedule, when there is competition of multiple applications (graphs) for the resources. [3] 4. How to automate the efficient handling of requirements, such as advance resource reservation. [4] In the talk, findings of Manchester’s previous work on the above topics will behighlighted and supported by extensive simulation results. In summary, our key findingsindicate that: (i) scheduling techniques that make the most efficient exploitation of thestructure of the graph can cope better with run-time changes; (ii) certain levels of Qualityof Service can be obtained without compromising the performance of individualworkflow schedules. The talk will conclude with a number of problems that merit furtherinvestigation. References[1] Rizos Sakellariou, Henan Zhao. A Hybrid Heuristic for DAG Scheduling onHeterogeneous Systems. Proceedings of the 13 IEEE Heterogeneous ComputingWorkshop (HCW’04) (in conjunction with IPDPS 2004), Santa Fe, April 2006, IEEEComputer Society Press, 2004. [2] Rizos Sakellariou, Henan Zhao. A low-cost rescheduling policy for efficientmapping of workflows on grid systems. Scientific Programming, 12(4), December2004, pp. 253-262. [3] Henan Zhao, Rizos Sakellariou. Scheduling Multiple DAGs onto HeterogeneousSystems. Proceedings of the 15 Heterogeneous Computing Workshop (HCW'06) (inconjunction with IPDPS 2006), Rhodes, Apr. 2006, IEEE Computer Society Press. [4] Henan Zhao, Rizos Sakellariou. Advance Reservation Policies for Workflows.Proceedings of the 12 Workshop on Job Scheduling Strategies for Parallel Processing,2006. Supporting the Composition of Distributed Business ProcessesPaulo TraversoITC/IRST, Italy Web services provide a universal basis for the integration of business processes that aredistributed among the most disparate entities, both within an organization and acrossorganizational borders. One of the major research challenges is the automatedcomposition of web services that describe statefull processes, i.e., services that, in orderto be executed, require to follow a given flow of interaction . The ultimate goal is toconstruct tools that provide an automated and transparent support to the entire businessprocess life-cycle, from analysis and execution to their adaptation to a new workingenvironment or emerging business requirements. In the talk, I will discuss some of themain research challenges in this scenario, and present some solutions to these challenges. I will first focus on some solutions to the tasks of analysing, monitoring and composingdistributed business processes. I will then discuss some challenges for future research. Workflow optimisation through service selection David W. Walker, Cardiff University This talk presents an approach to dynamic workflow management and optimisation usingperformance data. We discuss strategies for choosing an optimal service (based on userspecified criteria) from several semantically equivalent Web Services. Such an approachinvolves finding “similar” services, by first pruning the set of discovered services basedon service metadata, and subsequently selecting an optimal service based on datarecorded during prior executions of a service and/or current machine loads. We describethe current implementation of the system, and demonstrate this by a BLAST (used inBioInformatics for Protein-alignment) example by using the Ganglia monitoring tool toget performance data.