Grid Data Streaming

Flourish development of grid computing have been seen in recent years which has enabled researchers to collaborate more efficiently by sharing computing, storage, and instrumental resources. Data grids focusing on large-scale data sharing and processing are most popular and data management is one of most essential functionalities in a grid software infrastructure. Applications such as astronomical observations, large-scale numerical simulation, and sensor networks generate more and more data, which constitutes great challenges to storage and processing capabilities. Most of these data intensive applications can be considered as data stream processing with fixed processing patterns and periodical looping. Grid data streaming management is gaining more and more attention in the grid community. In this work, a detailed survey of current grid data streaming research efforts is provided and features of corresponding implementations are summarized. While traditional grid data management systems provide functions like data transfers, placements and locating, data streaming in a grid environment requires additional supports, e.g. data cleanup and transfer scheduling. For example, at storage-constraint grid nodes, data can be streamed, made available to corresponding applications in an on-demand manner, and finally cleaned up after processing is completed. Grid data streaming management is particularly essential to enable grid applications on CPU-rich but storage-limit grid nodes. In this work, a grid data streaming environment is proposed with detailed system analysis and design. Several additional modules, e.g. performance sensors, predictors and schedulers, are implemented. Initial experimental results show that data streaming leads to a better utilization of data storage and improves system performance significantly. Key Words—Grid computing, data streams, and data streaming applications. * E-mail: [email protected]. This work is funded by the Ministry of Education of China under the quality engineering program for higher education and the Ministry of Science and Technology of China under the national 863 high-tech R&D program (grant No. 2006AA10Z237). Wen Zhang, Junwei Cao, Lianchen Liu, and Cheng Wu 2