A Lightweight Data Integration Architecture

Content syndication specifications such as Atom have become a popular mechanism to disseminate information across the Internet, with many sites providing Atom feeds for users to subscribe to and consume. Such a scenario typifies the originally intended use of Atom; however, our research has explored an alternative domain for this syndication technology. This research has evaluated Atom for its potential as a lightweight platform to support data integration from a set of data sources to a single target database. The implementation of the Atom-based architecture that we developed for this research combines freely available server-side scripting technology with the simplified asynchronous connection scheme that content syndication technology offers. We use several use cases each with different degrees of complexity, yet sharing common requirements, as a guide in the development of our prototype. In order to evaluate our Atom-based architecture, our experimental design required the construction of an evaluation framework that measured the prototype’s impact upon the network and computation resources it consumed. These measurements were compared with observations of response time requirements between operational and analytical processing systems. The experiments carried out to evaluate the Atom-based data integration architecture have shown that the architecture has potential in facilitating a lightweight data integration solution. Our research has shown that an Atom-based architecture is capable of operating within a range of conditions and environments, and with further development, would be capable of greater processing efficiency and wider compatibility with other types of data structures.