Data Stream Warehousing In Tidalrace

Big data is a ubiquitous feature of large modern enterprises. Many organizations generate huge amounts of on-line streaming data – examples include network monitoring, Twitter feeds, financial data, and industrial application monitoring. Making effective use of these data streams can be challenging. While Data Stream Management Systems can provide support for realtime alerting and data reduction, many applications require complex analytics on a data history to best make use of the streams. We have been developing technologies for data stream warehousing, starting with the DataDepot [13] system. A data stream warehouse continually ingests data streams, computes complex derived data products, and stores long (perhaps yearslong) histories. To take advantage of new technologies, we have developed a next-generation data stream warehousing system. In this paper we describe the Tidalrace system, our motivations for developing it, and architectural features of Tidalrace that support data stream warehousing.