An Approximate Lp-Di erence Algorithm for Massive Data Streams

Several recent papers have shown how to approximate the diierence P i jai ? bi j or P jai ? bi j 2 between two functions, when the function values ai and bi are given in a data stream, and their order is chosen by an adversary. These algorithms use little space (much less than would be needed to store the entire stream) and little time to process each item in the stream and approximate with small relative error. Using diierent techniques, we show how to approximate the L p-diierence P i jai ?bij p for any rational-valued p 2 (0; 2], with comparable eeciency and error. We also show how to approximate P i jai ? bi j p for larger values of p but with a worse error guarantee. Our results ll in gaps left by recent work, by providing an algorithm that is precisely tunable for the application at hand. These results can be used to assess the diierence between two chronologically or physically separated massive data sets, making one quick pass over each data set, without buuering the data or requiring the data source to pause. For example, one can use our techniques to judge whether the traac on two remote network routers are similar without requiring either router to transmit a copy of its traac. A web search engine could use such algorithms to construct a library of small \sketches," one for each distinct page on the web; one can approximate the extent to which new web pages duplicate old ones by comparing the sketches of the web pages. Such techniques will become increasingly important as the enormous scale, distributional nature, and one-pass processing requirements of data sets become more commonplace.