Keeping Web Indices up-to-date

Search engines play a crucial role in the Web. Without search engines large parts of the Web becomes inaccessible for the majority of users. Search engines can make new and smaller sites accessible at low cost. Without them, other media, such as Television, would be needed to advertise the existence new site on the Web, only large commercial sites can follow this path. The Web would be endangered to become dominated by a few, well known sites. A crucial problem of search engines is to keep their index up-to-date. Especially if the index grows, the effort needed to update the index increases, since Web documents are dynamic and thus already stored data becomes obsolete. There have been various attempts to monitor the evolve-ment of the Web [1][2]. However, we believe, that change model used in prior work overestimates the rate of change due to an inadequate change model. Our change model has been adapted from the information retrieval field to distinguish index relevant changes from irrelevant modifications in Web documents , e.g. simple spelling corrections or dynamic advertisement links. We have monitored multiple smaller collections of documents over a time period of six month to measure the documents change. Not all changes in Web documents need to be index relevant. E.g. links in the documents might have been updated, some spelling has been improved or the document has been extended with more material of the same kind. Therefore, Web change estimations based on Bit identity, e.g. using checksum, typically overestimate the change in documents [1][2]. We have applied a more refined change model, by using a well understood technique from information retrieval, the vector retrieval model [7][8]. In this model the frequency of occurrence of all words in a document form a vector describing this document. To account for the relative relevancy of words, the words are additionally weighted inversely to their appearance in documents (inverse document frequency). This model has been widely applied in digital library indexing and also in the context of the Web [3][4][5]. The change of two instances is calculated by forming the scalar product of their vectors. The change of two documents is calculated by the following equality in the vector retrieval model. Multiple samples of 10K documents from the Web have been regularly revisited and the change to the original documents has been assessed using the vector retrieval model (Figure 2). The sample has …