Lexical and semantic clustering by Web links

Recent Web searching and mining tools are combining text and link analysis to improve ranking and crawling algorithms. The central assumption behind such approaches is that there is a correlation between the graph structure of the Web and the text and meaning of pages. Here I formalize and quantitatively validate two conjectures drawing connections from linkage information to lexical and semantic Web content. The link-content conjecture states that a page is similar to the pages that link to it, and the link-cluster conjecture that pages about the same topic are clustered together. These results shed light on the connection between the success of the latest Web mining techniques and the small world topology of the Web. From a normative perspective, the present analysis suggests that smart Web crawlers can take advantage of local text and link clues to locate relevant pages in a more efficient and scalable way than current search engine robots.