Information gathering by crawlers on the web is of practical interest. We consider a simplified model for crawling complex networks such as the web graph, which is a variation of the robot vacuum edge-cleaning process of Messinger and Nowakowski. In our model, a crawler visits nodes via a deterministic walk determined by their weightings which change during the process deterministically. The mi...

The growths of online Social Networks in scale and amount of information are immense in recent years. The analyses of the structure of online social networks have thus drawn much research interests. Before the analyses, the information and the characteristics of the structure have to be obtained. However, the complexity of today’s web technologies imposes challenges for collecting the data. In ...

Modern social networks have become sources for vast quantities of data. Having access to such big data can be very useful for various researchers and data scientists. In this paper we describe Loklak, an open source distributed peer to peer crawler and scraper for supporting such research on platforms like Twitter, Weibo and other social networks. Social networks such as Twitter and Weibo pose ...

with increase in the size of Web, the search engine relies on Web Crawlers to build and maintain the index of billions of pages for efficient searching. The creation and maintenance of Web indices is done by Web crawlers, the crawlers recursively traverses and downloads Web pages on behalf of search engines. The exponential growth of Web poses many challenges for crawlers.This paper makes an at...

Automatic Web comment detection could significantly facilitate information retrieval systems, e.g., a focused Web crawler. In this paper, we propose a text genre classifier for Web text segments as intermediate step for Web comment detection in Web pages. Different feature types and classifiers are analyzed for this purpose. We compare the two-level approach to state-ofthe-art techniques operat...

Anti-P2P companies have begun to launch Internet attacks against BitTorrent swarms. In this paper, we analyze how successful these attacks are at impeding the distribution of targeted files. We present the results of both passive and active measurements. For our active measurements, we developed a crawler that contacts all the peers in any given swarm, determines whether the swarm is under atta...

Web crawler is a software program that browses WWW in an automated or orderly fashion, and the process is known as web crawling. A web crawler creates the copy of the visited pages so that when required later on, it will index the pages and processing becomes faster. This paper discuss the various techniques of the web crawling through which search becomes faster. In this paper studied has been...