Usage of Dedicated Data Structures for URL Databases in a Large-Scale Crawling

Within the beginning of the Internet there was always a need for an automatic browsing its resources for many purposes like: indexing, cataloguing, validating, monitoring, etc. Because of a todays large volume of the World Wide Web the term Internet is often wrongly identified with the single HTTP protocol service. The process of browsing the World Wide Web in automated manner is called crawling very likely by analogy between traversing the WWW using URL anchors1 in the HTML pages to a bug crawling. A crawling process is realised by dedicated software named crawlers or robots. Because of a large scope of possible applications, crawlers can vary with architecture,