A Study of Mobile Agent Platforms for Distributed Web Crawling
نویسندگان
چکیده
In the traditional centralized crawling techniques the pages from all over the web are brought to the search engine side which results a lot of unnecessary Internet traffic. In the distributed crawling with migrating agents, the mobile agent is send to the web server side that brings only required pages to the search engine side which reduces unnecessary overhead. The mobile (migrating) agents are processes capable of roaming a wide area networks, interacting with foreign hosts, gathering information on behalf of its owner and coming back having performed the duties set by its user. The mobile agents acquire the information from the local environment, filter locally and either stored with the agent or forwarded to some receiving destination. In this paper we focus on use of mobile agents in web crawling, various available options of mobile agent technologies and enhancements needed. KeywordsWeb search engine, migrating agents, agent platforms
منابع مشابه
Using Migrating Agents in Designing Web Search Engines and Property Analysis of Available Platforms
Mobile agent is a type of software agent, a composition of computer software and data. It is able to migrate from one computer to another autonomously and continue its execution on the destination computer. When very large volumes of data are stored at remote hosts, these data can be processed in the locality of the data rather than transferring over the network. The main concern is, move the c...
متن کاملOn the Performance of Distributed Search by Mobile Agents
When using mobile agents in distributed search, the overall performance is influenced by several factors. In this paper, we study some of them and their impact. Several experiments were carried out to study the impact of network size, network topology, and the number of agents. These experiments were performed on two well-known mobile agent platforms: AGLET (a commercial environment), and TACOM...
متن کاملDistributed Indexing of the Web Using Migrating Crawlers
Due to the tremendous increase rate and the high change frequency of Web documents, maintaining an up-to-date index for searching purposes (search engines) is becoming a challenge. The traditional crawling methods are no longer able to catch up with the constantly updating and growing Web. Realizing the problem, in this paper we suggest an alternative distributed crawling method with the use of...
متن کاملUCYMICRA: Distributed Indexing of the Web Using Migrating Crawlers
Due to the tremendous increase rate and the high change frequency of Web documents, maintaining an up-to-date index for searching purposes (search engines) is becoming a challenge. The traditional crawling methods are no longer able to catch up with the constantly updating and growing Web. Realizing the problem, in this paper we suggest an alternative distributed crawling method with the use of...
متن کاملMobile Agent Platforms for Web Databases : A Qualitative and Quantitative
In this paper we present practical experiences gathered from the employment of two popular Java-based mobile-agent platforms, IBM's Aglets and Mitsubishi's Concordia. We present some basic distributed computing models and describe their adaptation to the mobile-agent paradigm. Upon these models we develop a set of frameworks for distributed database access over the WorldWide Web, using IBM's Ag...
متن کامل