Designing a Distributed search engine for Farsi/English web pages

In this paper we have tried to model, design and test a prototype of Farsi/English search engine. The engine has the duty of covering the web media features such as heterogeneity, volatility and huge amount of unstructured worldwide information. These features as well as the rapid advance in technology, challenge the effectiveness of classical Information Retrieval (IR) techniques. Although a growing number of sites with Farsi language support exist, still few research works have been done regarding the computational linguistic approach to this language, particularly in the field of thesaurus construction and stemming. In this paper, we have tried to utilize the past research experiences to design Farsi/English search engine. It seems that Unicode is sufficiently capable of preparing a conclusive environment within this respect specially regarding the issues of indexing and searching web pages. Many common Farsi code-pages are now being used which are to be converted into Unicode, in order to cover most of the existing Farsi web pages. To handle the complexity in analysis and design of the system and generate a visual easy-toscale model Unified Modeling Language (UML) was utilized; and to assure scalability, distributed functionality and reliability, we tried to use some successful industrial solutions: Relational Database in managing the web-page indices and Clustering techniques to balance the high workload on the user interface and index management unit. We’ve chosen Common Object Request Broker Architecture (CORBA) due to distributed object-oriented design of the system and our agent-oriented trends in future. We’ve tried to apply CORBA design ideas to build a scalable and platform-independent framework.