About the creation of a parallel bilingual corpora of web-publications

The algorithm of the creation texts parallel corpora was presented. The algorithm is based on the use of "key words" in text documents, and on the means of their automated translation. Key words were singled out by means of using Russian and Ukrainian morphological dictionaries, as well as dictionaries of the translation of nouns for the Russian and Ukrainianlanguages. Besides, to calculate the weights of the terms in the documents, empiric-statistic rules were used. The algorithm under consideration was realized in the form of a program complex, integrated into the content-monitoring InfoStream system. As a result, a parallel bilingual corpora of web-publications containing about 30 thousand documents, was created 1. Introduction The algorithms of singling out so-called "key words" have important functions in both theory and practice. Many algorithms of singling out key words are based in vector representation and they use statistic properties of the texts. Frequency word lists in one or several languages are mostly used in singling out key words (or word bases). The creation of a frequency word list based on the morphological dictionary (MD) using a text corpora of the documents is described in this paper, as well as the development of the algorithm of singling out key words with the use of frequency MD and a well-known approach TF IDF [1]. Based on the analysis of the key words, which were automatically singled out, and their translation into another language, the procedure of identification of duplicate documents, presented in various languages, was realized. As it is well-known, at present the task of creating multilingual parallel text bodies is very relevant [2-4]. A suggested approach made it possible to create a bilingual Ukrainian-Russian parallel corpora of the texts from web-publications in Russian and Ukrainian languages. According to the experts, the estimated accuracy of the suggested algorithm is 98 %. 2. Description of the algorithm The following procedures were used to create the parallel texts corpora:-development of MD;-creation of frequency dictionaries on the basis of existing MDs;-creation of the translation dictionaries;-realization of the algorithm of singling out key words in the document;-translation of key words of the document into another language;-realization of the algorithm of identifying duplicates based on the analysis of key words and their translations. 2.1. Making morphological dictionaries Available electronic dictionaries were taken for Russian and Ukrainian languages (ispell with over 1.102 thousand word forms in the Ukrainian language and Zalizniak's dictionary which …