The Recognition Method of Unknown Chinese Words in Fragments Based on Mutual Information

This paper presents a method of using mutual information to improve the recognition algorithm of unknown Chinese words, it can resolve the complexity of weight settings and the increasing garbage strings caused by the omnisegmentation of fragments that affected the efficiency of unknown Chinese words recognition existed in the literature[7]. The process of the method is as following: first, segment the text, and then segment the fragments that get in the first step to generate a temporary dictionary, then use rules and frequency information to calculate the mutual information of every string in the temporary dictionary. Finally, the greedy algorithm is used to obtain the longest path of each fragment, so to abstract the unknown Chinese words in the fragments.