Statistical / Rule - based Hybrid Phrase Break

In this paper, we present a new phrase break detection architecture that integrates proba-bilistic approach with rule-based error correction. The architecture consists of a probabilis-tic phrase break detector and a transformational rule-based post error corrector. The probabilistic method alone usually suuers from performance degradation due to inherent data sparseness problems. So we adopted transformational rule-based error correction to overcome these training data limitations. The probabilistic phrase break detector segments the POS sequences into several phrases according to word trigram probabilities. The probabilis-tic phrase break detection only covers a limited range of contextual information. Moreover, the module does not see the morpheme tag selectively and relative distance to the other phrase breaks. The initially phrase break tagged morpheme sequence is corrected with error correcting rules. The rules are learned by comparing the correctly tagged phrase break corpus with the output of the probabilistic phrase break detector. The rule-based post error correction provided more accurate results even with the phrase break detector that has poor performance. Our main contributions include presenting transformational rule-based error correction for phrase break detection. Also, probabilistic phrase break detection was implemented as an initial annotator of the transformational rule-based error correction. The architecture can provide accurate results even with the phrase break detector that has poor initial performance. Moreover, the system can be exibly tuned to new corpus without massive retraining .