Ending-based Strategies for Part-of-Speech Tagging

Probabilistic approaches to part-of-speech tagging rely primarily on whole-word statis­ tics about word/tag combinations as well as contextual information. But experience shows about 4 per cent of tokens encountered in test sets are unknown even when the train­ ing set is as large as a million words. Unseen words are tagged using secondary strategies that exploit word features such as endings, capitalizations and punctuation marks. In this work, word-ending statistics are pri­ mary and whole-word statistics are sec­ ondary. First, a tagger was trained and tested on word endings only. Subsequent ex­ periments added back whole-word statistics for the N words occurring most frequently in the training set. As N grew larger, per­ formance was expected to improve, in the limit performing the same as word-based tag­ gers. Surprisingly, the ending-based tag­ ger initially performed nearly as well as the word-based tagger; in the best case, its per­ formance significantly exceeded that of the word-based tagger. Lastly, and unexpect­ edly, an effect of negative returns was ob­ servedas N grew larger, performance gen­ erally improved and then declined. By vary­ ing factors such as ending length and tag-list strategy, we achieved a success rate of 97.5 percent.