UNLT: Urdu Natural Language Toolkit

Abstract This study describes a Natural Language Processing (NLP) toolkit, as the first contribution of larger project, for an under-resourced language—Urdu. In previous studies, standard NLP toolkits have been developed English and many other languages. There is also dire need text processing tools methods Urdu, despite it being widely spoken in different parts world with large amount digital readily available. presents version UNLT (Urdu Toolkit) which contains three key required Urdu pipeline; word tokenizer, sentence part-of-speech (POS) tagger. The tokenizer employs morpheme matching algorithm coupled state-of-the-art stochastic n -gram language model back-off smoothing characteristics space omission problem. insertion problem compound words tackled using dictionary look-up technique. combination various machine learning, rule-based, regular-expressions, techniques. Finally, POS taggers are based on Hidden Markov Model Maximum Entropy-based addition, we gold training testing data sets to improve evaluate performance new techniques tokenization, tagging. For comparison purposes, compared proposed approaches several methods. Our UNLT, sets, supporting resources all free publicly available academic use.