Script, Language, and Labels: Overcoming Three Discrepancies for Low-Resource Language Specialization

Although multilingual pretrained models (mPLMs) enabled support of various natural language processing in diverse languages, its limited coverage 100+ languages lets 6500+ remain ‘unseen’. One common approach for an unseen is specializing the model it as target, by performing additional masked modeling (MLM) with target corpus. However, we argue that, due to discrepancy from MLM pretraining, a naive specialization such can be suboptimal. Specifically, pose three discrepancies overcome. Script and linguistic related seen hinder positive transfer, which propose maximize representation similarity, unlike existing approaches maximizing overlaps. In addition, label space prediction vary across reinitialize top layers more effective adaptation. Experiments over four different families tasks shows that our method improves task performance statistical significance, while previous fails to.