Aligning Comments to News Articles on a Budget

Disagreement among text annotators as a part of human (expert) labeling process produces noisy labels, which affect the performance supervised learning algorithms for natural language processing. Using only high agreement annotations introduces another challenge: data imbalance problem. We study this challenge within problem relating user comments to content news article. show that traditional techniques from imbalanced data, such oversampling, using weighted loss functions, or assigning weak labels crowdsourcing, may not be sufficient modeling complex temporal relationships between articles and comments. In study, we propose framework aligning (1) imbalanced data characterized with (2) different degrees xmlns:xlink="http://www.w3.org/1999/xlink">annotator agreement , under (3) xmlns:xlink="http://www.w3.org/1999/xlink">constrained budget computing resources. Within framework, Semi-Automatic Labeling solution based on Human-AI collaboration. compare our proposed technique handling synthetic generation xmlns:xlink="http://www.w3.org/1999/xlink">article-comment alignment problem where goal is determine category an article-comment pair represents how relevant comment Finding effective efficient essential because it time-consuming prohibitively costly manually label sufficiently large amount pairs semantic understanding article its discover collaboration outperforms all alternative by 17% accuracy. When there no time budget re-labeling some pairs, found synonym augmentation reasonable alternative. also provide detailed analysis effect humans in loop use unlabeled data.