Polysemy Deciphering Network for Robust Human–Object Interaction Detection

Human–Object Interaction (HOI) detection is important to human-centric scene understanding tasks. Existing works tend assume that the same verb has similar visual characteristics in different HOI categories, an approach ignores diverse semantic meanings of verb. To address this issue, paper, we propose a novel Polysemy Deciphering Network (PD-Net) decodes polysemy verbs for three distinct ways. First, refine features be polysemy-aware through use two modules: namely, Language Prior-guided Channel Attention (LPCA) and Prior-based Feature Augmentation (LPFA). LPCA highlights elements human object appearance each category identified; moreover, LPFA augments pose spatial using language priors, enabling classifiers receive hints reduce intra-class variation Second, introduce Polysemy-Aware Modal Fusion module, which guides PD-Net make decisions based on feature types deemed more according priors. Third, relieve problem sharing semantically categories. Furthermore, expedite research problem, build new benchmark dataset named HOI-VerbPolysemy (HOI-VP), includes common (predicates) have real world. Finally, deciphering verbs, our demonstrated outperform state-of-the-art methods by significant margins HICO-DET, V-COCO, HOI-VP databases. Code data paper are available at https://github.com/MuchHair/PD-Net .