HybridPrompt: Bridging Language Models and Human Priors in Prompt Tuning for Visual Question Answering

Visual Question Answering (VQA) aims to answer the natural language question about a given image by understanding multimodal content. However, quality of most existing visual-language pre-training (VLP) methods is still limited, mainly due to: (1) Incompatibility. Upstream tasks are generally incompatible with downstream answering tasks, which makes knowledge from model not well transferable and greatly limits their performance in few-shot scenarios; (2) Under-fitting. They do integrate human priors compensate for universal models, so as fit challenging VQA problem generate reliable answers. To address these issues, we propose HybridPrompt, cloze- verify-style hybrid prompt framework bridging models tuning VQA. Specifically, first modify input questions into cloze-style prompts narrow gap between upstream task, ensures that can be better transferred subsequent prior-guided tuning. Then, imitate cognitive process brain introduce topic sample related construct dynamic learnable template learning. Finally, add fixed-length free-parameters further enhance generalizability scalability learning model. Experimental results verify effectiveness showing it achieves competitive against previous on widely-used VQAv2 dataset obtains new state-of-the-art results. Our code released at: https://github.com/zhizhi111/hybrid.