Dual Modality Prompt Tuning for Vision-Language Pre-Trained Model

With the emergence of large pre-trained vison-language model like CLIP, transferable representations can be adapted to a wide range downstream tasks via prompt tuning. Prompt tuning tries probe beneficial information for from general knowledge stored in model. A recently proposed method named Context Optimization (CoOp) introduces set learnable vectors as text language side. However, alone only adjust synthesized "classifier", while computed visual features image encoder not affected , thus leading sub-optimal solutions. In this paper, we propose novel Dual-modality Tuning (DPT) paradigm through learning and prompts simultaneously. To make final feature concentrate more on target concept, Class-Aware Visual (CAVPT) scheme is further our DPT, where class-aware generated dynamically by performing cross attention between patch token embeddings encode both task-related instance information. Extensive experimental results 11 datasets demonstrate effectiveness generalization ability method. Our code available https://github.com/fanrena/DPT.