Single-Stream Multi-level Alignment for Vision-Language Pretraining

AbstractSelf-supervised vision-language pretraining from pure images and text with a contrastive loss is effective, but ignores fine-grained alignment due to dual-stream architecture that aligns image representations only on global level. Earlier, supervised, non-contrastive methods were capable of finer-grained alignment, required dense annotations not scalable. We propose single stream language at multiple levels: global, patch-token, conceptual/semantic, using two novel tasks: symmetric cross-modality reconstruction (XMM) pseudo-labeled key word prediction (PSL). In XMM, we mask input tokens one modality use cross-modal information reconstruct the masked token, thus improving between modalities. PSL, attention select keywords in caption, momentum encoder recommend other important are missing caption represented image, then train visual predict presence those keywords, helping it learn semantic concepts essential for grounding textual token an region. demonstrate competitive performance improved data efficiency image-text retrieval, grounding, question answering/reasoning against larger models trained more data. Code available zaidkhan.me/SIMLA.KeywordsVision-language modelingCross-modality learning