HADA: A Graph-Based Amalgamation Framework in Image-text Retrieval

Many models have been proposed for vision and language tasks, especially the image-text retrieval task. State-of-the-art (SOTA) in this challenge contain hundreds of millions parameters. They also were pretrained on large external datasets that proven to significantly improve overall performance. However, it is not easy propose a new model with novel architecture intensively train massive dataset many GPUs surpass SOTA already available use Internet. In paper, we compact graph-based framework named HADA, which can combine produce better result rather than starting from scratch. Firstly, created graph structure nodes features extracted edges connecting them. The was employed capture fuse information every model. Then neural network applied update connection between get representative embedding vector an image text. Finally, cosine similarity match images their relevant texts vice versa ensure low inference time. Our experiments show that, although HADA contained tiny number trainable parameters, could increase baseline performance by more $$3.6\%$$ terms evaluation metrics Flickr30k dataset. Additionally, did any only required single GPU due small parameters required. source code at https://github.com/m2man/HADA .