Matching Visual Features to Hierarchical Semantic Topics for Image Paragraph Captioning

Observing a set of images and their corresponding paragraph-captions, challenging task is to learn how produce semantically coherent paragraph describe the visual content an image. Inspired by recent successes in integrating semantic topics into this task, paper develops plug-and-play hierarchical-topic-guided image generation framework, which couples extractor with deep topic model guide learning language model. To capture correlations between text at multiple levels abstraction from images, we design variational inference network build mapping features textual captions. generation, learned hierarchical are integrated model, including Long Short-Term Memory (LSTM) Transformer, jointly optimized. Experiments on public datasets demonstrate that proposed models, competitive many state-of-the-art approaches terms standard evaluation metrics, can be used both distill interpretable multi-layer generate diverse We release our code https://github.com/DandanGuo1993/VTCM-based-image-paragraph-caption.git