论文标题
视觉线索：图像段落字幕的桥接视觉和语言基础
Visual Clues: Bridging Vision and Language Foundations for Image Paragraph Captioning
论文作者
论文摘要
人们说：“一张照片值一千字”。那么，我们如何从图像中获取丰富的信息？我们认为，通过使用视觉线索来桥接大型的视觉基础模型和语言模型，我们可以这样做，而无需任何额外的跨模式训练。得益于基础模型的强大零击功能，我们首先使用视觉基础模型来构建图像的丰富语义表示（例如，图像标签，对象属性 /位置，字幕）作为结构化的文本提示，称为视觉线索。基于视觉线索，我们使用大型语言模型为视觉内容生成一系列综合描述，然后再次通过视觉模型对其进行验证，以选择与图像最合适的候选人。我们通过定量和定性测量评估生成的描述的质量。结果证明了这种结构化语义表示的有效性。
People say, "A picture is worth a thousand words". Then how can we get the rich information out of the image? We argue that by using visual clues to bridge large pretrained vision foundation models and language models, we can do so without any extra cross-modal training. Thanks to the strong zero-shot capability of foundation models, we start by constructing a rich semantic representation of the image (e.g., image tags, object attributes / locations, captions) as a structured textual prompt, called visual clues, using a vision foundation model. Based on visual clues, we use large language model to produce a series of comprehensive descriptions for the visual content, which is then verified by the vision model again to select the candidate that aligns best with the image. We evaluate the quality of generated descriptions by quantitative and qualitative measurement. The results demonstrate the effectiveness of such a structured semantic representation.
