论文标题
通过视觉概念和分层对齐方式进行有效的视觉预测
Efficient Vision-Language Pretraining with Visual Concepts and Hierarchical Alignment
论文作者
论文摘要
远见和语言预测已成为解决多模式下游任务的普遍方法。当前的趋势是朝着更大的模型和预处理数据集迈进。从长远来看，这一计算次数的冲动似乎是不合理的，而是朝着可持续的解决方案迈进，并且事实上排除了资源有限的学术实验室。 In this work, we propose a new framework, dubbed ViCHA, that efficiently exploits the input data to boost the learning by: (a) a new hierarchical cross-modal alignment loss, (b) new self-supervised scheme based on masked image modeling, (c) leveraging image-level annotations, called Visual Concepts, obtained with existing foundation models such as CLIP to boost the performance of the image encoder.尽管对数据的预估计少了四倍，但我们的VICHA策略在下游任务上的其他方法（例如图像文本检索，VQA，视觉推理，视觉效果和视觉接地）都优于其他方法。该代码将在此处公开提供：https：//github.com/mshukor/vicha
Vision and Language Pretraining has become the prevalent approach for tackling multimodal downstream tasks. The current trend is to move towards ever larger models and pretraining datasets. This computational headlong rush does not seem reasonable in the long term to move toward sustainable solutions, and de facto excludes academic laboratories with limited resources. In this work, we propose a new framework, dubbed ViCHA, that efficiently exploits the input data to boost the learning by: (a) a new hierarchical cross-modal alignment loss, (b) new self-supervised scheme based on masked image modeling, (c) leveraging image-level annotations, called Visual Concepts, obtained with existing foundation models such as CLIP to boost the performance of the image encoder. Although pretrained on four times less data, our ViCHA strategy outperforms other approaches on several downstream tasks such as Image-Text Retrieval, VQA, Visual Reasoning, Visual Entailment and Visual Grounding. The code will be made publicly available here: https://github.com/mshukor/ViCHA
