论文标题
熔岩：对比度视频预训练的语言音频视觉对齐
LAVA: Language Audio Vision Alignment for Contrastive Video Pre-Training
论文作者
论文摘要
生成视频数据的表示对于推进机器感知领域至关重要。当前大多数技术都依赖于手工注销的数据，这些数据可能很难使用，生成昂贵且难以扩展。在这项工作中，我们提出了一种基于对比度学习的新颖学习方法，熔岩能够以一种自我监督的方式学习联合语言，音频和视频表示。我们使用变压器编码器在动力学700数据集上预先训练熔岩，以学习每种模式的表示形式。然后，我们证明，熔岩在使用未标记的数据的一小部分时，与当前最先进的自我监督和弱监督预处理技术进行了竞争性能。
Generating representations of video data is of key importance in advancing the field of machine perception. Most current techniques rely on hand-annotated data, which can be difficult to work with, expensive to generate, and hard to scale. In this work, we propose a novel learning approach based on contrastive learning, LAVA, which is capable of learning joint language, audio, and video representations in a self-supervised manner. We pre-train LAVA on the Kinetics 700 dataset using transformer encoders to learn representations for each modality. We then demonstrate that LAVA performs competitively with the current state-of-the-art self-supervised and weakly-supervised pretraining techniques on UCF-101 and HMDB-51 video action recognition while using a fraction of the unlabeled data.
