论文标题
无监督的多模式神经机器翻译带有伪视觉旋转的
Unsupervised Multimodal Neural Machine Translation with Pseudo Visual Pivoting
论文作者
论文摘要
无监督的机器翻译（MT）最近仅通过单语库来取得令人印象深刻的结果。但是，在潜在空间中关联源目标句子仍然具有挑战性。随着人们会说不同的语言在生物学上具有相似的视觉系统，通过视觉内容实现更好的对齐的潜力是有希望的，但在无监督的多模式MT（MMT）中却没有探索。在本文中，我们研究了如何利用视觉内容来消除歧义并促进无监督的MMT中的潜在空间对齐。我们的模型采用了多模式的背面翻译，并具有伪视觉枢纽，我们可以在其中学习共享的多语言视觉语义嵌入空间，并将视觉上可行的字幕纳入额外的弱监督。广泛使用的Multi30k数据集的实验结果表明，提出的模型在最新方法上显着改善，并在测试时间无法获得图像时会很好地概括。
Unsupervised machine translation (MT) has recently achieved impressive results with monolingual corpora only. However, it is still challenging to associate source-target sentences in the latent space. As people speak different languages biologically share similar visual systems, the potential of achieving better alignment through visual content is promising yet under-explored in unsupervised multimodal MT (MMT). In this paper, we investigate how to utilize visual content for disambiguation and promoting latent space alignment in unsupervised MMT. Our model employs multimodal back-translation and features pseudo visual pivoting in which we learn a shared multilingual visual-semantic embedding space and incorporate visually-pivoted captioning as additional weak supervision. The experimental results on the widely used Multi30K dataset show that the proposed model significantly improves over the state-of-the-art methods and generalizes well when the images are not available at the testing time.
