论文标题
多语言BERT中跨语性能力和语言特定信息的研究
A Study of Cross-Lingual Ability and Language-specific Information in Multilingual BERT
论文作者
论文摘要
最近，多语言BERT在跨语性转移任务上非常出色，优于静态非上下文的单词嵌入。在这项工作中，我们提供了一项深入的实验研究，以补充现有的跨语性能力文献。我们比较了非上下文和上下文化表示模型与相同数据的跨语性能力。我们发现数据尺寸和上下文窗口大小是可转让性的关键因素。我们还观察到多语言BERT中的语言特定信息。通过操纵潜在表示，我们可以控制多语言BERT的输出语言，并实现无监督的令牌翻译。我们进一步表明，基于观察结果，有一种计算廉价但有效的方法来提高多语言BERT的跨语性能力。
Recently, multilingual BERT works remarkably well on cross-lingual transfer tasks, superior to static non-contextualized word embeddings. In this work, we provide an in-depth experimental study to supplement the existing literature of cross-lingual ability. We compare the cross-lingual ability of non-contextualized and contextualized representation model with the same data. We found that datasize and context window size are crucial factors to the transferability. We also observe the language-specific information in multilingual BERT. By manipulating the latent representations, we can control the output languages of multilingual BERT, and achieve unsupervised token translation. We further show that based on the observation, there is a computationally cheap but effective approach to improve the cross-lingual ability of multilingual BERT.
