论文标题
对比度搜索是您需要的神经文本生成
Contrastive Search Is What You Need For Neural Text Generation
论文作者
论文摘要
使用自回归语言模型（LMS）生成文本对于许多自然语言处理（NLP）应用非常重要。此任务的先前解决方案通常会产生包含退化表达或缺乏语义一致性的文本。最近，苏等人。基于语言模型的各向同性表示空间，引入了一种新的解码方法，即对比度搜索，并在各种基准上获得了新的艺术状态。另外，Su等。认为自回旋LMS（例如GPT-2）的表示是本质上各向异性的，这也是先前的研究。因此，为了确保语言模型遵循各向同性分布，Su等。提出了一种对比学习方案SIMCTG，该方案通过其他培训来校准语言模型的表示。 在这项研究中，我们首先回答了一个问题：“自回旋的LMS真的各向异性吗？”。为此，我们广泛评估了16种主要语言中LMS的各向同性。出乎意料的是，我们发现各向异性问题仅存在于两个特定的英语GPT-2-MALL/MEDID模型中。另一方面，所有其他评估的LM都是自然的各向同性，这与先前研究得出的结论相反。根据我们的发现，我们进一步评估了使用现成的LMS对16种语言的四代任务进行对比度搜索解码方法。我们的实验结果表明，对比搜索在没有任何其他培训的情况下明显胜过以前的解码方法。更值得注意的是，在16种评估的语言中，有12种对比搜索与人类评估所判断的人类水平的表现相当。我们的代码和其他相关资源可在https://github.com/yxuansu/contrastive_search_is_is_what_you_need上公开获得。
Generating text with autoregressive language models (LMs) is of great importance to many natural language processing (NLP) applications. Previous solutions for this task often produce text that contains degenerative expressions or lacks semantic consistency. Recently, Su et al. introduced a new decoding method, contrastive search, based on the isotropic representation space of the language model and obtained new state of the art on various benchmarks. Additionally, Su et al. argued that the representations of autoregressive LMs (e.g. GPT-2) are intrinsically anisotropic which is also shared by previous studies. Therefore, to ensure the language model follows an isotropic distribution, Su et al. proposed a contrastive learning scheme, SimCTG, which calibrates the language model's representations through additional training. In this study, we first answer the question: "Are autoregressive LMs really anisotropic?". To this end, we extensively evaluate the isotropy of LMs across 16 major languages. Surprisingly, we find that the anisotropic problem only exists in the two specific English GPT-2-small/medium models. On the other hand, all other evaluated LMs are naturally isotropic which is in contrast to the conclusion drawn by previous studies. Based on our findings, we further assess the contrastive search decoding method using off-the-shelf LMs on four generation tasks across 16 languages. Our experimental results demonstrate that contrastive search significantly outperforms previous decoding methods without any additional training. More notably, on 12 out of the 16 evaluated languages, contrastive search performs comparably with human-level performances as judged by human evaluations. Our code and other related resources are publicly available at https://github.com/yxuansu/Contrastive_Search_Is_What_You_Need.
