论文标题
缺少反历史的事实，NLP对错误信息的不现实核对事实检查
Missing Counter-Evidence Renders NLP Fact-Checking Unrealistic for Misinformation
论文作者
论文摘要
当可信信息受到限制时，在不确定性的时期出现了错误信息。对于基于NLP的事实检查，这是挑战，因为它依赖于对现实，这可能尚不可用。尽管对自动核对事实检查的兴趣越来越大，但尚不清楚自动化方法是否能够现实地反驳有害现实世界中的错误信息。在这里，我们将NLP事实核对与专业核对者进行对比和比较，在没有反证的情况下如何打击错误信息。在我们的分析中，我们表明，通过设计，现有的NLP任务定义进行事实检查不能像专业事实检查者对大多数索赔一样驳斥错误信息。然后，我们定义了两个要求，即数据集中的证据必须实现现实事实检查：必须（1）足以反驳索赔，并且（2）未从现有的事实检查文章中泄漏。我们调查了现有的事实检查数据集，发现所有数据集都无法满足这两个标准。最后，我们执行实验，以证明在大规模事实检查数据集中训练的模型依赖于泄漏的证据，这使得它们在实际情况下不适合。综上所述，我们表明当前的NLP事实检查无法现实地打击现实世界中的错误信息，因为它取决于对数据中反证的不切实际的假设。
Misinformation emerges in times of uncertainty when credible information is limited. This is challenging for NLP-based fact-checking as it relies on counter-evidence, which may not yet be available. Despite increasing interest in automatic fact-checking, it is still unclear if automated approaches can realistically refute harmful real-world misinformation. Here, we contrast and compare NLP fact-checking with how professional fact-checkers combat misinformation in the absence of counter-evidence. In our analysis, we show that, by design, existing NLP task definitions for fact-checking cannot refute misinformation as professional fact-checkers do for the majority of claims. We then define two requirements that the evidence in datasets must fulfill for realistic fact-checking: It must be (1) sufficient to refute the claim and (2) not leaked from existing fact-checking articles. We survey existing fact-checking datasets and find that all of them fail to satisfy both criteria. Finally, we perform experiments to demonstrate that models trained on a large-scale fact-checking dataset rely on leaked evidence, which makes them unsuitable in real-world scenarios. Taken together, we show that current NLP fact-checking cannot realistically combat real-world misinformation because it depends on unrealistic assumptions about counter-evidence in the data.
