论文标题
实体开关数据集：审核指定实体识别模型的内域鲁棒性的方法
Entity-Switched Datasets: An Approach to Auditing the In-Domain Robustness of Named Entity Recognition Models
论文作者
论文摘要
命名实体识别系统在包括英语新闻的标准数据集上表现良好。但是，鉴于数据的匮乏，很难就识别多种实体的系统的鲁棒性得出结论。我们提出了一种审核系统内鲁棒性鲁棒性的方法，专门针对由于实体的国籍而引起的绩效差异。我们创建了实体开关数据集，其中原始文本中的命名实体被相同类型的合理命名实体所取代，但具有不同的国籍。我们发现，即使在域内，最新的系统的性能也差异很大：在相同的背景下，来自某些起源的实体比其他地方的实体更可靠地认可。系统在美国和印度实体上表现最佳，并且在越南和印尼实体上最差。这种审计方法可以促进更强大的命名实体识别系统的开发，并允许该领域的研究考虑在其他预测技术工作中受到更高关注的公平标准。
Named entity recognition systems perform well on standard datasets comprising English news. But given the paucity of data, it is difficult to draw conclusions about the robustness of systems with respect to recognizing a diverse set of entities. We propose a method for auditing the in-domain robustness of systems, focusing specifically on differences in performance due to the national origin of entities. We create entity-switched datasets, in which named entities in the original texts are replaced by plausible named entities of the same type but of different national origin. We find that state-of-the-art systems' performance vary widely even in-domain: In the same context, entities from certain origins are more reliably recognized than entities from elsewhere. Systems perform best on American and Indian entities, and worst on Vietnamese and Indonesian entities. This auditing approach can facilitate the development of more robust named entity recognition systems, and will allow research in this area to consider fairness criteria that have received heightened attention in other predictive technology work.
