论文标题
ISEA：NLP模型语义错误分析的交互式管道
iSEA: An Interactive Pipeline for Semantic Error Analysis of NLP Models
论文作者
论文摘要
NLP模型中的错误分析对于成功的模型开发和部署至关重要。诊断错误的一种常见方法是识别模型产生最多错误的数据集中的亚群。但是，现有方法通常基于预定义的特征定义亚群，这要求用户提前形成错误的假设。为了补充这些方法，我们提出了ISEA，这是NLP模型中语义误差分析的交互式管道，该管道自动发现在人类互动系统中具有高误差率的语义界面亚群。 ISEA使模型开发人员能够通过发现的子群来了解更多有关其模型错误的信息，从而通过对发现的子群的交互式分析来验证错误源，并通过定义自定义子群来检验有关模型错误的假设。该工具支持在令牌和概念级别上易用错误亚群的语义描述，以及预定的高级特征。通过用例和专家访谈，我们演示了ISEA如何有助于理解和分析。
Error analysis in NLP models is essential to successful model development and deployment. One common approach for diagnosing errors is to identify subpopulations in the dataset where the model produces the most errors. However, existing approaches typically define subpopulations based on pre-defined features, which requires users to form hypotheses of errors in advance. To complement these approaches, we propose iSEA, an Interactive Pipeline for Semantic Error Analysis in NLP Models, which automatically discovers semantically-grounded subpopulations with high error rates in the context of a human-in-the-loop interactive system. iSEA enables model developers to learn more about their model errors through discovered subpopulations, validate the sources of errors through interactive analysis on the discovered subpopulations, and test hypotheses about model errors by defining custom subpopulations. The tool supports semantic descriptions of error-prone subpopulations at the token and concept level, as well as pre-defined higher-level features. Through use cases and expert interviews, we demonstrate how iSEA can assist error understanding and analysis.
