论文标题
关于上下文化语言表示的内在和外在公平评估指标
On the Intrinsic and Extrinsic Fairness Evaluation Metrics for Contextualized Language Representations
论文作者
论文摘要
已经引入了多个指标来衡量各种自然语言处理任务中的公平性。这些指标可以大致分为两个类别：1）\ emph {外部指标}，用于评估下游应用程序中的公平性和2）\ emph {intinsic cormitrics}，以估算上下文化语言表示模型的公平性。在本文中，我们使用19个情境化的语言模型进行了偏见概念之间的内在和外在指标之间的广泛相关研究。我们发现，即使在校正度量未对准，评估数据集中的噪声以及混杂因素（例如外部指标的实验配置）时，内在和外在指标不一定在其原始设置中相关。 ％al
Multiple metrics have been introduced to measure fairness in various natural language processing tasks. These metrics can be roughly categorized into two categories: 1) \emph{extrinsic metrics} for evaluating fairness in downstream applications and 2) \emph{intrinsic metrics} for estimating fairness in upstream contextualized language representation models. In this paper, we conduct an extensive correlation study between intrinsic and extrinsic metrics across bias notions using 19 contextualized language models. We find that intrinsic and extrinsic metrics do not necessarily correlate in their original setting, even when correcting for metric misalignments, noise in evaluation datasets, and confounding factors such as experiment configuration for extrinsic metrics. %al
