论文标题
设计封闭的人类延期延期管道
Designing Closed Human-in-the-loop Deferral Pipelines
论文作者
论文摘要
在混合人机递延框架中，分类器可能会将不确定的案例推迟给人类决策者（他们本身是容易犯错的）。先前在同时培训此类分类器和延期模型的工作通常已经假设在培训期间可以访问甲骨文，以获取用于培训样本的真正课堂标签，但实际上，通常没有这样的甲骨文。相比之下，我们考虑了一条“封闭的”决策管道，其中在延期中使用的相同犯错的人类决策者还提供培训标签。如何使用不完美和有偏见的人类专家标签来训练公平，准确的延期框架？我们的主要见解是，通过利用弱的先前信息，我们可以将专家与输入示例相匹配，以确保所得延期框架的公平性和准确性，即使使用不完美和有偏见的专家代替地面真相标签。通过理论分析和对两个任务的评估，我们的方法的功效既可以显示出来。
In hybrid human-machine deferral frameworks, a classifier can defer uncertain cases to human decision-makers (who are often themselves fallible). Prior work on simultaneous training of such classifier and deferral models has typically assumed access to an oracle during training to obtain true class labels for training samples, but in practice there often is no such oracle. In contrast, we consider a "closed" decision-making pipeline in which the same fallible human decision-makers used in deferral also provide training labels. How can imperfect and biased human expert labels be used to train a fair and accurate deferral framework? Our key insight is that by exploiting weak prior information, we can match experts to input examples to ensure fairness and accuracy of the resulting deferral framework, even when imperfect and biased experts are used in place of ground truth labels. The efficacy of our approach is shown both by theoretical analysis and by evaluation on two tasks.
