论文标题
人文：校准人类匹配超出一项任务
HumanAL: Calibrating Human Matching Beyond a Single Task
论文作者
论文摘要
这项工作提供了一种关于将人类输入用作标签的新颖观点，承认人类可能会犯错。我们为人类注释者构建一个行为概况，用作提供的输入的特征表示。我们表明，通过利用黑盒机器学习，我们可以考虑人类的行为并校准其输入以提高标签质量。为了支持我们的索赔并提供概念验证，我们尝试了三个不同的匹配任务，即架构匹配，实体匹配和文本匹配。我们的经验评估表明，该方法可以在包括跨域（跨不同匹配任务）在内的多种设置中提高收集标签的质量。
This work offers a novel view on the use of human input as labels, acknowledging that humans may err. We build a behavioral profile for human annotators which is used as a feature representation of the provided input. We show that by utilizing black-box machine learning, we can take into account human behavior and calibrate their input to improve the labeling quality. To support our claims and provide a proof-of-concept, we experiment with three different matching tasks, namely, schema matching, entity matching and text matching. Our empirical evaluation suggests that the method can improve the quality of gathered labels in multiple settings including cross-domain (across different matching tasks).
