论文标题
通过传递性对准观察者协议
Aligning Intraobserver Agreement by Transitivity
论文作者
论文摘要
注释可重复性和准确性取决于注释者中的良好一致性。我们提出了一种新的方法，用于测量注释者一致性或观察者内部协议（IA）。提出的方法基于传播，这是在理性决策背景下对此进行了彻底研究的措施。与注释者IA的常用测试 - 重测策略相反，传递性度量对重新测试策略引入的几种偏见不太敏感。我们提出了一个代表定理，表明可以将符合透明度的相对判断数据映射到量表（根据测量理论）。我们还讨论了传递性的进一步应用，作为数据收集设计的一部分，以解决相对判断数据收集的二次复杂性问题。
Annotation reproducibility and accuracy rely on good consistency within annotators. We propose a novel method for measuring within annotator consistency or annotator Intraobserver Agreement (IA). The proposed approach is based on transitivity, a measure that has been thoroughly studied in the context of rational decision-making. The transitivity measure, in contrast with the commonly used test-retest strategy for annotator IA, is less sensitive to the several types of bias introduced by the test-retest strategy. We present a representation theorem to the effect that relative judgement data that meet transitivity can be mapped to a scale (in terms of measurement theory). We also discuss a further application of transitivity as part of data collection design for addressing the problem of the quadratic complexity of data collection of relative judgements.
