论文标题
部分可观测时空混沌系统的无模型预测
Of Human Criteria and Automatic Metrics: A Benchmark of the Evaluation of Story Generation
论文作者
论文摘要
自动故事产生（ASG）的研究在很大程度上依赖于人类和自动评估。但是，尚无共识，即使用哪些人类评估标准，也没有分析自动标准与它们的相关性如何。在本文中，我们建议重新评估ASG评估。我们介绍了由社会科学文学精心促进的6种正交和全面的人类标准。我们还提出了Hanna，这是一个由10种不同ASG系统制作的1,056个故事的注释数据集。汉娜（Hanna）允许我们定量评估72个自动指标与人类标准的相关性。我们的分析强调了ASG当前指标的弱点，并允许我们为ASG评估提出实用建议。
Research on Automatic Story Generation (ASG) relies heavily on human and automatic evaluation. However, there is no consensus on which human evaluation criteria to use, and no analysis of how well automatic criteria correlate with them. In this paper, we propose to re-evaluate ASG evaluation. We introduce a set of 6 orthogonal and comprehensive human criteria, carefully motivated by the social sciences literature. We also present HANNA, an annotated dataset of 1,056 stories produced by 10 different ASG systems. HANNA allows us to quantitatively evaluate the correlations of 72 automatic metrics with human criteria. Our analysis highlights the weaknesses of current metrics for ASG and allows us to formulate practical recommendations for ASG evaluation.
