论文标题
ClassAction Prediction：在美国进行集体诉讼案件的法律判决预测的具有挑战性的基准
ClassActionPrediction: A Challenging Benchmark for Legal Judgment Prediction of Class Action Cases in the US
论文作者
论文摘要
法律自然语言处理（NLP）的研究领域最近非常活跃，法律判断预测（LJP）成为最广泛研究的任务之一。迄今为止，大多数公开发布的LJP数据集源自民法国家。在这项工作中，我们首次发布了一个具有挑战性的LJP数据集，该数据集专注于美国的集体诉讼案例。它是普通法系统中的第一个数据集，它专注于涉及投诉的更艰巨，更现实的任务，而不是法院撰写的经常使用的事实。此外，我们通过收集专家人类预测来研究任务的难度，这表明即使人类专家也只能达到该数据集的53％的精度。尽管仅考虑了前2,048个令牌，但我们的longformer模型显然优于人类基线（63％）。此外，我们执行了详细的误差分析，发现长形模型的校准明显好于人类专家。最后，我们公开发布数据集和用于实验的代码。
The research field of Legal Natural Language Processing (NLP) has been very active recently, with Legal Judgment Prediction (LJP) becoming one of the most extensively studied tasks. To date, most publicly released LJP datasets originate from countries with civil law. In this work, we release, for the first time, a challenging LJP dataset focused on class action cases in the US. It is the first dataset in the common law system that focuses on the harder and more realistic task involving the complaints as input instead of the often used facts summary written by the court. Additionally, we study the difficulty of the task by collecting expert human predictions, showing that even human experts can only reach 53% accuracy on this dataset. Our Longformer model clearly outperforms the human baseline (63%), despite only considering the first 2,048 tokens. Furthermore, we perform a detailed error analysis and find that the Longformer model is significantly better calibrated than the human experts. Finally, we publicly release the dataset and the code used for the experiments.
