论文标题
使用Venn的自然语言理解模型的校准 - 供应指标
Calibration of Natural Language Understanding Models with Venn--ABERS Predictors
论文作者
论文摘要
变形金刚目前是自然语言理解（NLU）任务的最先进的变压器，容易产生未校准的预测或极端概率，从而根据其输出相对困难而做出不同的决策过程。在本文中，我们建议建立几个电感维纳（IVAP），这些预测因子（IVAP）可以根据预先训练的变压器的选择在最小的假设下可以很好地校准。我们在一组不同的NLU任务上测试了它们的性能，并表明它们能够产生均匀分布在[0,1]间隔上的精心校准的概率预测，同时均保留了原始模型的预测准确性。
Transformers, currently the state-of-the-art in natural language understanding (NLU) tasks, are prone to generate uncalibrated predictions or extreme probabilities, making the process of taking different decisions based on their output relatively difficult. In this paper we propose to build several inductive Venn--ABERS predictors (IVAP), which are guaranteed to be well calibrated under minimal assumptions, based on a selection of pre-trained transformers. We test their performance over a set of diverse NLU tasks and show that they are capable of producing well-calibrated probabilistic predictions that are uniformly spread over the [0,1] interval -- all while retaining the original model's predictive accuracy.
