论文标题
通过神经模型近似值的近似条件覆盖范围和校准
Approximate Conditional Coverage & Calibration via Neural Model Approximations
论文作者
论文摘要
用于量化分类模型的不确定性作为预测集的典型的逃亡者是类别的单例集校准。也就是说，此类集合应映射到精心校准的选择性分类器的输出，与观察到的相似实例的频率匹配。最近提出了对深网的适应性和局部保形的p值的作品并不能保证这种行为，也不能从经验上实现这种行为。取而代之的是，我们使用强信号来实现来自KNN的基于KNN的变压器网络的近似值的可靠性，以构建Mondrian共形预测指标的数据驱动分区，这些分区被视为弱选择性分类器，然后通过新的感应性Venn预测指标（Venn-Admit-Mit-Mit-Mit-Mit-Mit-Mit-Predictor）对其进行校准。最终的选择性分类器在保守的但实际上有用的意义上是对给定阈值进行了良好校准的。它们固有地对数据分区比例的变化进行了强大的稳健性，而直接的保守启发式方法为协变性转移提供了额外的鲁棒性。我们将最近的多个代表性和具有挑战性的自然语言处理分类任务（包括平衡和分配偏移的设置）的保形预测变量所产生的数量进行比较和对比。
A typical desideratum for quantifying the uncertainty from a classification model as a prediction set is class-conditional singleton set calibration. That is, such sets should map to the output of well-calibrated selective classifiers, matching the observed frequencies of similar instances. Recent works proposing adaptive and localized conformal p-values for deep networks do not guarantee this behavior, nor do they achieve it empirically. Instead, we use the strong signals for prediction reliability from KNN-based approximations of Transformer networks to construct data-driven partitions for Mondrian Conformal Predictors, which are treated as weak selective classifiers that are then calibrated via a new Inductive Venn Predictor, the Venn-ADMIT Predictor. The resulting selective classifiers are well-calibrated, in a conservative but practically useful sense for a given threshold. They are inherently robust to changes in the proportions of the data partitions, and straightforward conservative heuristics provide additional robustness to covariate shifts. We compare and contrast to the quantities produced by recent Conformal Predictors on several representative and challenging natural language processing classification tasks, including class-imbalanced and distribution-shifted settings.
