论文标题
通过元学习学习软标签
Learning Soft Labels via Meta Learning
论文作者
论文摘要
一壁标签并不代表概念之间的软决策边界，因此，对其进行训练的模型很容易过度拟合。使用软标签作为目标提供正则化，但是在优化的不同阶段，不同的软标签可能是最佳的。同样，在嘈杂注释存在下使用固定标签的培训会导致泛化。为了解决这些限制，我们提出了一个框架，在该框架中，我们将标签视为可学习的参数，并将其与模型参数一起优化。学识渊博的标签不断适应模型的状态，从而提供动态正则化。当应用于监督图像分类的任务时，我们的方法会导致不同数据集和体系结构的一致收益。例如，在CIFAR100上，动态学习的标签将RESNET18提高了2.1％。当应用于包含嘈杂标签的数据集时，学识渊博的标签纠正了注释错误，并通过显着的边距改进了最新的标签。最后，我们表明，学到的标签捕获了班级之间的语义关系，从而改善了蒸馏的下游任务的教师模型。
One-hot labels do not represent soft decision boundaries among concepts, and hence, models trained on them are prone to overfitting. Using soft labels as targets provide regularization, but different soft labels might be optimal at different stages of optimization. Also, training with fixed labels in the presence of noisy annotations leads to worse generalization. To address these limitations, we propose a framework, where we treat the labels as learnable parameters, and optimize them along with model parameters. The learned labels continuously adapt themselves to the model's state, thereby providing dynamic regularization. When applied to the task of supervised image-classification, our method leads to consistent gains across different datasets and architectures. For instance, dynamically learned labels improve ResNet18 by 2.1% on CIFAR100. When applied to dataset containing noisy labels, the learned labels correct the annotation mistakes, and improves over state-of-the-art by a significant margin. Finally, we show that learned labels capture semantic relationship between classes, and thereby improve teacher models for the downstream task of distillation.
