论文标题
深度神经网络中的基于保证金的正则化和选择性抽样
Margin-Based Regularization and Selective Sampling in Deep Neural Networks
论文作者
论文摘要
我们为深神经网络（DNNS）得出了一种新的基于保证金的正则化公式，称为多修细胞正则化（MMR）。 MMR的灵感来自于用于浅线性分类器边缘分析的原理，例如支持向量机（SVM）。与SVM不同，MMR通过边界球的半径（即数据中特征向量的最大规范）持续缩放，这在训练过程中正在不断变化。我们从经验上证明，通过对损失函数的简单补充，我们的方法可以在跨域的各种分类任务上获得更好的结果。使用相同的概念，我们还得出了选择性抽样方案，并通过根据最小的边距评分（MMS）选择样品来证明DNN的加速训练。该分数衡量输入应进行的最小位移量，直到其预测分类切换为止。我们在三个图像分类任务和六个语言文本分类任务上评估了建议的方法。具体而言，我们使用最先进的卷积神经网络（CNNS）和BERT-base架构对CIFAR10，CIFAR100和ImaTeNet的经验结果提高了，用于MNLI，QQP，QNLI，MRPC，MRPC，SST-2和RTE基准。
We derive a new margin-based regularization formulation, termed multi-margin regularization (MMR), for deep neural networks (DNNs). The MMR is inspired by principles that were applied in margin analysis of shallow linear classifiers, e.g., support vector machine (SVM). Unlike SVM, MMR is continuously scaled by the radius of the bounding sphere (i.e., the maximal norm of the feature vector in the data), which is constantly changing during training. We empirically demonstrate that by a simple supplement to the loss function, our method achieves better results on various classification tasks across domains. Using the same concept, we also derive a selective sampling scheme and demonstrate accelerated training of DNNs by selecting samples according to a minimal margin score (MMS). This score measures the minimal amount of displacement an input should undergo until its predicted classification is switched. We evaluate our proposed methods on three image classification tasks and six language text classification tasks. Specifically, we show improved empirical results on CIFAR10, CIFAR100 and ImageNet using state-of-the-art convolutional neural networks (CNNs) and BERT-BASE architecture for the MNLI, QQP, QNLI, MRPC, SST-2 and RTE benchmarks.
