论文标题
从量化的DNN到可量化的DNN
From Quantized DNNs to Quantizable DNNs
论文作者
论文摘要
本文提出了可量化的DNN，这是一种特殊类型的DNN，可以在执行过程中灵活地量化其位宽度（此后称为“位模式”），而无需进行进一步的重新训练。为了同时优化所有位模式，提出了所有位模式的组合损失，该组合损失范围从低位模式到32位模式，可以实施一致的预测。基于一致性的损失也可以视为训练期间的某些正规化形式。由于不同位模式中矩阵乘法的输出具有不同的分布，因此我们引入了特定于位的批准归一化，以减少不同位模式之间的冲突。在CIFAR100和Imagenet上进行的实验表明，与量化的DNN相比，可量化的DNN不仅具有更好的灵活性，而且还具有更高的分类精度。消融研究进一步验证，通过基于一致性损失的正则化确实可以改善模型的泛化性能。
This paper proposes Quantizable DNNs, a special type of DNNs that can flexibly quantize its bit-width (denoted as `bit modes' thereafter) during execution without further re-training. To simultaneously optimize for all bit modes, a combinational loss of all bit modes is proposed, which enforces consistent predictions ranging from low-bit mode to 32-bit mode. This Consistency-based Loss may also be viewed as certain form of regularization during training. Because outputs of matrix multiplication in different bit modes have different distributions, we introduce Bit-Specific Batch Normalization so as to reduce conflicts among different bit modes. Experiments on CIFAR100 and ImageNet have shown that compared to quantized DNNs, Quantizable DNNs not only have much better flexibility, but also achieve even higher classification accuracy. Ablation studies further verify that the regularization through the consistency-based loss indeed improves the model's generalization performance.
