论文标题
通过渠道知识蒸馏进行稠密预测
Channel-wise Knowledge Distillation for Dense Prediction
论文作者
论文摘要
知识蒸馏（KD）已被证明是训练紧凑型模型的简单有效工具。几乎所有用于密集预测任务的KD变体都使学生和教师网络在空间域中的特征地图保持一致，通常是通过最大程度地减少点和/或成对的差异来最大程度地减少点。观察到，在语义细分中，每个通道的某些层特征激活倾向于编码场景类别的显着性（类似于类激活映射），我们建议在学生和教师网络之间的频道方面保持特征。为此，我们首先使用SoftMax归一化将每个通道的特征映射转换为一个概率图，然后最小化两个网络的相应通道的Kullback-Leibler（KL）差异。通过这样做，我们的方法着重于模仿网络之间的频道的软分布。特别是，KL Divergence使学习能够更加关注频道图的最显着区域，这大概是对应于语义分割的最有用的信号。实验表明，我们的渠道蒸馏的表现要优于几乎所有现有的空间蒸馏方法来大大进行语义分割，并且在训练过程中所需的计算成本较小。我们始终在具有各种网络结构的三个基准上实现卓越的性能。代码可用：https：//git.io/distiller
Knowledge distillation (KD) has been proven to be a simple and effective tool for training compact models. Almost all KD variants for dense prediction tasks align the student and teacher networks' feature maps in the spatial domain, typically by minimizing point-wise and/or pair-wise discrepancy. Observing that in semantic segmentation, some layers' feature activations of each channel tend to encode saliency of scene categories (analogue to class activation mapping), we propose to align features channel-wise between the student and teacher networks. To this end, we first transform the feature map of each channel into a probabilty map using softmax normalization, and then minimize the Kullback-Leibler (KL) divergence of the corresponding channels of the two networks. By doing so, our method focuses on mimicking the soft distributions of channels between networks. In particular, the KL divergence enables learning to pay more attention to the most salient regions of the channel-wise maps, presumably corresponding to the most useful signals for semantic segmentation. Experiments demonstrate that our channel-wise distillation outperforms almost all existing spatial distillation methods for semantic segmentation considerably, and requires less computational cost during training. We consistently achieve superior performance on three benchmarks with various network structures. Code is available at: https://git.io/Distiller
