论文标题
通过从在线自我解释中提取知识来进行内省学习
Introspective Learning by Distilling Knowledge from Online Self-explanation
论文作者
论文摘要
近年来，已经提出了许多解释方法来解释深层神经网络的个体分类。但是，如何利用创建的解释来改善学习过程的探索较少。作为特权信息，模型的解释可用于指导模型本身的学习过程。在社区中，另一项经过深入研究的特权信息用于指导模型的培训是来自强大的教师模型的知识。这项工作的目的是利用自我解释来通过从知识蒸馏中借用想法来改善学习过程。首先，我们研究了从教师网络转移到学生网络的知识的有效组成部分。我们的调查表明，非地面真相类中的响应和教师成果中的类似信息的响应都有助于知识蒸馏的成功。在结论中，我们提出了通过从在线自我解释中提取知识来实施内省学习的实施。接受内省学习程序训练的模型优于接受标准学习程序的训练的模型，以及接受不同正则化方法训练的模型。与从同行网络或教师网络中学到的模型相比，我们的模型还表现出竞争性的表现，并且不需要同行和教师。
In recent years, many explanation methods have been proposed to explain individual classifications of deep neural networks. However, how to leverage the created explanations to improve the learning process has been less explored. As the privileged information, the explanations of a model can be used to guide the learning process of the model itself. In the community, another intensively investigated privileged information used to guide the training of a model is the knowledge from a powerful teacher model. The goal of this work is to leverage the self-explanation to improve the learning process by borrowing ideas from knowledge distillation. We start by investigating the effective components of the knowledge transferred from the teacher network to the student network. Our investigation reveals that both the responses in non-ground-truth classes and class-similarity information in teacher's outputs contribute to the success of the knowledge distillation. Motivated by the conclusion, we propose an implementation of introspective learning by distilling knowledge from online self-explanations. The models trained with the introspective learning procedure outperform the ones trained with the standard learning procedure, as well as the ones trained with different regularization methods. When compared to the models learned from peer networks or teacher networks, our models also show competitive performance and requires neither peers nor teachers.
