论文标题
对无监督适应过程中有选择地更新神经元的超级神经元修剪的调查和分析
Investigation and Analysis of Hyper and Hypo neuron pruning to selectively update neurons during Unsupervised Adaptation
论文作者
论文摘要
看不见的或不域的数据可以严重降低神经网络模型的性能，表明该模型未能概括看不见的数据。神经净修剪不仅可以帮助减少模型的大小，而且可以提高模型的概括能力。修剪方法寻找低升高神经元的贡献较小，因此可以从模型中删除模型的决策。这项工作调查了修剪方法是否成功地检测高疗法（主要是活跃或超级）或低降压（几乎没有活性或降低）的神经元，以及去除此类神经元是否可以帮助提高模型的概括能力。传统的盲目适应技术更新整体或一部分层，但从未探索过选择性地更新一个或多个层的单个神经元。该工作着重于卷积神经网络（CNN）的完全连接的层，首先可以选择性地适应某些神经元（由Hyper和Hypo神经元组成），然后进行全网络微调。使用自动语音识别的任务，这项工作表明了从模型中删除超级神经元和降压神经元如何可以改善模型对室外语音数据的性能以及与传统盲人模型适应相比，选择性神经元适应性如何确保改善性能。
Unseen or out-of-domain data can seriously degrade the performance of a neural network model, indicating the model's failure to generalize to unseen data. Neural net pruning can not only help to reduce a model's size but can improve the model's generalization capacity as well. Pruning approaches look for low-salient neurons that are less contributive to a model's decision and hence can be removed from the model. This work investigates if pruning approaches are successful in detecting neurons that are either high-salient (mostly active or hyper) or low-salient (barely active or hypo), and whether removal of such neurons can help to improve the model's generalization capacity. Traditional blind adaptation techniques update either the whole or a subset of layers, but have never explored selectively updating individual neurons across one or more layers. Focusing on the fully connected layers of a convolutional neural network (CNN), this work shows that it may be possible to selectively adapt certain neurons (consisting of the hyper and the hypo neurons) first, followed by a full-network fine tuning. Using the task of automatic speech recognition, this work demonstrates how the removal of hyper and hypo neurons from a model can improve the model's performance on out-of-domain speech data and how selective neuron adaptation can ensure improved performance when compared to traditional blind model adaptation.
