论文标题
木鱼：神经网络压缩的有效二阶近似
WoodFisher: Efficient Second-Order Approximation for Neural Network Compression
论文作者
论文摘要
二阶信息，以Hessian或Inverse-Hessian-Vector产品的形式形式，是解决优化问题的基本工具。最近，在深层神经网络的背景下利用这些信息引起了重大兴趣。但是，在这种情况下，对现有近似值的质量知之甚少。我们的工作研究了这个问题，通过现有方法确定了问题，并提出了一种称为Woodfisher的方法来计算对逆Hessian的忠实有效估计。 我们的主要应用是神经网络压缩，我们建立在经典的最佳脑损伤/外科医生框架上。我们证明，伍德菲尔（Woodfisher）的表现明显优于流行的一次性修剪方法。此外，即使考虑了迭代，逐步修剪，我们的方法也会导致对最先进方法的测试准确性提高，用于修剪流行的神经网络（例如Resnet-50，Mobilenetv1），接受了对标准图像分类数据集（例如ImageNet imemenet imenet ilsvrc）的培训。我们检查了如何扩展我们的方法以考虑一阶信息，并说明了其自动设置层的修剪阈值并在有限数据制度中执行压缩的能力。该代码可在以下链接，https://github.com/ist-daslab/woodfisher中找到。
Second-order information, in the form of Hessian- or Inverse-Hessian-vector products, is a fundamental tool for solving optimization problems. Recently, there has been significant interest in utilizing this information in the context of deep neural networks; however, relatively little is known about the quality of existing approximations in this context. Our work examines this question, identifies issues with existing approaches, and proposes a method called WoodFisher to compute a faithful and efficient estimate of the inverse Hessian. Our main application is to neural network compression, where we build on the classic Optimal Brain Damage/Surgeon framework. We demonstrate that WoodFisher significantly outperforms popular state-of-the-art methods for one-shot pruning. Further, even when iterative, gradual pruning is considered, our method results in a gain in test accuracy over the state-of-the-art approaches, for pruning popular neural networks (like ResNet-50, MobileNetV1) trained on standard image classification datasets such as ImageNet ILSVRC. We examine how our method can be extended to take into account first-order information, as well as illustrate its ability to automatically set layer-wise pruning thresholds and perform compression in the limited-data regime. The code is available at the following link, https://github.com/IST-DASLab/WoodFisher.
