论文标题
使深度神经网络紧凑的后果
The Ramifications of Making Deep Neural Networks Compact
论文作者
论文摘要
深度神经网络（DNNS）研究的最新趋势是使网络更加紧凑。设计紧凑的DNN背后的动机是提高能源效率，因为由于具有较低的内存足迹，紧凑的DNN具有较低的芯片访问量，从而提高了能源效率。但是，我们表明，使DNNS紧凑具有间接和微妙的含义，而这些含义却没有得到很好的理解。减少DNN中参数的数量增加了激活的数量，而激活的数量又增加了内存足迹。我们在Tesla P100 GPU上评估了几个最近提供的紧凑型DNN，并表明它们的“激活与参数比”在1.4至32.8之间。此外，“模型大小比率”的“内存脚印”范围在15至443之间。这表明较高的激活会导致大型内存足迹，从而增加了片上/片上的数据移动。此外，这些减少参数的技术降低了算术强度，从而增加了片上/离片内存储器带宽的要求。由于这些因素，紧凑型DNN的能源效率可能会大大降低，这是针对设计紧凑型DNN的原始动机。
The recent trend in deep neural networks (DNNs) research is to make the networks more compact. The motivation behind designing compact DNNs is to improve energy efficiency since by virtue of having lower memory footprint, compact DNNs have lower number of off-chip accesses which improves energy efficiency. However, we show that making DNNs compact has indirect and subtle implications which are not well-understood. Reducing the number of parameters in DNNs increases the number of activations which, in turn, increases the memory footprint. We evaluate several recently-proposed compact DNNs on Tesla P100 GPU and show that their "activations to parameters ratio" ranges between 1.4 to 32.8. Further, the "memory-footprint to model size ratio" ranges between 15 to 443. This shows that a higher number of activations causes large memory footprint which increases on-chip/off-chip data movements. Furthermore, these parameter-reducing techniques reduce the arithmetic intensity which increases on-chip/off-chip memory bandwidth requirement. Due to these factors, the energy efficiency of compact DNNs may be significantly reduced which is against the original motivation for designing compact DNNs.
