论文标题
通过拓扑熵的神经网络的深度宽度权衡
Depth-Width Trade-offs for Neural Networks via Topological Entropy
论文作者
论文摘要
深度学习理论研究的核心问题之一是了解结构特性（例如深度，宽度和节点数量）如何影响深神经网络的表达。在这项工作中，我们展示了深层神经网络的表达性与动力学系统的拓扑熵之间的新联系，该连接可用于表征神经网络的深度宽度权衡。我们通过结构参数在神经网络的拓扑熵上提供上限。具体而言，具有$ L $层的Relu Network的拓扑熵和每层$ M $节点的上限由$ O（L \ log M）$限制。此外，如果神经网络是某些功能$ f $的良好近似值，则神经网络的大小相对于$ f $的拓扑熵具有指数下限。此外，我们讨论了拓扑熵，振荡，周期和Lipschitz常数之间的关系。
One of the central problems in the study of deep learning theory is to understand how the structure properties, such as depth, width and the number of nodes, affect the expressivity of deep neural networks. In this work, we show a new connection between the expressivity of deep neural networks and topological entropy from dynamical system, which can be used to characterize depth-width trade-offs of neural networks. We provide an upper bound on the topological entropy of neural networks with continuous semi-algebraic units by the structure parameters. Specifically, the topological entropy of ReLU network with $l$ layers and $m$ nodes per layer is upper bounded by $O(l\log m)$. Besides, if the neural network is a good approximation of some function $f$, then the size of the neural network has an exponential lower bound with respect to the topological entropy of $f$. Moreover, we discuss the relationship between topological entropy, the number of oscillations, periods and Lipschitz constant.
