论文标题
具有较小重量和深度分离障碍的神经网络
Neural Networks with Small Weights and Depth-Separation Barriers
论文作者
论文摘要
在研究神经网络的表现力时，一个重要的问题是，假设它们的大小是有限的，是否有一些功能只能通过足够深的网络近似。但是，对于持续的深度，现有的结果仅限于$ 2 $和3美元的深度，而更高深度的结果是一个重要的开放问题。在本文中，我们专注于馈电relu网络，并证明了将这种结果超过深度$ 4 $的基本障碍，这是通过减少开放问题和电路复杂性的自然隔离障碍的。为了证明这一点，我们研究了一个看似无关的独立兴趣问题：即，是否有多项式结合的函数需要超级多项式权重才能与恒定深度的神经网络近似。我们通过证明一个函数可以通过多项式大小的恒定深度$ k $网络近似具有较大权重的函数来为该问题提供负面和建设性的答案，它也可以通过多项式尺寸的深度$ 3K+3 $网络近似，其权重是多种限制的。
In studying the expressiveness of neural networks, an important question is whether there are functions which can only be approximated by sufficiently deep networks, assuming their size is bounded. However, for constant depths, existing results are limited to depths $2$ and $3$, and achieving results for higher depths has been an important open question. In this paper, we focus on feedforward ReLU networks, and prove fundamental barriers to proving such results beyond depth $4$, by reduction to open problems and natural-proof barriers in circuit complexity. To show this, we study a seemingly unrelated problem of independent interest: Namely, whether there are polynomially-bounded functions which require super-polynomial weights in order to approximate with constant-depth neural networks. We provide a negative and constructive answer to that question, by showing that if a function can be approximated by a polynomially-sized, constant depth $k$ network with arbitrarily large weights, it can also be approximated by a polynomially-sized, depth $3k+3$ network, whose weights are polynomially bounded.
