论文标题
深度神经网络及其最佳性的非凸稀疏正规化
Nonconvex sparse regularization for deep neural networks and its optimality
论文作者
论文摘要
最近的理论研究证明，通过通过一定的稀疏性约束来最大程度地降低经验风险来获得的深神经网络（DNN）估计量可以达到回归和分类问题的最佳收敛率。但是，稀疏性约束需要了解真实模型的某些属性，而这些模型在实践中不可用。此外，由于稀疏性约束的离散性质，很难计算。在本文中，我们提出了一种针对稀疏DNN的新型惩罚估计方法，该方法解决了稀疏性约束中存在的上述问题。我们为提出的稀疏占DNN估计量的过剩风险建立了甲骨文不平等，并为多个学习任务提供了收敛率。特别是，我们证明稀疏的估计量可以适应各种非参数回归问题的最小收敛速率。对于计算，我们开发了一种有效的基于梯度的优化算法，该算法可以保证目标函数的单调降低。
Recent theoretical studies proved that deep neural network (DNN) estimators obtained by minimizing empirical risk with a certain sparsity constraint can attain optimal convergence rates for regression and classification problems. However, the sparsity constraint requires to know certain properties of the true model, which are not available in practice. Moreover, computation is difficult due to the discrete nature of the sparsity constraint. In this paper, we propose a novel penalized estimation method for sparse DNNs, which resolves the aforementioned problems existing in the sparsity constraint. We establish an oracle inequality for the excess risk of the proposed sparse-penalized DNN estimator and derive convergence rates for several learning tasks. In particular, we prove that the sparse-penalized estimator can adaptively attain minimax convergence rates for various nonparametric regression problems. For computation, we develop an efficient gradient-based optimization algorithm that guarantees the monotonic reduction of the objective function.
