论文标题
Lipschitz连续损失功能和加权组L0-符号约束的稀疏训练
Sparse Training with Lipschitz Continuous Loss Functions and a Weighted Group L0-norm Constraint
论文作者
论文摘要
本文是由深度神经网络训练的结构性稀疏性激励的。我们研究了加权组L0-Norm约束，并介绍该集合的投影和正常锥体。使用随机平滑，我们开发了零和一阶算法，以最大程度地减少Lipschitz的连续函数，该函数受任何可以投射到的闭合集合的约束。对于两个相关收敛标准的拟议算法证明了非反应收敛的保证，可以被视为近似固定点。使用所提出的算法给出了两种进一步的方法：一种具有高概率的非质子收敛保证，另一种具有渐近性保证，几乎可以肯定地确定固定点。我们特别相信，这些是第一个对于约束Lipschitz连续损失函数的第一个非质子收敛结果。
This paper is motivated by structured sparsity for deep neural network training. We study a weighted group L0-norm constraint, and present the projection and normal cone of this set. Using randomized smoothing, we develop zeroth and first-order algorithms for minimizing a Lipschitz continuous function constrained by any closed set which can be projected onto. Non-asymptotic convergence guarantees are proven in expectation for the proposed algorithms for two related convergence criteria which can be considered as approximate stationary points. Two further methods are given using the proposed algorithms: one with non-asymptotic convergence guarantees in high probability, and the other with asymptotic guarantees to a stationary point almost surely. We believe in particular that these are the first such non-asymptotic convergence results for constrained Lipschitz continuous loss functions.
