论文标题
适当的自适应学习率优化算法的学习率，用于培训深神经网络
Appropriate Learning Rates of Adaptive Learning Rate Optimization Algorithms for Training Deep Neural Networks
论文作者
论文摘要
本文涉及深度学习中的非convex随机优化问题，并提供适当的学习率，以这些学习率，适应性学习率优化算法（例如亚当和Amsgrad）可以近似问题的固定点。特别是，提供了恒定和降低的学习率，以近似问题的固定点。我们的结果还确保自适应学习率优化算法可以近似凸随机优化问题的全球最小化器。自适应学习率优化算法在文本和图像分类的数值实验中进行了检查。实验表明，具有恒定学习率的算法的性能优于学习率降低的算法。
This paper deals with nonconvex stochastic optimization problems in deep learning and provides appropriate learning rates with which adaptive learning rate optimization algorithms, such as Adam and AMSGrad, can approximate a stationary point of the problem. In particular, constant and diminishing learning rates are provided to approximate a stationary point of the problem. Our results also guarantee that the adaptive learning rate optimization algorithms can approximate global minimizers of convex stochastic optimization problems. The adaptive learning rate optimization algorithms are examined in numerical experiments on text and image classification. The experiments show that the algorithms with constant learning rates perform better than ones with diminishing learning rates.
