论文标题
关于对抗训练的损失格局：确定挑战以及如何克服挑战
On the Loss Landscape of Adversarial Training: Identifying Challenges and How to Overcome Them
论文作者
论文摘要
我们分析了对抗训练对机器学习模型损失格局的影响。为此，我们首先提供对不同对抗预算下对抗性损失功能的性质的分析研究。然后，我们证明，由于曲率增加和散落的梯度，对抗性损失景观对优化不利。我们的结论是通过数值分析来验证的，这些分析表明，大型对抗预算下的训练阻碍了次优的随机初始化，导致非呈现梯度，并使模型找到更尖的最小值。基于这些观察结果，我们表明，定期的对抗调度（PAS）策略可以有效克服这些挑战，从而产生比香草对抗性训练更好的结果，同时对学习率的选择敏感得多。
We analyze the influence of adversarial training on the loss landscape of machine learning models. To this end, we first provide analytical studies of the properties of adversarial loss functions under different adversarial budgets. We then demonstrate that the adversarial loss landscape is less favorable to optimization, due to increased curvature and more scattered gradients. Our conclusions are validated by numerical analyses, which show that training under large adversarial budgets impede the escape from suboptimal random initialization, cause non-vanishing gradients and make the model find sharper minima. Based on these observations, we show that a periodic adversarial scheduling (PAS) strategy can effectively overcome these challenges, yielding better results than vanilla adversarial training while being much less sensitive to the choice of learning rate.
