论文标题
通过正交梯度下降来保证持续学习的保证
Generalisation Guarantees for Continual Learning with Orthogonal Gradient Descent
论文作者
论文摘要
在持续的学习环境中，深层神经网络容易遭受灾难性的遗忘。提出正交梯度下降以应对挑战。但是，尚未证明理论保证。我们提出了一个理论框架，用于研究神经切线核制度中的持续学习算法。该框架包括通过转移学习，概括和任务相似性的任务和代理对模型的封闭形式表达。在此框架中，我们证明了OGD对灾难性遗忘是可靠的，然后得出了SGD和OGD的第一个概括，以持续学习。最后，我们研究了OGD实践中此框架的局限性，并强调了神经切线内核变化对使用OGD的持续学习的重要性。
In Continual Learning settings, deep neural networks are prone to Catastrophic Forgetting. Orthogonal Gradient Descent was proposed to tackle the challenge. However, no theoretical guarantees have been proven yet. We present a theoretical framework to study Continual Learning algorithms in the Neural Tangent Kernel regime. This framework comprises closed form expression of the model through tasks and proxies for Transfer Learning, generalisation and tasks similarity. In this framework, we prove that OGD is robust to Catastrophic Forgetting then derive the first generalisation bound for SGD and OGD for Continual Learning. Finally, we study the limits of this framework in practice for OGD and highlight the importance of the Neural Tangent Kernel variation for Continual Learning with OGD.
