论文标题
Lyapunov引导的复发性神经网络性能的表示
Lyapunov-Guided Representation of Recurrent Neural Network Performance
论文作者
论文摘要
复发性神经网络（RNN）是用于序列和多元时间序列数据的普遍计算系统。尽管已知RNN的几种强大的体系结构，但尚不清楚如何将RNN初始化，体系结构和其他超参数与给定任务的准确性联系起来。在这项工作中，我们建议将RNN视为动力系统，并通过Lyapunov光谱分析将超参数与精度相关联，该方法专为非线性动力学系统设计。为了解决RNN功能超出现有的Lyapunov光谱分析的事实，我们建议通过自动编码器从Lyapunov Spectrum中推断出相关特征，并将其潜在表示（AELLE）嵌入。我们对各种RNN体系结构的研究表明，Aelle成功地将RNN Lyapunov Spectrum与准确性相关联。此外，Aelle学到的潜在代表性可以从同一任务中推广到新颖的投入，并在RNN培训过程的早期形成。后一种属性允许预测训练完成后RNN会收敛的准确性。我们得出的结论是，通过Lyapunov Spectrum以及Aelle的RNN表示，为组织和解释RNN体系结构的变体提供了一种新颖的方法。
Recurrent Neural Networks (RNN) are ubiquitous computing systems for sequences and multivariate time series data. While several robust architectures of RNN are known, it is unclear how to relate RNN initialization, architecture, and other hyperparameters with accuracy for a given task. In this work, we propose to treat RNN as dynamical systems and to correlate hyperparameters with accuracy through Lyapunov spectral analysis, a methodology specifically designed for nonlinear dynamical systems. To address the fact that RNN features go beyond the existing Lyapunov spectral analysis, we propose to infer relevant features from the Lyapunov spectrum with an Autoencoder and an embedding of its latent representation (AeLLE). Our studies of various RNN architectures show that AeLLE successfully correlates RNN Lyapunov spectrum with accuracy. Furthermore, the latent representation learned by AeLLE is generalizable to novel inputs from the same task and is formed early in the process of RNN training. The latter property allows for the prediction of the accuracy to which RNN would converge when training is complete. We conclude that representation of RNN through Lyapunov spectrum along with AeLLE provides a novel method for organization and interpretation of variants of RNN architectures.
