论文标题
监督超参数配置的多保真竞赛
Supervising the Multi-Fidelity Race of Hyperparameter Configurations
论文作者
论文摘要
多保真（灰色盒）超参数优化技术（HPO）最近出现是调整深度学习方法的有希望的方向。但是，现有的方法遭受了HPO预算对超参数配置的优化分配。在这项工作中，我们介绍了Dyhpo，这是一种贝叶斯优化方法，该方法学会了决定在所有可行配置中的动态竞赛中进一步训练哪种超参数配置。我们为高斯过程提出了一个新的深内核，该内核嵌入了学习曲线动力学，以及包含多计划信息的采集函数。我们通过包括50个数据集（表格，图像，NLP）和不同体系结构（MLP，CNN/NAS，RNN）的大规模实验来证明DYHPO与最先进的超参数优化方法的显着优势。
Multi-fidelity (gray-box) hyperparameter optimization techniques (HPO) have recently emerged as a promising direction for tuning Deep Learning methods. However, existing methods suffer from a sub-optimal allocation of the HPO budget to the hyperparameter configurations. In this work, we introduce DyHPO, a Bayesian Optimization method that learns to decide which hyperparameter configuration to train further in a dynamic race among all feasible configurations. We propose a new deep kernel for Gaussian Processes that embeds the learning curve dynamics, and an acquisition function that incorporates multi-budget information. We demonstrate the significant superiority of DyHPO against state-of-the-art hyperparameter optimization methods through large-scale experiments comprising 50 datasets (Tabular, Image, NLP) and diverse architectures (MLP, CNN/NAS, RNN).
