论文标题
访问辅助信息的优化
Optimization with Access to Auxiliary Information
论文作者
论文摘要
我们调查了最小化目标功能$ f $的基本优化问题，该问题的计算梯度昂贵或有限的可用性，允许访问某些辅助侧功能$ H $，其梯度便宜或更可用。这种表述捕获了许多实际相关性的设置，例如i）在SGD中重新使用批次，ii）转移学习，iii）联合学习，iv）iv）训练压缩模型/辍学等。我们建议在所有这些设置中适用的两种通用新算法。我们还证明，在目标和侧面信息之间的Hessian相似性假设下，我们可以从该框架中受益。当这种相似性度量很小时，将获得益处。当辅助噪声与目标函数相关时，我们还显示了随机性的潜在益处。
We investigate the fundamental optimization question of minimizing a target function $f$, whose gradients are expensive to compute or have limited availability, given access to some auxiliary side function $h$ whose gradients are cheap or more available. This formulation captures many settings of practical relevance, such as i) re-using batches in SGD, ii) transfer learning, iii) federated learning, iv) training with compressed models/dropout, Et cetera. We propose two generic new algorithms that apply in all these settings; we also prove that we can benefit from this framework under the Hessian similarity assumption between the target and side information. A benefit is obtained when this similarity measure is small; we also show a potential benefit from stochasticity when the auxiliary noise is correlated with that of the target function.
