论文标题
通过有条件的互信息推理有关概括的推理
Reasoning About Generalization via Conditional Mutual Information
论文作者
论文摘要
我们提供了一个信息理论框架，用于研究机器学习算法的概括属性。我们的框架将现有方法联系在一起，包括统一的收敛范围和最新的自适应数据分析方法。具体而言，我们使用条件共同信息（CMI）来量化学习算法的输出（即训练有素的模型），可以识别输入（即训练数据）。我们表明，CMI的界限可以从VC维度，压缩方案，差异隐私和其他方法中获得。然后，我们证明有限的CMI意味着各种形式的概括。
We provide an information-theoretic framework for studying the generalization properties of machine learning algorithms. Our framework ties together existing approaches, including uniform convergence bounds and recent methods for adaptive data analysis. Specifically, we use Conditional Mutual Information (CMI) to quantify how well the input (i.e., the training data) can be recognized given the output (i.e., the trained model) of the learning algorithm. We show that bounds on CMI can be obtained from VC dimension, compression schemes, differential privacy, and other methods. We then show that bounded CMI implies various forms of generalization.
