论文标题
方差协方差正规化在自我监督表示中实施成对独立性
Variance Covariance Regularization Enforces Pairwise Independence in Self-Supervised Representations
论文作者
论文摘要
通过限制或正规化投影仪输出的协方差矩阵，避免了诸如VICREG，BARLOW双胞胎或W-MSE之类的自我监督学习（SSL）方法。这项研究强调了这种策略的重要特性，我们将其创造了差异差异正则化（VCREG）。更准确地说，我们表明{\ em VCREG合并到MLP投影仪在学习表示的特征之间实现成对独立性}。通过桥接VCREG将投影仪的输出应用于投影仪的输入应用的内核独立标准来得出。我们从经验上验证了（i）我们提供的发现，我们提供了投影仪的特征有利于成对独立性的证据，（ii）我们证明成对独立性对跨域的泛化有益，（iii）我们证明了VCREG的范围通过使用它来解决独立组件分析而超越了SSL。这提供了SSL中MLP投影仪的第一个理论动机和解释。
Self-Supervised Learning (SSL) methods such as VICReg, Barlow Twins or W-MSE avoid collapse of their joint embedding architectures by constraining or regularizing the covariance matrix of their projector's output. This study highlights important properties of such strategy, which we coin Variance-Covariance regularization (VCReg). More precisely, we show that {\em VCReg combined to a MLP projector enforces pairwise independence between the features of the learned representation}. This result emerges by bridging VCReg applied on the projector's output to kernel independence criteria applied on the projector's input. We empirically validate our findings where (i) we put in evidence which projector's characteristics favor pairwise independence, (ii) we demonstrate pairwise independence to be beneficial for out-of-domain generalization, (iii) we demonstrate that the scope of VCReg goes beyond SSL by using it to solve Independent Component Analysis. This provides the first theoretical motivation and explanation of MLP projectors in SSL.
