论文标题
我们在自我监督的学习中最大化什么？
What Do We Maximize in Self-Supervised Learning?
论文作者
论文摘要
在本文中，我们研究了自我监督的学习方法，尤其是VICREG，以提供对其构建的信息理论理解。作为第一步，我们演示了如何获得确定性网络的信息理论数量，为依赖随机模型的先前工作提供了一种可能的替代方法。这使我们能够证明如何（重新）从第一原则及其对数据分布的假设发现。此外，我们从经验上证明了我们的假设的有效性，证实了我们对Vicreg的新理解。最后，我们认为，我们获得的派生和见解可以推广到许多其他SSL方法，这为对SSL和转移学习的理论和实际理解开辟了新的途径。
In this paper, we examine self-supervised learning methods, particularly VICReg, to provide an information-theoretical understanding of their construction. As a first step, we demonstrate how information-theoretic quantities can be obtained for a deterministic network, offering a possible alternative to prior work that relies on stochastic models. This enables us to demonstrate how VICReg can be (re)discovered from first principles and its assumptions about data distribution. Furthermore, we empirically demonstrate the validity of our assumptions, confirming our novel understanding of VICReg. Finally, we believe that the derivation and insights we obtain can be generalized to many other SSL methods, opening new avenues for theoretical and practical understanding of SSL and transfer learning.
