论文标题
研究幅度修剪对对比度学习方法的影响
Studying the impact of magnitude pruning on contrastive learning methods
论文作者
论文摘要
我们研究了不同修剪技术对具有对比损失功能的深层神经网络所学的表示的影响。我们的工作发现，相对于经过传统的跨透明术损失训练的模型，在高稀疏度水平下，对比度学习的示例数量更高。为了理解这种明显的差异，我们使用诸如派（Hooker等，2019），Q-Score（Kalibhat等，2022）和PD-Score（Baldock等人，2021年）等指标来衡量前介学对学习表示表示的影响。我们的分析表明，修剪方法实施的时间表很重要。我们发现，当在训练阶段早期引入修剪时，稀疏性对学习表示的质量的负面影响最高。
We study the impact of different pruning techniques on the representation learned by deep neural networks trained with contrastive loss functions. Our work finds that at high sparsity levels, contrastive learning results in a higher number of misclassified examples relative to models trained with traditional cross-entropy loss. To understand this pronounced difference, we use metrics such as the number of PIEs (Hooker et al., 2019), Q-Score (Kalibhat et al., 2022), and PD-Score (Baldock et al., 2021) to measure the impact of pruning on the learned representation quality. Our analysis suggests the schedule of the pruning method implementation matters. We find that the negative impact of sparsity on the quality of the learned representation is the highest when pruning is introduced early on in the training phase.
