论文标题
研究神经元在融合异质神经网络中的干扰
Investigating Neuron Disturbing in Fusing Heterogeneous Neural Networks
论文作者
论文摘要
将在单独的沟通回合中融合的深度学习模型是联合学习的直接实现。尽管当前的模型融合方法在将神经网络与几乎相同的体系结构的融合中显示出实验有效，但理论上很少分析它们。在本文中，我们揭示了神经元令人不安的现象，其中来自异质局部模型的神经元相互影响。我们从贝叶斯的观点中提供了详细的解释，该解释结合了客户之间的数据异质性和神经网络的属性。此外，为了验证我们的发现，我们提出了一种实验方法，该方法通过自适应选择一个称为AMS的局部模型来排除神经元的干扰和融合神经网络，以根据输入执行预测。该实验表明，与一般模型融合和集合方法相比，AMS在数据异质性上更强大。这意味着需要考虑模型融合中神经干扰的必要性。此外，AMS可用于将架构变化的模型作为实验算法进行融合，我们还列出了几种可能的AMS扩展，以供将来的工作。
Fusing deep learning models trained on separately located clients into a global model in a one-shot communication round is a straightforward implementation of Federated Learning. Although current model fusion methods are shown experimentally valid in fusing neural networks with almost identical architectures, they are rarely theoretically analyzed. In this paper, we reveal the phenomenon of neuron disturbing, where neurons from heterogeneous local models interfere with each other mutually. We give detailed explanations from a Bayesian viewpoint combining the data heterogeneity among clients and properties of neural networks. Furthermore, to validate our findings, we propose an experimental method that excludes neuron disturbing and fuses neural networks via adaptively selecting a local model, called AMS, to execute the prediction according to the input. The experiments demonstrate that AMS is more robust in data heterogeneity than general model fusion and ensemble methods. This implies the necessity of considering neural disturbing in model fusion. Besides, AMS is available for fusing models with varying architectures as an experimental algorithm, and we also list several possible extensions of AMS for future work.
