论文标题
修复贝叶斯神经网络的一个简单技巧
One Simple Trick to Fix Your Bayesian Neural Network
论文作者
论文摘要
贝叶斯神经网络（BNN）中最流行的估计方法之一是平均场变异推理（MFVI）。在这项工作中，我们表明具有RELU激活功能的神经网络会诱导后代，而后者很难与MFVI相吻合。我们为这种现象提供了理论上的理由，通过经验研究，并报告了一系列实验的结果，以研究激活函数对BNNS校准的影响。我们发现，使用泄漏的恢复激活会导致高斯的重量后代更多，并且比基于RELU的对应物的预期校准误差（ECE）较低。
One of the most popular estimation methods in Bayesian neural networks (BNN) is mean-field variational inference (MFVI). In this work, we show that neural networks with ReLU activation function induce posteriors, that are hard to fit with MFVI. We provide a theoretical justification for this phenomenon, study it empirically, and report the results of a series of experiments to investigate the effect of activation function on the calibration of BNNs. We find that using Leaky ReLU activations leads to more Gaussian-like weight posteriors and achieves a lower expected calibration error (ECE) than its ReLU-based counterpart.
