CUED Publications database

Deterministic variational inference for robust Bayesian neural networks

Wu, A and Nowozin, S and Meeds, E and Turner, RE and Hernández-Lobato, JM and Gaunt, AL (2019) Deterministic variational inference for robust Bayesian neural networks. 7th International Conference on Learning Representations, ICLR 2019.

Full text not available from this repository.

Abstract

© 7th International Conference on Learning Representations, ICLR 2019. All Rights Reserved. Bayesian neural networks (BNNs) hold great promise as a flexible and principled solution to deal with uncertainty when learning from finite data. Among approaches to realize probabilistic inference in deep neural networks, variational Bayes (VB) is theoretically grounded, generally applicable, and computationally efficient. With wide recognition of potential advantages, why is it that variational Bayes has seen very limited practical use for BNNs in real applications? We argue that variational inference in neural networks is fragile: successful implementations require careful initialization and tuning of prior variances, as well as controlling the variance of Monte Carlo gradient estimates. We provide two innovations that aim to turn VB into a robust inference tool for Bayesian neural networks: first, we introduce a novel deterministic method to approximate moments in neural networks, eliminating gradient variance; second, we introduce a hierarchical prior for parameters and a novel Empirical Bayes procedure for automatically selecting prior variances. Combining these two innovations, the resulting method is highly efficient and robust. On the application of heteroscedastic regression we demonstrate good predictive performance over alternative approaches.

Item Type: Article
Uncontrolled Keywords: cs.LG cs.LG stat.ML
Subjects: UNSPECIFIED
Divisions: Div F > Computational and Biological Learning
Depositing User: Cron Job
Date Deposited: 30 May 2019 01:15
Last Modified: 18 Aug 2020 12:49
DOI: