CUED Publications database

Avoiding pathologies in very deep networks

Duvenaud, D and Rippel, O and Adams, RP and Ghahramani, Z (2014) Avoiding pathologies in very deep networks. Journal of Machine Learning Research, 33. pp. 202-210. ISSN 1532-4435

Full text not available from this repository.


Choosing appropriate architectures and regularization strategies of deep networks is crucial to good predictive performance. To shed light on this problem, we analyze the analogous problem of constructing useful priors on compositions of functions. Specifically, we study the deep Gaussian process, a type of infinitely-wide, deep neural network. We show that in standard architectures, the representational capacity of the network tends to capture fewer degrees of freedom as the number of layers increases, retaining only a single degree of freedom in the limit. We propose an alternate network architecture which does not suffer from this pathology. We also examine deep covariance functions, obtained by composing infinitely many feature transforms. Lastly, we characterize the class of models obtained by performing dropout on Gaussian processes.

Item Type: Article
Divisions: Div F > Computational and Biological Learning
Depositing User: Cron Job
Date Deposited: 17 Jul 2017 19:44
Last Modified: 22 May 2018 06:27