CUED Publications database

Recurrent neural network language model training with noise contrastive estimation for speech recognition

Chen, X and Liu, X and Gales, MJF and Woodland, PC (2015) Recurrent neural network language model training with noise contrastive estimation for speech recognition. In: UNSPECIFIED pp. 5411-5415..

Full text not available from this repository.

Abstract

© 2015 IEEE. In recent years recurrent neural network language models (RNNLMs) have been successfully applied to a range of tasks including speech recognition. However, an important issue that limits the quantity of data used, and their possible application areas, is the computational cost in training. A significant part of this cost is associated with the softmax function at the output layer, as this requires a normalization term to be explicitly calculated. This impacts both the training and testing speed, especially when a large output vocabulary is used. To address this problem, noise contrastive estimation (NCE) is explored in RNNLM training. NCE does not require the above normalization during both training and testing. It is insensitive to the output layer size. On a large vocabulary conversational telephone speech recognition task, a doubling in training speed on a GPU and a 56 times speed up in test time evaluation on a CPU were obtained.

Item Type: Conference or Workshop Item (UNSPECIFIED)
Subjects: UNSPECIFIED
Divisions: Div F > Machine Intelligence
Depositing User: Cron Job
Date Deposited: 17 Jul 2017 19:01
Last Modified: 26 Oct 2017 01:46
DOI: