CUED Publications database

Entropy estimation: Simulation, theory and a case study

Kontoyiannis, I (2006) Entropy estimation: Simulation, theory and a case study. In: UNSPECIFIED 257-..

Full text not available from this repository.


We consider the statistical problem of estimating the entropy of finite-alphabet data generated from an unknown stationary process. We examine a series of estimators, including: (1) The standard maximum-likelihood or "plug-in" estimator; (2) Four different estimators based on the family of Lempel-Ziv compression algorithms; (3) A different plug-in estimator especially tailored to renewal processes; and (4) The natural estimator derived from the Context-Tree Weighting method (CTW). Some of these estimators are well-known, and some are new. We first summarize numerous theoretical properties of these estimators: Conditions for consistency, estimates of their bias and variance, methods for approximating the estimation error and for obtaining confidence intervals. Several new theoretical results are developed. We show how the theory offers preliminary indications of their relative performance, and we discuss how the theoretical results offer guidelines for tuning the parameters involved in the estimation process. Then we present an extensive simulation study on various types of synthetic data and under various conditions. We compare their performance and comment on the strengths and weaknesses of the various methods. For each estimator, we develop a precise method for calculating the estimation error based on any specific data set. Finally we report the performance of these entropy estimators on the (binary) spike trains of 28 neurons recorded simultaneously for a one-hour period from the primary motor and dorsal premotor cortices of a quietly seated monkey not engaged in a task behavior. Based on joint work with Yun Gao and Elle Bienenstock. ©2006 IEEE.

Item Type: Conference or Workshop Item (UNSPECIFIED)
Divisions: Div F > Signal Processing and Communications
Depositing User: Cron Job
Date Deposited: 08 Jan 2018 20:12
Last Modified: 18 Aug 2020 12:42