Measuring the performance of neural models

Research output: Journal Publications and Reviews (RGC: 21, 22, 62)21_Publication in refereed journal

22 Scopus Citations
View graph of relations

Author(s)

Detail(s)

Original languageEnglish
Article number10
Journal / PublicationFrontiers in Computational Neuroscience
Volume10
Issue numberFEB
Publication statusPublished - 10 Feb 2015
Externally publishedYes

Abstract

Good metrics of the performance of a statistical or computational model are essential for model comparison and selection. Here, we address the design of performance metrics for models that aim to predict neural responses to sensory inputs. This is particularly difficult because the responses of sensory neurons are inherently variable, even in response to repeated presentations of identical stimuli. In this situation, standard metrics (such as the correlation coefficient) fail because they do not distinguish between explainable variance (the part of the neural response that is systematically dependent on the stimulus) and response variability (the part of the neural response that is not systematically dependent on the stimulus, and cannot be explained by modeling the stimulus-response relationship). As a result, models which perfectly describe the systematic stimulus-response relationship may appear to perform poorly. Two metrics have previously been proposed which account for this inherent variability: Signal Power Explained (SPE, Sahani and Linden, 2003), and the normalized correlation coefficient (CCnorm, Hsu et al., 2004). Here, we analyze these metrics, and show that they are intimately related. However, SPE has no lower bound, and we show that, even for good models, SPE can yield negative values that are difficult to interpret. CCnorm is better behaved in that it is effectively bounded between -1 and 1, and values below zero are very rare in practice and easy to interpret. However, it was hitherto not possible to calculate CCnorm directly; instead, it was estimated using imprecise and laborious resampling techniques. Here, we identify a new approach that can calculate CCnorm quickly and accurately. As a result, we argue that it is now a better choice of metric than SPE to accurately evaluate the performance of neural models.

Research Area(s)

  • Model selection, Neural coding, Receptive field, Sensory neuron, Signal power, Statistical modeling

Citation Format(s)

Measuring the performance of neural models. / Schoppe, Oliver; Harper, Nicol S.; Willmore, Ben D. B.; King, Andrew J; Schnupp, Jan W. H.

In: Frontiers in Computational Neuroscience, Vol. 10, No. FEB, 10, 10.02.2015.

Research output: Journal Publications and Reviews (RGC: 21, 22, 62)21_Publication in refereed journal