The Quantitative Evaluation of Functional Neuroimaging Experiments: The NPAIRS  Stephen C. Strother, Jon Anderson, Lars Kai Hansen
 Abstract  We introduce a dataanalysis framework and performance metrics for evaluating and optimizing the interaction between activation tasks, experimental designs, and the methodological choices and tools for data acquisition, preprocessing, data analysis, and extraction of statistical parametric maps (SPMs). Our NPAIRS (nonparametric prediction, activation, influence, and reproducibility resampling) framework provides an alternative to simulations and ROC curves by using real PET and fMRI data sets to examine the relationship between prediction accuracy and the signaltonoise ratios (SNRs) associated with reproducible SPMs. Using crossvalidation resampling we plot trainingtest set predictions of the experimental design variables (e.g., brainstate labels) versus reproducibility SNR metrics for the associated SPMs. We demonstrate the utility of this framework across the wide range of performance metrics obtained from \[/sup 15/ O]water PET studies of 12 age and sexmatched data sets performing different motor tasks (8 subjects/set). For the 12 data sets we apply NPAIRS with both univariate and multivariate dataanalysis approaches to: (1) demonstrate that this framework may be used to obtain reproducible SPMs from any dataanalysis approach on a common Z score scale (rSPM{ Z }); (2) demonstrate that the histogram of a rSPM{ Z } image may be modeled as the sum of a dataanalysisdependent noise distribution and a taskdependent, Gaussian signal distribution that scales monotonically with our reproducibility performance metric; (3) explore the relation between prediction and reproducibility performance metrics with an emphasis on biasvariance tradeoffs for flexible, multivariate models; and (4) measure the broad range of reproducibility SNRs and the significant influence of individual subjects. A companion paper describes learning curves for four of these 12 data sets, which describe an alternative mutualinformation prediction metric and NPAIRS reproducibility as a function of trainingset sizes from 2 to 18 subjects. We propose the NPAIRS framework as a validation tool for testing and optimizing methodological choices and tools in functional neuroimaging  Keywords  NPAIRS, neuroimaging, modeling  Type  Journal paper [With referee]  Journal  NeuroImage  Year  2002 Vol. 15 No. 4 pp. 747771  Electronic version(s)  [pdf]  BibTeX data  [bibtex]  IMM Group(s)  Intelligent Signal Processing 
