Estimating misclassification error: a closer look at cross-validation based methods.
In: BMC Research Notes, Jg. 5 (2012-02-01), Heft 1, S. 1-11
Online
academicJournal
Zugriff:
Background: To estimate a classifier's error in predicting future observations, bootstrap methods have been proposed as reduced-variation alternatives to traditional cross-validation (CV) methods based on sampling without replacement. Monte Carlo (MC) simulation studies aimed at estimating the true misclassification error conditional on the training set are commonly used to compare CV methods. We conducted an MC simulation study to compare a new method of bootstrap CV (BCV) to k-fold CV for estimating classification error. Findings: For the low-dimensional conditions simulated, the modest positive bias of k-fold CV contrasted sharply with the substantial negative bias of the new BCV method. This behavior was corroborated using a real-world dataset of prognostic gene-expression profiles in breast cancer patients. Our simulation results demonstrate some extreme characteristics of variance and bias that can occur due to a fault in the design of CV exercises aimed at estimating the true conditional error of a classifier, and that appear not to have been fully appreciated in previous studies. Although CV is a sound practice for estimating a classifier's generalization error, using CV to estimate the fixed misclassification error of a trained classifier conditional on the training set is problematic. While MC simulation of this estimation exercise can correctly represent the average bias of a classifier, it will overstate the between-run variance of the bias. Conclusions: We recommend k-fold CV over the new BCV method for estimating a classifier's generalization error. The extreme negative bias of BCV is too high a price to pay for its reduced variance. [ABSTRACT FROM AUTHOR]
Copyright of BMC Research Notes is the property of BioMed Central and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)
Titel: |
Estimating misclassification error: a closer look at cross-validation based methods.
|
---|---|
Autor/in / Beteiligte Person: | Ounpraseuth, Songthip ; Lensing, Shelly Y. ; Spencer, Horace J. ; Kodell, Ralph L. |
Link: | |
Zeitschrift: | BMC Research Notes, Jg. 5 (2012-02-01), Heft 1, S. 1-11 |
Veröffentlichung: | 2012 |
Medientyp: | academicJournal |
ISSN: | 1756-0500 (print) |
DOI: | 10.1186/1756-0500-5-656 |
Schlagwort: |
|
Sonstiges: |
|