Journal
INTERNATIONAL JOURNAL ON ARTIFICIAL INTELLIGENCE TOOLS
Volume 24, Issue 5, Pages -Publisher
WORLD SCIENTIFIC PUBL CO PTE LTD
DOI: 10.1142/S0218213015400230
Keywords
Performance estimation; model selection; cross validation; stratification; comparative evaluation
Categories
Funding
- STATegra EU FP7 project [306000]
- EPILOGEAS GSRT ARISTEIA II project [3446]
Ask authors/readers for more resources
In a typical supervised data analysis task, one needs to perform the following two tasks: (a) select an optimal combination of learning methods (e.g., for variable selection and classifier) and tune their hyper-parameters (e.g., K in K-NN), also called model selection, and (b) provide an estimate of the performance of the final, reported model. Combining the two tasks is not trivial because when one selects the set of hyper-parameters that seem to provide the best estimated performance, this estimation is optimistic (biased/overfitted) due to performing multiple statistical comparisons. In this paper, we discuss the theoretical properties of performance estimation when model selection is present and we confirm that the simple Cross-Validation with model selection is indeed optimistic (overestimates performance) in small sample scenarios and should be avoided. We present in detail and investigate the theoretical properties of the Nested Cross Validation and a method by Tibshirani and Tibshirani for removing the estimation bias. In computational experiments with real datasets both protocols provide conservative estimation of performance and should be preferred. These statements hold true even if feature selection is performed as preprocessing.
Authors
I am an author on this paper
Click your name to claim this paper and add it to your profile.
Reviews
Recommended
No Data Available