Validation of an integrated method for determining cognitive ability: Implications for routine assessments and clinical trials

Introduction: Although accurate diagnosis of deficit of mild intensity is critical, various methods are used to assess, dichotomize and integrate performance, with no validated gold standard. This study described and validated a framework for the analysis of cognitive performance. Methods: This study was performed by using the Groupe de Reflexion sur L'Evaluation des Fonctions EXecutives (GREFEX) database (724 controls and 461 patients) examined by 7 tests assessing executive functions. The first phase determined the criteria for the cutoff scores, the second phase, the effect of test number on diagnostic accuracy and the third phase, the best methods for combining test scores into an overall summary score. Four validation criteria were used: determination of impaired performance as compared to expected one, false-positive rate <= 5%, detection of both single and multiple impairments with optimal sensitivity. Results: The procedure based on 5th percentile cutoffs determined from standardized residuals was the most appropriate procedure. Although area under the curve (AUC) increased with the number of scores (p =.0001), the false-positive rate also increased (p =.0001), resulting in suboptimal sensitivity for detecting selective impairment. Two overall summary scores, the average of the seven process scores and the Item Response Theory (IRT) score, had significantly (p =.0001) higher AUCs, even for patients with a selective impairment, and provided higher resulting prevalence of dysexecutive disorders (p =.0001). Conclusions: The present study provides and validates a generative framework for the interpretation of cognitive data. Two overall summary score met all 4 validation criteria. A practical consequence is the need to profoundly modify the analysis and interpretation of cognitive assessments for both routine use and clinical research. (C) 2014 Elsevier Ltd. All rights reserved.