Multilevel CFA with Ordered Categorical Data: A Simulation Study Comparing Fit Indices Across Robust Estimation Methods

Within a multilevel confirmatory factor analysis framework, we investigated the ability of commonly used fit indices to discriminate between correctly specified models and misspecified models. Receiver operating characteristics (ROC) analyses were used to evaluate the performance of fit indices. Combining ROC analyses with checks of the convergence rates across Monte Carlo replications and ANOVA for investigating the variation in fit scores across replications, we found converging evidence for the utility of the investigated fit indices. Optimal thresholds based on maximizing sensitivity and specificity for detection of the true model were identified by the highest sensitivity and specificity and found to vary across different robust estimation methods (i.e., MLR, ULSMV, and WLSMV). The estimation method and sample size influenced the performance of common fit indices to detect misspecification of the level-1 model. All fit indices investigated performed poorly for detecting misspecification of the level-2 model when the level-2 sample size was below 100. We offer recommendations of commonly reported fit indices to use (and not use), cutoff criteria to use for specific estimation methods, and cautions about the use of recommended cutoff criteria for ML-CFA.

Automated summary

Within a multilevel confirmatory factor analysis framework, this study examined the ability of commonly used fit indices to differentiate between correctly and incorrectly specified models. Different robust estimation methods and sample sizes were found to influence the performance of fit indices in detecting model misspecification. Recommendations were provided for using appropriate cutoff criteria for specific estimation methods and cautions were given about the use of recommended cutoff criteria for ML-CFA.