Performance of intraclass correlation coefficient (ICC) as a reliability index under various distributions in scale reliability studies

Many published scale validation studies determine inter-rater reliability using the intra-class correlation coefficient . However, the use of this statistic must consider its advantages, limitations, and applicability. This paper evaluates how interaction of subject distribution, sample size, and levels of rater disagreement affects and provides an approach for obtaining relevant estimates under suboptimal conditions. Simulation results suggest that for a fixed number of subjects, from the convex distribution is smaller than for the uniform distribution, which in turn is smaller than for the concave distribution. The variance component estimates also show that the dissimilarity of among distributions is attributed to the study design (ie, distribution of subjects) component of subject variability and not the scale quality component of rater error variability. The dependency of on the distribution of subjects makes it difficult to compare results across reliability studies. Hence, it is proposed that reliability studies should be designed using a uniform distribution of subjects because of the standardization it provides for representing objective disagreement. In the absence of uniform distribution, a sampling method is proposed to reduce the non-uniformity. In addition, as expected, high levels of disagreement result in low , and when the type of distribution is fixed, any increase in the number of subjects beyond a moderately large specification such as does not have a major impact on ICC.