Validation of the behavior and concept based assessment of professionalism competence in postgraduate first-year residents

Background: The evaluation of professional behaviors and concepts of postgraduate first-year (PGY(1)) residents has been identified as an area for development. This study examined the efficiency of the professionalism-assessing objective structured clinical examination (OSCE), 360 degrees evaluation, and mini-Clinical Examination Exercise scores (mini-CEX; p-OSCE, p-360 degrees evaluation, and p-mini-CEX scores). Methods: Between January 2009 and January 2012, 189 PGY(1) residents were evaluated for behavior- and concept-based professionalism competence based on the above three methods using two checklists unique to each case. Data were analyzed for reliability, inter-rater agreement, interval changes, and gender-related difference for each method. Results: The test reliabilities of p-OSCE, p-360 degrees evaluation, and p-mini-CEX were acceptable. Further, the reliability of concept and combined p-OSCE was higher than that of behavior p-OSCE. In addition, the concept OSCE p-scores and behavior 360 degrees evaluation p-scores were significantly improved after 6 months of training. The inter-rater agreements were relatively good in p-OSCE and p-360 degrees evaluation. Interestingly, male PGY(1) residents had higher behavior 360 degrees evaluation p-scores from nurses than those of females, whereas female PGY(1) residents had higher behavior 360 degrees evaluation p-scores from patients than those of males. Behavior and concept OSCE p-scores were positively correlated with behavior 360 degrees evaluation p-scores. In comparison with p-360 degrees evaluation, the combination of p-360 degrees evaluation + OSCE + mini-CEX significantly increases their reliabilities. Conclusion: The current study suggests that the p-OSCE, p-360 degrees evaluation, and p-mini-CEX are feasible methods for evaluating professionalism in clinical training of PGY(1) residents. Combination of the above three evaluations, participation, and support from multiple constituencies and multiple representatives provides good reliability and adds credibility in the assessment of professionalism competence. Copyright (c) 2013 Elsevier Taiwan LLC and the Chinese Medical Association. All rights reserved.