Development and pilot-testing of a feasible, reliable, and valid operative competency assessment tool for endoscopic sinus surgery

Background: Otolaryngology residency programs are required by the Accreditation Council for Graduate Medical Education to evaluate residents' operative competency. Many such tools based on the model of objective structured assessment of technical skills (OSATSs) have been developed in other surgical specialties, but no such instruments exist for otolaryngologic procedures except for tonsillectomy. Endoscopic sinus surgery (ESS) is among the most common rhinologic procedure and lends itself to objective evaluation of operative competency. The purpose of this study was to develop and test a new tool for ESS, focusing on feasibility, content and construct validity, and interrater agreement that can be used for such assessment in the operating room and the cadaver dissection course. Methods: Faculty input via the modified Delphi technique helped develop the content Of a new OSATS-based instrument. The instrument underwent serial improvements based on 3 years of endoscopic sinus surgery (ESS) cadaver courses. All evaluations were used to calculate construct validity while paired observations were used to determine interrater agreement. Regional and national faculty input was incorporated for increasing generalizability. Internal consistency was calculated using Cronbach's alpha. Results: A total of 51 assessments were completed for 28 residents who were evaluated by 15 faculty members as they performed ESS oil cadavers over a period of 3 years. A high degree of internal consistency (0.99) and feasibility was noted for the instrument, which took 7 minutes to complete. The interrater agreement improved with focused faculty development for the 3rd year of the course. Conclusion: Our results and experience suggest that a feasible, reliable, and valid instrument for objective evaluation of operative competency can be developed for ESS. Further experience at other otolaryngology programs and efforts focused on faculty development will be needed to enhance faculty buy-in. The instrument can be used for formative and summative feedback as well as for identifying residents needing remediation. (Am J Rhinol Allergy 23, 354-359, 2009; doi: 10.2500/ajra.2009.23.3275)