Journal
OTOLARYNGOLOGY-HEAD AND NECK SURGERY
Volume 147, Issue 6, Pages 1131-1135Publisher
SAGE PUBLICATIONS LTD
DOI: 10.1177/0194599812459703
Keywords
simulation; OSATS; competency assessment; airway foreign body; bronchoscopy; resident education
Categories
Funding
- Olympus
- Medtronic
- AMS
- Resident Research Grant from the American Academy of Otolaryngology-Head & Neck Surgery Foundation
- University of Minnesota SimPORTAL Support Services Grant
Ask authors/readers for more resources
Objective. To determine the interrater reliability and construct validity of 3 separate assessment tools for assessing trainee skills in pediatric airway endoscopy simulation. Design. An Objective Structured Assessment of Technical Skills (OSATS) was developed in which examinees were asked to name and assemble the airway foreign body instruments and retrieve a foreign body from an infant airway mannequin. Each examinee's performance was assessed in a blinded fashion by 3 pediatric otolaryngology faculty at separate residency programs using 3 assessment tools: (1) objective quantifiable measures list (eg, assists needed, forceps openings, foreign body drops), (2) 15-point OSATS checklist, and (3) Global Rating Index for Technical Skills (GRITS). Setting. Otolaryngology residency program. Subjects. Examinees (medical students, n = 3; otolaryngology residents, n = 17; pediatric otolaryngology faculty, n = 3) and raters (n = 3). Main Outcome Measures. Interrater reliability and construct validity. Results. Anonymized split-screen videos simultaneously capturing each examinee's instrument handling and the endoscopic videos were created for all 23 examinees. Nineteen videos were chosen for review by 3 raters. The interrater reliability as measured by the intraclass correlation for objective quantifiable measures ranged from 0.46 to 0.98. The intraclass correlation coefficient was 0.95 for the 15-point OSATS checklist and 0.95 for the GRITS; both showed a high degree of construct validity with scores correlating with previous experience. Conclusion. Assessment tools for skills assessments must have high interrater reliability and construct validity. When assessing trainee skills in pediatric airway foreign body scenarios, the 15-point OSATS checklist developed by this group or the GRITS meets these criteria.
Authors
I am an author on this paper
Click your name to claim this paper and add it to your profile.
Reviews
Recommended
No Data Available