A mobile application to facilitate implementation of programmatic assessment in anaesthesia training

Background: Programmatic assessment is a concept to support learning through continuously providing information about learner progress to trainees and supervisors. Central to the concept are multiple low-stakes workplace-based assessments and meaningful feedback opportunities. Mobile technology may facilitate frequent and concise workplace-based assessments and trigger meaningful feedback. We designed a mobile application (app) for real-time use at the workplace utilising the concept of entrustable professional activities. As the primary outcome, we analysed completion times and as the secondary outcome the quality of documented learning goals. Methods: The prEPAred app requests trainees and supervisors to rate level of supervision of a professional activity directly after completion. Subsequently, ratings are compared, and supervisors may provide feedback via the app. We tested the app in five anaesthesiology departments at major teaching hospitals, analysing completion times, agreement on ratings, and quality of documented learning goals. Results: We recorded 1518 assessments from 159 trainees and 89 supervisors. Median time for level of supervision rating was 56 (inter-quartile range: 39-85) s for trainees and 17 (11-30) s for supervisors. Learning goals via the app were documented in 767 cases (50.5%). Median feedback time was 2 min, 31 s (confidence interval [CI]: 1 min, 20 s to 5 min, 20 s). In 443 (29%) cases, a specific learning goal was documented. A post hoc analysis revealed that the odds of documenting learning goals increased if trainees rated the level of supervision higher than their supervisors (odds ratio 1.39; CI: 1.03-1.87). Conclusions: The prEPAred mobile app enabled frequent and concise documentation of workplace-based assessments. Disagreement in level of supervision rating stimulated documentation of specific learning goals indicating more meaningful feedback. Thus, the tool could advance workplace-based assessments towards programmatic assessment.

Automated summary

This study designed a mobile application to document workplace-based assessments in real-time and stimulate meaningful feedback. The results showed that the app effectively documented learning goals and disagreement in ratings triggered the documentation of specific learning goals, enhancing the feedback process.