Abstract
Background: The Accreditation Council for Graduate Medical Education has developed milestones including procedural skills under the core competency of patient care. Laceration repair is a procedure the Pediatric Residency Review Committee states residents should have training in, and progress in that training should be monitored by residency programs. There exists no validated tool to evaluate pediatric resident laceration repair performance.
Research Question: Does a novel tool for the evaluation of resident laceration repair performance in a simulated laceration repair in the Pediatric Emergency Department (PED) demonstrate validity and reliability as measured by concordance in scores across reviewers and alignment of scores with proxies for procedural skill?
Methodology: A novel tool was developed to evaluate PED resident laceration repair performance. It is based on the validated Objective Structured Assessment of Technical Skills, & consists of two components: a global rating scale (GRS) and a checklist. 30 pediatric and family medicine residents were filmed performing simulated laceration repair. Each video was evaluated by five Emergency or Pediatric Emergency Medicine physicians. Evaluators were blinded to the identity of the proceduralist. To estimate agreement between the evaluators, concordance correlation coefficients (CCC) were calculated for both the GRS and checklist scores. Average scores were calculated for each resident and compared across levels of training and laceration repair experience. For each item, score ranges were calculated for each proceduralist, and then the median range was calculated for each item.
Results: 13 interns, 4 second year, and 13 third year residents were filmed performing laceration repair procedures. The CCC showed fair concordance across reviewers for both the checklist (0.55, 95% CI 0.38-0.69) and the GRS (0.53, 95% CI 0.36-0.67). Scores for both tools improved with increased years of training and reported procedural experience (figures 1 &2). The median range of scores for each 5-point item on the GRS varied between 0.0-2.0.
Conclusions: A novel tool to evaluate resident laceration repair performance in a PED showed fair agreement across reviewers. Proceduralist scores improved by training and experience. The study tool is not precise enough for summative evaluation, but it may be useful to distinguish between trainees who have and have not attained competence in laceration repair for formative feedback. While there was fair variability between scores of reviewers, the gradient scoring of the GRS allowed item scores for each reviewer to provide a more refined evaluation than the simple binary checklist item scores.
