Abstract

A comprehensive evaluation strategy is essential to determine the effectiveness and efficiency of a simulation program. The BSN program in a Canadian university employed Kirkpatrick's four-level Model of Evaluation Training (MET) as a theoretical framework to evaluate its simulation program at the reaction, learning, behavior, and results levels. The reaction level focuses on student satisfaction, self-perceived outcomes, and self-reflection, and is measured by the simulation effectiveness tool: modified (SET-M). The learning level emphasizes the achievement of learning outcomes in each simulation activity. This level of evaluation is based on a list of behavioral indicators developed and validated by the simulation team and subject matter experts from the teaching teams for each simulation scenario. Meanwhile, since most of the simulation activities in the BSN program are embedded in the clinical courses, the behavior-level outcome is determined by the desired practice observed in clinical practice and measured by the Creighton competency evaluation instrument (CCEI). As safety and quality of patient care are the overarching learning outcomes in the clinical courses and BSN program, the results-level is measured through the clinical course pass and fail rate.

The four-level of evaluation strategy was piloted on three simulation activities in the winter 2022 term and expanded to seven simulation activities in the spring 2022 term. The findings from the winter 2022 term pilot have revealed similar gaps in the learning and behavior levels of outcomes. For instance, only 60% of students re-assessed the patient after providing nursing intervention in both simulation activities and clinical practice. More findings will be shared after the spring 2022 term data collection.