Experiential Learning in Patient Safety: A Multi-Center Study Examining Emergency Medicine Residents' Situational Awareness of Hazards via Simulation

Introduction The 2016 Clinical Learning Environment Review established that experiential patient safety curricula for residents are uncommon. Moreover, these curricula do not incorporate non-technical skills linked to safety, such as situational awareness (SA). We developed an in-situ patient safety simulation exercise incorporating core SA concepts and subsequently assessed exercise feasibility and acceptability, and measured residents' safety SA. Methods A simulation scenario and mock chart were designed, incorporating 16 patient safety hazards. Residents at two institutions reviewed the chart and had 10 minutes in an emergency department room with the simulated scenario to document identified hazards, followed by a facilitated debriefing. Pre- and post-exercise surveys were completed. We used regression analyses to compare exercise performance and survey responses by training year, and measures of proportional difference and association for survey responses. Results This study included 76 of 104 eligible residents (73.1%). Around 56.5% initially reported being comfortable identifying hazards. During the exercise, hazards requiring higher SA were identified less frequently. Senior residents identified more hazards (OR 2.26; 95%CI 1.56-3.28) (mean 8.28, SD1.45); 93.4% expressed satisfaction with the session, and residents reporting comfort increased significantly (89.5%, p<0.001). Conclusion In-situ simulation incorporating SA concepts feasibly provides experiential safety education and enhances resident comfort with safety issues. Visible hazards were often identified; those requiring information synthesis were usually missed, suggesting a need for developing more robust resident SA. While interns demonstrated the poorest performance, senior residents only identified 50% of errors, indicating that patient safety education enhancing SA should begin early and continue longitudinally.


Introduction
National healthcare agencies and educational organizations have identified patient safety as a key intervention target across a broad range of clinical environments [1,2]. The emergency department (ED) is particularly susceptible to safety hazards, given large volumes of high-acuity patients, distractions, and interdisciplinary teams prone to circadian fatigue [3]. Hazards may lead to "adverse events" (4.1/100 patient visits), which constitute injuries or harms suffered by patients due to medical care, as well as "near misses" (5.4/100 patient visits), wherein an event could have had "adverse consequences but did not and was indistinguishable from fully fledged adverse events in all but outcome [4]." Adverse events may be preventable and are associated with recidivism and mortality [5,6].
Despite this, patient safety curricula targeting safety hazards remain deficient in graduate medical education (GME) and little is known about residents' awareness of safety hazards in clinical settings [7]. In 2016, the Clinical Learning Environment Review (CLER) reported that residents lack awareness of key safety concepts, and "while many Clinical Learning Environments (CLEs) provided didactic training in patient safety, it was uncommon for CLEs to provide . . . opportunities for experiential learning [2]." Studies involving nurses, pharmacists, medical students, and interns have demonstrated the feasibility of simulation (SIM) models to both understand existing trainee knowledge and enhance experiential patient safety learning [8][9][10][11][12][13]. However, implementation and evaluation of SIM-based patient safety exercises are lacking for residency trainees; a recent systematic review found that of 27 residency-based programs for teaching patient safety, none used SIM [14].
Additionally, despite growing recognition that non-technical skills (NTS) are of increasing importance in safe patient care [1], patient safety curricula explicitly incorporating non-technical skills (NTS) are limited. NTS are cognitive and social competencies that complement technical skills and permit safe and efficient communication, decision-making, and task performance [15]. Such skills have long been implicated in enhancing safety in high-risk industries such as aviation and the military. Specifically, situational awareness (SA), defined as "perception of elements in the environment within a volume of time and space, comprehension of their meaning, and projection of their status in the near future," is an integral NTS in improving patient safety in complex environments [1,[15][16][17][18]. It has been suggested as a key skill in emergency medicine (EM) and may enhance physicians' ability to recognize and predict sources of harm.
In anesthesia, surgery, critical care, and EM, SIM has been suggested as an ideal environment to address clinicians' SA [19][20][21]. However, SA is rarely incorporated into patient safety education in EM [1,15,16]. Moreover, SIM-based safety education is rarely conducted in the context of the clinical environment, often occurring instead in SIM centers, which do not always emulate the complexity of real clinical settings, such as ambient physical stressors, distractions, and clinical devices that can negatively impact SA [19,[21][22][23].
Given the paucity of experiential curricula for GME patient safety education, the importance of SA in identifying hazards, and the potential utility of SIM in safety education and in heightening SA, we incorporated concepts of SA into a SIM-based patient safety exercise designed to assess and potentially enhance residents' SA of safety hazards in the clinical environment of the ED.
Our objectives were to assess resident SA of patient safety in clinical settings and to evaluate the feasibility and acceptability of employing an in-situ SIM in patient safety education.
This work was presented as a poster presentation at the Society for Academic Emergency Medicine's Annual Meeting in May 2018 in Indianapolis, Indiana.

Study setting
This study was conducted with EM residents at two Accreditation Council for Graduate Medical Education (ACGME)-accredited three-year programs at tertiary care centers. Program 1 has 10 residents per postgraduate year (PGY); program 2 has 16 residents per PGY. Neither had an existing formal patient safety curriculum. Both institutions' ethical review boards approved this study as exempt as participation was part of each site's routine residency educational curriculum.

Study design
We conducted a prospective, observational study utilizing in-situ SIM with pre-and post-exercise surveys. Our primary objectives were to (1) obtain a baseline surrogate measure of residents' SA by tabulating how many hazards residents were able to identify in a simulated environment that mimicked some elements of their actual patient care environments, and (2) assess intervention feasibility and acceptability. Intervention impact on residents' comfort with hazard identification represented a secondary outcome.
Common and serious hazards to patient safety in local ED environments were identified by a panel of clinical leaders, including nursing staff and physicians, and supplemented by a literature review. Subsequently, 16 safety hazards of varying complexity deemed important to the local context and aligned with national patient safety priorities were identified [3,24]. Hazards were classified into levels of SA based on Endsley's model [18]. (Tables 1-3). Study investigators determined what constituted "correct" hazard identification and rectification (i.e., proposed solution).   Patient is a fall risk and not wearing fall bracelet 2 6 (7.9)

TABLE 3: Hazards simulated, associated level of situational awareness, and proportion of residents identifying hazard correctly
Hazards were then incorporated into a SIM scenario which included a mock patient chart and a simulated patient room set-up of the mock patient. The mock chart was integrated into the patient health record system used at each individual site (Supplemental Appendix 1). The SIM was conducted in an ED patient room not in use at that time. A SimMan® 3G mannequin (Laerdal Medical, Stavanger, Norway) and existing ED equipment, including monitors, infusion pumps, and computers, were utilized to simulate hazards; both sites had identical setups and pilot setups were photographed to ensure consistency.
Surveys were designed to capture the acceptability and feasibility of the SIM as an educational intervention and its impact on resident comfort; the latter has been identified as a potential area of deficiency [25]. Face validity was confirmed via group consensus (Supplemental Appendix 2 and 3).
SIM sessions were incorporated into existing didactic time, lasted less than an hour, and could be completed with a single faculty facilitator. A five-minute-long pre-briefing using a standardized script was conducted by one facilitator (clinician-educator) at each site (Supplemental Appendix 4).
After a pre-intervention survey, residents independently reviewed the mock chart at a clinical workstation, while exposed to ambient stressors and sources of distraction, including noises inherent to the clinical environment. They then entered the room in groups of four to five, to minimize the need for scenario repetition, and were given 10 minutes to independently document identified hazards and proposed solutions during the exercise. They were instructed not to interact with each other or manipulate any objects. No new information was provided and no changes in patient status occurred once residents had entered the room. A researcher monitored the exercise to ensure protocol adherence. During a subsequent facilitator-led debriefing session using a semi-structured script, participants discussed identified hazards and reflected on potential solutions, then completed a post-exercise survey.

Analysis
Survey responses and scored lists of documented hazards were tabulated into Excel (Microsoft Office, 2017) by two independent reviewers, with discrepancies adjudicated by group consensus. Analyses were conducted using STATA 12.0 (Stata Corp, College Station, TX); all levels of significance were set at p=0.05.
Descriptive statistics were used for the frequency of hazards identified. Linear regression was utilized to assess whether training site, year-in-training, and pre-intervention comfort predicted exercise performance. Descriptive statistics were used to summarize survey responses. Correlations between training year and reported experience and comfort with hazards and the number of correct solutions proposed were assessed with logistic regression. Responses between sites were compared with chi-square tests. Two-sample tests of proportions were used to compare self-reported comfort pre-and post-exercise.

Hazards identified during exercise
Hazards necessitating a lower level of SA were most commonly identified ( During debriefing, most solutions discussed by residents included directly addressing the issue themselves (e.g. raising the bed rail while in the room) or instructing nursing or housekeeping staff to address the hazard. There were fewer proposed upstream solutions, such as improving team education and awareness of hazards, or systematic scheduled checks to ensure the availability of key supplies.

Post-exercise comfort with addressing hazards
Residents reporting comfort with identifying and rectifying hazards increased significantly post-exercise to 68 (89.5%) and 64 (84.2%), respectively (p-value <0.001 at each site).

Discussion
SA, which has been implicated in harm reduction in industries such as aviation, is especially relevant to safety in acute care medicine [15][16][17]. Our study of a SIM-based pilot intervention conducted in-situ in the ED identifies gaps in existing resident awareness of safety hazards at two sites and demonstrates that an insitu SIM targeting safety hazards and grounded in concepts of SA is 1) feasible and acceptable to learners, and 2) may enhance resident comfort with addressing safety hazards.
In our study, incoming interns identified the fewest hazards. Patient safety curricula in medical schools are lacking; only 25% of American and Canadian medical schools report having such curricula. Of these, only 30% use SIM [26]. Moreover, junior trainees have limited clinical exposure and are vulnerable to cognitive overload and stress, which may contribute to a decreased ability to recognize critical cues (level 1 SA), limited working memory (level 2 SA), and projection of future courses of action (level 3 SA). While there are few substitutes for time spent in the clinical environment, SIM scenarios could serve as an important auxiliary tool for junior trainees to "increase" clinical experience, enhance cue recognition, and exercise working memory in clinical contexts.
Senior residents demonstrated superior performance as compared to junior residents. Greater clinical exposure permits "recognition-primed" decision-making, allowing advanced trainees to match situations to learned patterns to make rapid accurate projections [17,18,27]. A prior study of EM residents in the clinical environment also demonstrated that increased years of training were associated with increased clinical SA [28]. However, in our study, senior residents on average only identified 50% of hazards, and identification rates plateaued amongst PGY2 and 3 residents. Senior residents may incorrectly apply patterns to situations or develop automatic processing, which makes them less responsive to new stimuli, for instance, changes in patient condition or monitors alarming [17,18]. Repeated comparisons of mental models with actual situations permit trainees to cross-check their mental models; longitudinally-scheduled SIM exercises may help ensure that senior residents are not applying inappropriate mental models to high-risk situations.
Our findings also suggest that further development of level 2 and 3 SA in relation to patient safety is needed.
Hazards that were wholly dependent on visual cue detection and simple identification of elements (level 1 SA), such as exposed sharps, were frequently identified. However, hazards mandating synthesis of perceived disparate elements (level 2 SA), i.e. visual cues (e.g. an allergy band on a patient's wrist) and chart information (e.g. absence of allergies listed in the chart), were less frequently identified across all PGYs.
Hazards that demanded perception, synthesis, and subsequent anticipation of system vulnerabilities and patient deterioration (level 3 SA) were rarely identified (e.g. lack of bag-valve mask). While our findings are from a small sample size at two institutions, other studies corroborate the relative paucity of synthesis and anticipation skills of residents as they pertain to patient safety [29].
SIM represents an ideal instructional modality to maximize experiential learning and assess context-specific safety SA. Our study demonstrates that SIM-based interventions for patient safety education for residents can be reasonably performed in-situ with one facilitator using existing ED resources at a relatively low cost. Conducting the SIM within the clinical environment may also contribute to generating SA adapted to the complex spatial, auditory, and visual specifics of the trainees' clinical environment and allow interfacing with the electronic medical record (EMR) and existing clinical devices. Given the relative brevity of the exercise, it could be easily integrated into existing didactic time or clinical schedules, and may also lend itself to longitudinal curricula. Most participants expressed satisfaction with the intervention, indicating the acceptability of the format to learners.
Participants also perceived increased comfort with identifying hazards in the clinical setting after participating in the SIM and debriefings. Debriefings provided a reflective period that could have contributed to trainee metacognition which may have contributed to enhanced comfort or SA.
Successful implementation of this in-situ intervention was contingent on a number of factors, including the engagement of clinical stakeholders, such as medical directors and nursing leadership, who provided guidance on which hazards to incorporate and assisted with securing space and supplies. Integration of the activity into existing protected didactic times allowed us to obtain a robust number of participants for this intervention and also underscored the importance of the topic to residents.

Limitations
Our study was subject to certain limitations. Residents' comfort level was self-reported and subject to bias, and recall bias may have impacted previously identified hazards reported. Our exercise likely overestimated residents' ability to recognize hazards, as the scenario did not replicate the full cognitive load and task burden of clinical shifts, and residents were primed to look for hazards prior to the exercise. Though residents were instructed not to interact, there is still a small possibility of peer influence or discussion through nonverbal cues.
The hazards chosen in this study were culled from national literature but adapted to local priorities. As such, specific study findings may not be generalizable. However, the intervention described here is easily adaptable to fit local needs.

Conclusions
Our study suggests a need for enhancing EM residents' SA, especially at higher SA levels. Inferior performance at earlier stages of training indicates that safety education should be instituted early in residency; the plateau in senior residents' performance suggests that safety education should continue longitudinally. Our study also demonstrates that an in-situ SIM incorporating key concepts of SA is feasible, acceptable to learners, and may enhance learner comfort with addressing clinical hazards. The intervention described here is not resource-intensive and can be easily translated to other clinical sites to assess clinicians' SA or provide experiential patient safety education to trainees by adapting scenarios to sitespecific needs and hazards. Our approach lays the groundwork for future studies that could examine educational and behavioral impacts by tracking hazard identification across longitudinal exercises and by monitoring clinical harm rates or residents' use of safety reporting mechanisms.
To move beyond a baseline assessment of existing resident SA in static situations, future studies could also consider integrating tools such as the Situation Awareness Global Assessment Technique (SAGAT) to evaluate hazard awareness during the provision of simulated patient care or management of multiple simultaneous "patients," tasking residents to evaluate a patient and pausing the scenario to query residents about hazards. While the impact of such interventions on clinical outcomes is yet to be determined, the development of SA via in-situ SIM-based exercises could represent an important educational modality to assess and enhance the practice of safer clinical care.

Supplemental Appendix 1. Mock chart details
Chief complaint at presentation: Diarrhea, fever Brief note by physician screening patient in triage area: History of Present Illness: 60 yo male with history of COPD, opiate abuse presents with 2 days of subjective fevers and diarrhea. He says he was recently in a hospital for pneumonia treated with antibiotics. He has had a lot of diarrhea, it's watery and foul-smelling. He also feels lightheaded and has fallen multiple times at home. He generally can't make it to the bathroom on time.
Review of systems:

Supplemental Appendix 4. Pre-briefing instructions
Verbal instructions were as follows: "You will be entering this ED room with the task of identifying patient safety hazards and will be given time afterward to formulate solutions. You will have a patient chart to read prior to entering the room. Please observe and record any patient safety hazards you see but do not communicate with each other or physically interact with the simulated environment. Your responses will be anonymous. You will have ten minutes inside the room."

Additional Information Disclosures
Human subjects: Consent was obtained or waived by all participants in this study. UT Health San Antonio Institutional Review Board issued approval HSC20170127E. The IRB at UT Health San Antonio approved this research. Animal subjects: All authors have confirmed that this study did not involve animal subjects or tissue. Conflicts of interest: In compliance with the ICMJE uniform disclosure form, all authors declare the following: Payment/services info: All authors have declared that no financial support was received from any organization for the submitted work. Financial relationships: All authors have declared that they have no financial relationships at present or within the previous three years with any organizations that might have an interest in the submitted work. Other relationships: All authors have declared that there are no other relationships or activities that could appear to have influenced the submitted work.