Simulation as a Platform for Development of Entrustable Professional Activities: A Modular, Longitudinal Approach

Background The Association of American Medical Colleges (AAMC) has recently identified a list of integrated activities to be expected of all medical school graduates entering residency: the core Entrustable Professional Activities (EPAs). Direct observation and deliberate practice of individual EPA behaviors in the clinical setting has multiple challenges, and there is limited literature describing a comprehensive, longitudinal curriculum dedicated to formative EPA assessment. Approach We present a model curriculum to develop and provide formative assessment of EPA skills longitudinally throughout the clinical years. Each EPA-focused training session includes a simulation case followed by several small group activities with content related to the clinical vignette in the initial simulation. We have designed this curriculum to be longitudinal and modular, and present the general framework here. Outcomes Step-wise implementation began in 2013. Over 450 medical students have participated in the third year (MS3) clerkship sessions, 30 in the MS4 sub-internship sessions, and over 300 thus far in the fourth year (MS4) capstone course, including students from 10 different medical schools. MS3 sessions focused on EPAs 4, 7, 8, 9, 10, and MS4 sessions had an additional focus on EPA 8. The capstone course encompassed nearly all 13 EPAs in active simulation-based learning. Opportunities to provide formative assessment through on-the-spot feedback exist throughout the curriculum. Student feedback was overwhelmingly positive. Next steps We found that simulations are an effective method of providing formative assessment of EPAs that are exceptionally well-received by medical students. We have demonstrated that these can be implemented for medical students from multiple educational backgrounds. We believe that deliberate practice and longitudinal formative assessment is of utmost importance in effectively developing core EPAs prior to final entrustment decisions.


Introduction
The Association of American Medical Colleges (AAMC) has recently identified a list of integrated activities to be expected of all medical school graduates entering residency: the core Entrustable Professional Activities (EPAs) [1]. Entrustment denotes competency in specific professional activities that can be carried out without direct supervision on day one of residency [1]. It has been suggested that EPAs be incorporated into teaching in the fourth year (MS4), but formative assessment that begins earlier in clinical training allows for opportunities for feedback and improvement before entrustment decisions are made [2][3][4][5][6][7]. While EPA assessment should ideally occur in real-time clinical settings, work-place based assessment of some EPAs is often not feasible, and sometimes not ethical. For example, a medical student would not be expected to independently initiate management of a critically ill patient in most situations.
An alternate solution is to employ simulation as a platform for developing and providing formative assessment of EPA behaviors. Simulation can reproduce clinical scenarios akin to those encountered on the wards and in the clinic, and has been effectively utilized for internship bootcamp courses [8,9]. Simulation prompts critical thinking and action by engaging the learner in dynamic interactions with a standardized patient or manikin, whereas traditional objective structured clinical exams (OSCEs) are often static. By capturing changes in patient vital signs and clinical status in response to interventions, simulation requires the learner to interact with mixed media inputs at various time points, mimicking the stochastic nature of actual clinical encounters. Previous studies have explored the use of simulation to evaluate EPAs at the end of the MS4 year, and have often focused on a single EPA, namely EPA 10 [10,11]. Our curricular framework is different as it includes numerous formative checkpoints with opportunities to encompass all EPAs, and is embedded throughout the third and fourth years of medical school.

Approach
We developed a simulation-based curriculum that spans the MS3 and MS4 years and incorporates multiple opportunities for formative assessment during each exercise. This curriculum aims to fulfill all of the elements of the 2018 Ottawa Criteria for good formative assessment [12]. The exercises included in this approach provide specific and actionable feedback that is immediate (timely) and ongoing (longitudinal). By nature, simulation is low-stakes, in that patient care is not at risk, and in our use, students were not graded but provided with immediate feedback via debriefing undertaken as a group with a skilled faculty. Formative feedback is embedded at multiple points throughout the clinical exercise. Most importantly, these exercises are intended to stimulate learning, emphasizing the catalytic effect.
This approach is designed to be modular, flexible, and generalizable. Students begin the curriculum in the MS3 year, engaging in one workshop per clerkship, typically at the midway point of the clerkship. The content of the workshop is based on a case relevant to that clerkship. We have adapted this to multiple clerkships and currently utilize this approach in four of the core clerkships (Internal Medicine, Surgery, Neurology, and Pediatrics).
Intern capstone courses have demonstrated success in improving intern practical skills on the wards [9]. Incorporation of EPAs into the capstone fourth year "bootcamp" course began in 2013. Finding success with this approach, and discovering significant gaps in undergraduate EPA focused curricula (example: prescription writing and handoffs), deliberate practice of EPAs was undertaken earlier in the clinical years. The MS3 Clerkship simulations were implemented in January 2016 at Institution one and the MS4 clerkship simulations were implemented at Institution two in July 2016. The planning committee included bootcamp course directors, MS3 and MS4 clerkship directors, and simulation directors. Institution one is a northeast medical school graduating between 160 and 190 students per year. Institution two is a clinical site of the Mayo Clinic Alix School of Medicine with over 100 visiting fourth-year students each year.
Each workshop was based on a simulation case and designed to have modular elements that may be utilized or omitted based on goals of the session for each clerkship. We utilized either a high fidelity simulator manikin voiced by a faculty facilitator, or a trained standardized patient acting as the patient in each clinical scenario. Subsequent to the clinical scenario, several small group activities ensued with content tied to the clinical vignette used in the preceding simulation.

MS3 clerkship simulations
Mid-rotation during each participating clerkship, a specialty-specific simulation was held. In the highfidelity simulation, one faculty preceptor managed the monitor and voiced the manikin, while the other faculty preceptor acted as a nurse and subsequent debriefer.
Students engaged in each simulation in groups of three to six. Students first participated in a 30-minute simulation, followed by a 30-minute debriefing session. For the following 15 minutes, a clinical question directly applicable to the simulated case was posed to the group, as if on rounds. The students individually developed a clinical question utilizing the PICO (Patient, Intervention, Comparison, Outcomes) framework and emailed it to the preceptor [13]. The preceptor provided immediate feedback on students' responses. The students then sought an answer to their clinical question using electronic resources via their mobile devices. Feedback about optimal approaches for using evidence-based resources at the point of care was then provided to students. Following this exercise, the students were instructed to write the patient's discharge prescriptions (10 minutes). They then reviewed an answer key in real time and received feedback from the preceptor. These sessions lasted 90 minutes in total, with time for movement between stations. For a typical clerkship rotation group of 24 students, six small groups of four students required three preceptors for 3.5 hours in one-afternoon session.
Simulation cases were chosen to reflect cardinal manifestations of disease states likely to be encountered by preliminary, transitional, and categorical interns in all major specialties, such as thrombosis, bleeding, sepsis, arrhythmias, respiratory failure, encephalopathy, and acute neurologic deficit. An example simulation case is outlined in detail in Appendix A.
Group formative feedback was provided by the faculty preceptor who observed the simulation in the clinical debriefing exercise, following the structure of good debriefing techniques [14]. Subsequent EPA exercises easily led themselves to objective checklist-style evaluations, which were immediately shared with the learners. The clinical question building/information retrieval exercise assigned points based on the student's PICO and the resources used to answer the clinical question; however, the "grade" was intended for future research purposes, and only informal feedback was shared with the learners during the exercise. During the prescription writing exercise, debriefing was immediately undertaken, and corrections were shared with the group.

MS4 subinternship simulations
Building on this structure, near the end of the sub-internship, students on visiting rotations at Institution two engaged in a similar session. The clinical scenario was more advanced, with the simulated patient requiring more emergent care (i.e. unstable rapid atrial fibrillation in a patient with concomitant hypoxic respiratory failure.) Each session included a debrief session, formulating a clinical question using PICO, retrieval of evidence to answer the question, prescription writing, and a written handoff session. In the evening, students were called by a nurse to address a cross-cover issue, and provided with feedback on the way they managed the call. These sessions lasted 120 minutes, with additional time in the evening dedicated to nurse calls (approximately 5 minutes per call).

MS4 capstone course
The culmination of these activities manifested in the eight-day MS4 capstone course titled "Bootcamp." The goal of the capstone course was to build an active learning curriculum that incorporated nearly all EPAs. Table 2 demonstrates the EPAs assessed during the course. The week was organized around multiple simulation cases and was structured around small groups rotating through different sessions. Similar to the preceding MS3 and MS4 sessions, high fidelity simulations were complemented by the following exercises with a greater degree of complexity tailored to graduating fourth year students: evidence retrieval, order entering or discharge prescription writing, handoffs, and nurse calls, with the addition of informed consent encounters relevant to the day's simulation scenario. Individual and group feedback was provided in real time. The capstone course also included a procedure workshop and medical error root cause analysis module utilizing actual cases, encompassing EPAs 12 and 13.   [15,16]. Flipped classroom materials included a five-minute video modeling consent for blood transfusion and worksheets summarizing standard consent points for common procedures such as transfusion, central line placement, lumbar puncture, thoracentesis, and paracentesis. Informed consent was obtained for a blood transfusion and a bedside procedure. A unique aspect of the consent scenario was the inclusion of a "challenge statement" to allow evaluation of professionalism during these interactions. The challenge statements were based on common patient questions (i.e. "How many of these procedures have you done?" or "Doctor, have you ever seen someone have a bad reaction to a blood transfusion?" or "I cannot accept blood for religious reasons.") Students were given direct real-time feedback on their consent approach and content, as well as their professionalism in response to the challenge statements.

Outcomes
Students from ten different medical schools with a variety of educational backgrounds successfully participated in our simulation activities (Appendix B). Over 450 students have participated in the MS3 clerkship sessions at Institution one, 30 visiting students in the MS4 sub-internship sessions at Institution two, and over 300 thus far in the MS4 capstone course at Institution one. Despite different training backgrounds, all students were able to participate on a similar level, and found the activities highly beneficial. Formative assessment via direct observation was embedded at every point in the curriculum.

Student feedback
Feedback from students was overwhelmingly positive and highlighted the ability of a simulation setting to provide students with a safe environment to take a leading role in attempting to manage unstable patients, as well as simply providing an opportunity to translate what they learned in the classroom into actual practice. Quintessential student feedback below represents the students' perspective of these exercises: "The simulation session was extremely helpful and I wish that we had more sessions like this. It was useful to have to use new skills/think emergently, without putting a real human in danger. This type of experience can't really be learned from textbooks, and so the simulation session is a very good way to practice synthesizing our book knowledge with practical skills, along with practicing team work. All I can think that would improve the session is just having more of them-I found it to be very, very useful." "I thought that this was a useful, realistic exercise that allowed me to think on my feet and helped prepare me for what intern year responsibilities will entail." "I always value simulation sessions because they foster a fun and low pressure environment to learn about management. They force critical thinking in developing differentials and integrating new information as the case progresses, and allow for discussion that doesn't always happen in the time-limited clinical setting."

Next steps
We present simulation as a feasible and well-received method of providing multiple opportunities for formative assessment of core EPAs throughout the MS3 and MS4 years of undergraduate medical education. We demonstrate fulfillment of the Ottawa criteria for good formative assessment, particularly the catalytic effect to stimulate learning, as described in the student evaluation comments.
Implementation challenges include dedicating faculty time and effort, as well as simulation center availability. While high fidelity simulation with a manikin or standardized patient is ideal, low fidelity simulation (i.e. a monitor and facilitator) can be achieved on a small budget, with little detriment to the intent of the exercise. The time and resource-intensive nature of these exercises was balanced by the unique ability to deliver formative feedback based on direct observation with space and time dedicated to immediate thoughtful reflection on behalf of the students and facilitators.
Utilizing simulation as a platform allowed for direct observation of EPA behaviors and just-in-time feedback in a safe environment, mitigating risk to actual patients. Based on our experience and positive student feedback, at Institution one, we have implemented simulation even earlier in pre-clinical teaching, so simulation now takes place throughout the four years of the medical school curriculum. Consideration of a pre-subinternship bootcamp course may allow for solidification of clinical skills learned prior to real-world assessment of entrustable skills during the subinternship.

Conclusions
Student perception of learning value from the simulation-based activities was consistently and overwhelmingly positive. We found our curriculum to be generalizable, as we have demonstrated the feasibility of these exercises in medical students from multiple undergraduate medical schools. In summary, early adoption of EPA-based simulation curricula in the clinical years is feasible and generalizable, and lends itself to good formative assessment through catalytic learning. Overall Appearance:

Appendices
The patient is a robust 55-year-old male that appears uncomfortable, tachypnic, and somewhat anxious.
Actors and roles in the room at case start Standardized patient actor or manikin in the role of the patient, a faculty member, or another standardized patient in the role of the confederate RN, and learners in groups of 2-6

History of Present Illness
Mr. D is a 55-year-old male with hyperlipidemia and exercise induced asthma, otherwise in good health. Several hours prior to presentation, he awoke from sleep at 3 AM with "palpitations". He reports he felt like his heart was "pounding out of his chest". He tried to drink some water and relax. He has been very stressed at work and attributed this sensation to stress. He felt a little better and fell back asleep. In the morning, he felt the palpitations again and shortness of breath with any exertion. He was out of breath walking to the bathroom, so he presented to the ED.
If asked on review of systems, he denies chest pain. He denies syncope, diaphoresis, nausea, vomiting, or lightheadedness. He reports associated symptom of dyspnea, worse on exertion. He has never experienced any similar prior episodes. He denies lower extremity edema, orthopnea, paroxysmal nocturnal dyspnea, weight gain, increased abdominal girth, or exertional chest discomfort or dyspnea prior to this episode. He does report feeling tired as work has been stressful. Denies focal weakness or neurologic symptoms. No cough, no fevers, chills, or sick contacts.

Ideal scenario flow
Students enter and assign roles: one to take history, another to take notes, another to obtain vital signs and watch the monitor, another to examine patient. Two large bore IV access is requested. Initial history obtained and brief focused exam conducted to evaluate CV, lungs, neck veins, abdomen, and extremities, assessing for edema and pulses. Once vitals are available, learners request 12 lead EKG and interpret as atrial fibrillation, discuss management strategies and decide on rate control. Metoprolol 5 mg IV push or weight based bolus diltiazem are both acceptable choices for initial management. Once given, patient converts to sinus rhythm, but learners note that he is still quite symptomatic, and oxygen saturations are dropping. CXR and bloodwork including troponin and BNP (but not DDimer!) are requested. Patient is placed on nasal cannula, then 100% non-rebreather if necessary. Students request further history including focused review of systems and social history to rule out drug or alcohol abuse. At this time, usually discover that he has recently traveled and has a prior history of PICC-associated DVT. Chest X-ray is interpreted as clear. Students may request bedside point-of-care ultrasound at this time and note enlarged, hypokinetic RV and bilateral A-lines on lung exam. If not, students request CTA and discover bilateral proximal PE. Students initiate anticoagulation (rationale exists for Lovenox or heparin infusion as initial management, as this is classified as submassive PE). The scenario ends with students presenting the patient to their attending on the phone or in person.

Variant for more advanced learners (MS4)
A variant of this case for more advanced learners involves the patient presenting with hypotension in the setting of rapid atrial fibrillation. The students at this point must consider and prepare for cardioversion, although the patient does not always require cardioversion as vitals will stabilize with a small fluid bolus or small dose of beta blocker or amiodarone. The remainder of the case ensues as above but patient remains with borderline hypotension, prompting a discussion of potential thrombolysis. The subsequent debrief focuses more on the evidence behind management of submassive and massive PE rather than basic management of critically ill patient.

Additional Information Disclosures
Human subjects: All authors have confirmed that this study did not involve human participants or tissue. Animal subjects: All authors have confirmed that this study did not involve animal subjects or tissue.

Conflicts of interest:
In compliance with the ICMJE uniform disclosure form, all authors declare the following: Payment/services info: All authors have declared that no financial support was received from any organization for the submitted work. Financial relationships: All authors have declared that they have no financial relationships at present or within the previous three years with any organizations that might have an interest in the submitted work. Other relationships: All authors have declared that there are no other relationships or activities that could appear to have influenced the submitted work.