Defining Priority Areas for Critical Care Simulation: A Modified Delphi Consensus Project

Background Simulation is used in critical care for skill development, formative assessment, and interprofessional team performance. Healthcare educators need to balance the relatively high cost to deliver simulation education with the potential impact on healthcare quality. It is unclear how to prioritize simulation in critical care education, especially considering interprofessional needs across adult and pediatric populations. The objective of this study was to prioritize topics for critical care educators developing simulation-based educational interventions. Methodology A modified Delphi process was used to identify and prioritize critical care topics taught using simulation. We disseminated a multi-institutional survey to understand critical care simulation topics using a three-round modified Delphi technique. An expert panel was recruited based on their expertise with simulation-based education through the Society for Simulation in Healthcare and the Society of Critical Care Medicine lists. Critical care topics originated using content derived from multiple critical care board examination contents. Additional content for a critical care simulation-based curriculum was generated. Results Consensus and prioritization were achieved in three rounds, with 52 simulation experts participating. The first Delphi round surveyed priority topics in critical care content and generated additional topics for inclusion in round two. The second Delphi round added the content with the highest-ranked items from round one to generate a set of simulation-based topic priorities. The third Delphi round asked participants to determine the importance of each priority item taught via simulation compared to other modalities for clinical education. This round yielded 106 topics over four domains categorized into (1) Diagnosis and Management of Clinical Problems, (2) Procedural Skills, (3) Teamwork and Communication Skills, and (4) General Knowledge and Knowledge of Technical Adjuncts. Conclusions The modified Delphi survey revealed a prioritized, consensus-based list of topics and domains for critical care educators to focus on when creating a simulation-based critical care curriculum. Future work will focus on developing specific simulation-based critical care curricula.


Introduction
Simulation-based medical education (SBME) is widely accepted as an effective method to teach interprofessional and interdisciplinary team communication skills, technical skills, high-stakes rare events, and integrative case-based demonstration of higher-level skills [1]. Healthcare professionals across the continuum need mastery of these skills and core competencies from undergraduate medical education (UME) to postgraduate training in multiple healthcare disciplines and specialties. In critical care, simulation has historically focused primarily on resuscitation and procedural skills.
Evidence supports using SBME to teach numerous domains of critical care, spanning across multiple medical specialties and healthcare professions [2][3][4]. Educators and healthcare systems must balance the financial and personnel costs of SBME with the effectiveness of reaching desired learning and healthcare outcomes [5]. Although SBME is resource-intensive, its integration in curricula has dramatically expanded its reach, demonstrating improvements to patient-driven outcomes [6][7][8]. These include just-in-time infant lumbar puncture, simulation-based mastery learning of central venous line (CVL) placement, and using virtual reality to teach patient assessments [9][10][11][12][13].
Despite the evidence of its importance, there is no current widespread standardized curriculum or guidance on critical care simulation content for educators. Although several disciplines have developed guidelines for teaching critical care topics, none have exclusively prioritized the role of a simulation-based curriculum for this level of intensive care training [14,15]. Therefore, it remains unclear how to prioritize topics best suited to be taught via simulation for critical care healthcare professionals. Thus, we aimed to assess the current status of SMBE in critical care and develop priority areas for the effective and efficient utilization of simulation for critical care education.

Design and Delphi process
A modified Delphi was used to develop consensus around SBME priorities and present a list of content to guide future simulation-based educational priorities such as generating a standard critical care simulation curriculum [14][15][16]. The Delphi method relies on expert opinion to arrive at these conclusions. A six-step approach is adopted when utilizing this process: (1) stating the problem, (2) developing an initial questionnaire, (3) selecting experts, (4) iteratively polling experts, (5) identifying potential consensus points, and (6) reporting results. This approach allows for an iterative understanding of the question with feedback given by experts between cycles [16][17][18] (Figure 1).

Identification of simulation education experts for Delphi panel
The research team was composed of interprofessional members of the Society of Critical Care Medicine (SCCM) Research Section, Simulation Education Research Subcommittee. Simulation education experts were identified through the purposeful sampling of publicly available lists of simulation centers to identify experts in critical care simulation education for this panel. Representatives of accredited simulation centers available through the American Board of Anesthesiology and Society for Simulation in Healthcare were approached, along with members of the SCCM Education Research Committee. We defined "critical care simulation educator" as someone who uses any modality of technology-enhanced simulation to educate learners, utilizing evidence-based strategies directed at critical care practice objectives [19]. We established a panel of 52 experts from multiple professions consistent with the standard Delphi procedure. A panel of interprofessional content experts included physicians, nurses, educators, and pharmacists. The snowball method was used to sample participants, where each participant was asked in the initial survey to identify additional expert participants who then also received an invitation to participate in the study [20]. Experts were sent an email invitation explaining the study's purpose and a link to the survey for the first round of the Delphi. Participation was voluntary, and this study was approved by the University of Pittsburgh Institutional Review Board as a non-human subjects exempt survey research study. The participants remained anonymous to each other throughout the study.
Study data were collected and managed using REDCap electronic data capture tools co-hosted at the University of Pittsburgh and Stony Brook University. Surveys were distributed between December 2018 and June 2019. Responses were collected and de-identified for final data analysis. No limits on respondents were set a priori. Once panelists agreed to participate, a link to the first round of the survey was sent, along with detailed instructions on how to complete the Delphi surveys.
The writing group consisted of 12 members, a diverse interprofessional and interdisciplinary group of simulation educators and facilitators from neurocritical care (2), pediatric critical care (5), emergency medicine-critical care (1), pediatric emergency medicine (1), critical care nurse practitioners (1), and critical care pharmacist (1).

Survey design and iterative feedback
Initial first-round items were derived from blueprint content of critical care board examinations from the following specialties: the American Board of Internal Medicine, American Board of Surgery, American Board of Pediatrics, American Board of Emergency Medicine, and American Board of Anesthesiology. Descriptions of training program requirements in critical care medicine were reviewed to determine a list of topics for ranking in the survey. As all study team members were critical care simulation educators, a pilot test of the survey for readability, feasibility, and face validity was performed, and changes were made as needed.
Experts were asked to prioritize each topic on a three-point Likert scale (least important, somewhat important, most important). Free text options to describe the current simulation curriculum and its associated objectives were included. At the end of the first round, unique free-text items were analyzed and included in the second round as potential additions to curricular content.
Consensus was defined a priori as >80% agreement with "somewhat" or "most" important selected. Topics with a consensus of <80% were not included in the priority list for round two as they were thought to be unlikely to reach a consensus. Round two items were aggregated from round one, and potential topics for inclusion in a critical care simulation content outline were included in the final round of content generation. At the end of the second round, we selected the items from the list of consensus curriculum that achieved this threshold [16][17][18]21]. Following this round, we formulated potential consensus items for inclusion into the third round. This method was used, in part, to assure a more comprehensive capture of content.
For round three, items were again aggregated and then prioritized into three domains of content that were most important, somewhat important, or least important to include as SBME content. For each ranking, we asked participants to rate each topic using a three-point Likert scale (with options labeled 1 = best taught via simulation to 3 = should not be taught via simulation).
To obtain a prioritized list of critical care content, we used the methods proposed by Altschuld and Thomas, where content is ranked and scored according to the frequency of occurrence for prioritization scoring [22]. According to this method, minimum priority was defined as a prioritization score of >50, with strong prioritization >60. Those topics not included in the priority list scored <50 were classified as low priority and not included.

Outcome measures
The primary outcomes of interest were ranked in order of importance of areas of critical care content. We defined "simulation education" as any modality of technology-enhanced (computer, manikin, standardized patient, excluding animal, tissue, or cadaveric models) simulation, which is an educational tool with which the learner interacts to mimic an aspect of clinical care for the purpose of teaching or assessment in the delivery of critical care to adult or pediatric patients [19]. The primary outcome of the third round Delphi was to generate a list of prioritized content deemed important by this multi-professional group of critical care simulation experts.
Consensus was reached after round three, from which we developed a critical care simulation content outline. All recommended topics were mapped into four main learning domains: General Knowledge, Diagnosis and Management, Procedural Skills, and Teamwork and Communication; consensus was reported among all simulation specialists ( Figure 2).

Results
A total of 52 experts participated in the initial survey, 27 of whom completed all three rounds (52%). The majority of participants worked in critical care. There were 34 (65%) respondents who were physicians, followed by nurse practitioners (n = 6, 12%), nurses (n = 5, 10%), pharmacists (n = 4, 8%), respiratory therapists (n = 2, 4%), and other (n = 1, 2%). Experts most commonly represented pediatrics (n = 14, 27%) and emergency medicine (n = 9, 17%). These experts reported working in 16 states across all geographical regions of the United States. Additional demographic data of the simulation experts are presented in Table 1      Topics with moderate priority are listed in Table 6.

Topic Score Overall rank order
Deep knowledge of critical illness (beyond algorithms) 57 23 Overdoses and poisonings: medication indications and side effects (of drug(s) taken) 57 23 Anesthesia medication indications and effects (e.g., analgesic, sedatives, neuromuscular blockade) 52 27

Discussion
We developed simulation-based critical care education priorities through the consensus of simulation experts across all pediatric and adult critical care facets. In defining these domains and categories of core simulation-based critical care content, we hope to improve the prioritization of SBME within critical care. Identifying priority areas for simulation-based education may help stakeholders, including healthcare organizations, professional societies, and educational leadership, prioritize funding, budget allocations, faculty/staff support, research collaboratives, and program assessments. This content should aid training programs and simulation centers in evaluating learners and target learning needs specific to critical care medicine.
Prior work using the Delphi process for content generation has been used to generate UME content for critical care [15] and pediatric content for emergency medicine residents [14]. In both cases, the scope was different and not adequately specific to apply to critical care core competencies in the context of either graduate medical education or interprofessional education, again highlighting a need for this study. Not surprisingly, our results demonstrate strong agreement that simulation is well suited to teach several aspects of communication and procedural skills. A focus on aspects of cardiac arrest, shock, and other resuscitation scenarios was also noted to be important components of SBME in intensive care units. This study also demonstrates a potential role for SBME to teach more cognitive skills such as diagnosis and management of various clinical entities.
A key finding of our study was a generally good agreement on the topics for critical care education across teamwork, communication, and procedural skills. While pathophysiology and diagnosis may differ across the age spectrum and between different medical and surgical subspecialties, often the basics of procedures and especially the skills necessary for effective communication during resuscitation events transcend specialties and professions. This suggests that combining interprofessional critical care simulation programs across subspecialties may benefit from optimizing resource utilization. We hope that this will ultimately facilitate a shift from training in silos to training together through interprofessional simulations. In doing so, we might better approximate how we might create better learning activities for critical care teams.
Despite an underrepresentation of neurointensivist participation, neurological emergencies were identified as an area of high priority, potentially reflecting emerging awareness of the utility of SBME for teaching about neurological emergencies and technological advances that support creating such simulations.
Intensivists, including pediatric, emergency medicine, and trauma critical care providers, have essential roles in the initial identification and management of acute neurological injury and thus may have ranked these topics higher.
One of the main challenges of the widespread adoption of simulation education is insufficient faculty time and the limited ability to regularly pull trainees and interprofessional colleagues from clinical care responsibilities. Especially during the global coronavirus disease 2019 (COVID-19) pandemic, any disruption to the available pool of faculty and trainees available to provide bedside care is unlikely to be supported and may result in the cancellation of simulation courses altogether. By co-training learners from different specialties, no individual department is left with inadequate patient care coverage. Additionally, if course goals and objectives are uniform, faculty across different specialties could share the responsibility of teaching sessions. Finally, expanding the pool of potential learners and research subjects can further facilitate the performance of simulation education research. Additionally, the COVID-19 pandemic has shown the importance of cross-training healthcare providers who are not typically critical care providers for such surges. This priority list helps healthcare systems and simulation centers determine key objectives for rapid just-in-time training.
Our work has several significant limitations, including a variable survey response rate and drop-off between rounds resulting in a small sample size of predominantly physicians and a higher ratio of pediatric-to-adult intensivists. Additionally, it is not possible to extrapolate if the opinions expressed by educators truly reflect learners' needs. Third, our results may have been different had this study been performed after the onset of the COVID-19 pandemic.
The Delphi method has inherent limitations within the process itself. The intention is to gather and obtain consensus based upon experts' opinions and insight. It does not result in an evidence-based summary (a Delphi is performed precisely because of a lack of evidence within a specific area). Additionally, the final product is a list of statements that gained consensus but does not provide additional detail or depth to those answers. This study's specific aims focused on overall expert opinion for which consensus was sought over a wide range of content, spanning multiple disciplines, several specialties, and age ranges, thus limiting the ability to obtain in-depth input for any specific subspecialty or pathology. However, the broad net that was cast in this project may also be seen as a strength as it increases the generalizability of the findings. Given the scope of the study, some of the topics listed by subspecialty boards were more detailed than others, leading to a wide range in the specificity of the topics included.
The Delphi survey's extensive and iterative nature could have led to potential survey fatigue and content bias. The composition of our expert panel may have been biased as well. Although we tried to mitigate this by broadly recruiting experts in the field through multiple lists, the final panel was somewhat self-selecting by answering our recruitment emails. Given that this Delphi process was conducted remotely, as opposed to in-person, bias from a smaller group of "influential" experts was less likely to have occurred. However, using a remote method without an in-person component may have also contributed to the relatively high attrition rate experienced. While we had a reasonably sized group, it is unclear from existing literature what the ideal size of an expert panel should be. A group should generally be large enough to propose new ideas and overcome attrition issues. A total of 30 members have been suggested as a reasonable target to attain these goals and align with our final survey round [23]. Finally, while we sought to have a broad representation of experts across disciplines within the United States, these results lack international representation and likely differ in other countries, where variations in educational curricula, competing interests, and different priorities may be present.

Conclusions
In conclusion, we defined priority areas in SBME curricula for critical care using a modified Delphi process.
Healthcare systems and simulation centers should focus on these criteria when designing training curricula for critical care teams across medical professions and subspecialties and consider sharing resources between different groups of faculty and learners. As critical care practice changes with new medications, technology, and multispecialty teams, judicious use of simulation will help ensure timely and effective educational interventions that will also inform quality improvement and safety initiatives.

Additional Information Disclosures
Human subjects: Consent was obtained or waived by all participants in this study. Animal subjects: All authors have confirmed that this study did not involve animal subjects or tissue. Conflicts of interest: In compliance with the ICMJE uniform disclosure form, all authors declare the following: Payment/services info: All authors have declared that no financial support was received from any organization for the submitted work. Financial relationships: All authors have declared that they have no financial relationships at present or within the previous three years with any organizations that might have an interest in the submitted work. Other relationships: All authors have declared that there are no other relationships or activities that could appear to have influenced the submitted work.