Virtual Reality vs. High-Fidelity Mannequin-Based Simulation: A Pilot Randomized Trial Evaluating Learner Performance

Background Virtual Reality (VR) simulation has been found to be useful in learning technical and non-technical skills. However, empirical data about its efficacy in clinical education are limited. This pilot study compares the efficacy of VR to mannequin-based simulation for learners managing status epilepticus (SE). Methods Pediatric and emergency medicine interns at an academic tertiary care referral center were randomized to either VR (intervention, using an Oculus RiftⓇ (Occulus from Facebook, Facebook Inc., USA)) or mannequin-based (control) simulation for the same SE scenario. The control group participated in two mannequin-based simulation sessions while the intervention group had a VR session followed by a mannequin-based session. Sessions were one-one with an instructor and held three months apart. Performance was assessed by measuring the time-to-critical actions during the second session. Results Of 42 interns, 22 were in the intervention group and 20 in the control group. There was no statistical difference in time-to-critical actions for VR vs. standard groups; for example, VR times (in seconds) compared to standard times were 18.1 (SD 10.5) and 18.9 (SD 15.8) (p=.90) for oral suction, and 61.6 (SD 24.8) and 62.8 (SD 26.9) (p=.82) for IV lorazepam completion. Conclusion This pilot trial suggests that VR is feasible for clinical simulation. We did not find a significant difference between the two groups in learner performance. Larger studies are needed to corroborate our findings, investigate the best applications of VR in clinical training, and determine if it could lead to more rapid learning at a lower cost.


Introduction
Simulation is a popular instructional strategy that has been used in many fields including medical training [1] and applies the concepts of experiential learning [2]. It provides a safe and forgiving environment to practice [1,3,4], which became increasingly important with the restriction of duty hours and growing focus on patient safety [3,5]. This is especially true when it comes to rare but life-threatening conditions [4][5][6]. There are different modalities for simulation that are used in different settings and for different goals [4,7]. These modalities include task trainers, low and high-fidelity mannequins; however, all of these can be expensive to purchase and require significant resources to operate [2]. It can be used to improve technical and non-technical skills including team performance, communication, and efficiency [2]. There is a growing body of evidence proving its efficacy in medical education [4,8,9]. Studies have shown that simulation increases provider comfort and confidence managing critical situations or performing individual skills [10][11][12][13][14][15][16].
Virtual simulation is a newer method that has been gaining popularity in the last decade [17]. Virtual reality (VR) simulation can be designed with a high degree of realism and audiovisual stimulation that can completely seclude participants from their environment allowing for complete engagement [17] and even induce a stress response that can resemble, to some extent, a real-life encounter [18]. VR simulation can improve technical skills [5,19,20] and non-technical skills including teamwork, decision making, and situational awareness [5,17]. VR simulation carries the advantage of being less expensive than mannequinbased simulation, requires fewer resources to implement simulation scenarios, and simulated encounters can be carried out in different settings [5,21,22]. Kayw et al. reported improved post-intervention knowledge and cognitive skills in VR compared to traditional learning methods (lectures, textbooks, or digital sources) [23]. Data comparing the efficacy of VR simulation to mannequin-based simulation when it comes to performance and retention of information is scarce. In a simulated pediatric status epilepticus case, we assessed the feasibility of VR simulation as an effective learning modality and the performance of trainees (and their confidence) when randomly assigned to VR simulation vs a high-fidelity mannequin-based simulation control group.

Study design
This was a randomized control trial at an academic tertiary care referral center that receives trainees from different local institutes and programs. It is a level one pediatric trauma and burn care center. Participants included incoming pediatric and emergency medicine interns (academic year 2019-2020) who enrolled voluntarily. All incoming interns were offered the opportunity to participate. All participants had a similar level of experience and had recently graduated medical school. There were no exclusion criteria. Participants were randomly assigned to either a VR simulation group (intervention) or a high-fidelity mannequin-based simulation group (control) with a 1:1 ratio. Time slots were planned to alternate between VR simulation and mannequin-based simulation. Participants who were blinded to the type of simulation planned then selected the time slot that fits their schedule. All participants were proctored by the same instructor (MA) who was a senior pediatric emergency medicine fellow. The first session outlined the objectives of the study, allowed time for orientation, simulation of two cases, an anaphylaxis and status epilepticus followed by debriefing and a questionnaire. The cases were developed to be exactly the same in either VR or mannequin-based simulation. The second session was planned three to four months after the first. This was based on the data reported by Wik et al. that suggested different levels of skill retention at six months depending on the method of training [24]. All participants in the second session completed a case of status epilepticus on a low-fidelity mannequin. The primary outcome was to test trainee performance. It was assessed by extracting the time (in seconds) it took to perform critical actions from the video records of the second session, and a comparison was made between the intervention and control groups. The study was exempt by the institutional review board (IRB00194558). Written consent for participation and video recording was obtained. The manuscript writing process followed the consolidated standards of reporting trials (CONSORT) guidelines [25].

First (initial) Session
Participants in the intervention group started their session completing a survey assessing demographics, background in gaming, and subjective confidence in managing status epilepticus. The instructor reviewed the objectives with the participants. These included a stepwise approach to patient evaluation and timely management of recognized issues. Participants were assigned the team leader role whose responsibility is to evaluate the patient and choose the course of action while the nurse and respiratory technician avatars in the VR environment performed actions. The instructor showed screenshots of the VR environment to give a sense of what it looks like and introduced the VR headset and controllers to the participants. Participants were oriented to the VR functions in less than five minutes. This time was based on our experience using the modules/equipment during our regular training sessions. Participants played an orientation module that aimed at explaining how to navigate the simulation cases and perform various actions like examining the patient, choosing airway support equipment, or selecting medications. This module ran between three to five minutes. After orientation, the anaphylaxis case was played, which helped participants get a clear understanding of their role and how to use the various VR functions. After a two-minute break, the participants proceeded to complete the status epilepticus scenario. The debriefing was aimed at exploring participants' feelings and reviewing management guidelines for each case. A post-participation survey was done to assess participant's satisfaction using VR, cybersickness symptoms, and post-simulation subjective confidence.
The control group sessions were intentionally planned to follow the same structure as the intervention group and allowed the same amount of time for orientation. In these sessions, the instructor performed the actions of the nurse and respiratory technician. Each participant was allowed to explore the room, learn about the various functions of the mannequin, and check equipment availability in the airway, peripheral access, and medication carts. There was no orientation module for this group and the anaphylaxis case was started after participants felt comfortable in the environment. This was followed by the status epilepticus case and debriefing. Participants completed the subjective confidence survey at the end.

Second (assessment) session
Both groups went through a mannequin-based simulation case of status epilepticus as individuals with the instructor. Sessions were video recorded. These records were reviewed to assess the time (in seconds) taken to perform critical actions. Participants were asked to assess their subjective confidence again before and after this session.

Control group
The simulation was conducted in the pediatric emergency trauma bay. The equipment was intentionally laid out to look as similar as possible to the VR environment. The mannequin was placed on a stretcher and connected to a monitor that was on the patient's left side along with a list of the medications available to use, which was printed and placed on top of the medication cart on the same side. Airway equipment was placed at the head of the bed. These include a non-rebreather mask, face mask, nasal cannula, bag valve mask, endotracheal tube, laryngoscope, oral and nasal airway adjuncts, and a suction catheter. On the patient's right side, a peripheral intravenous catheter start kit, blood test vials, and a fluid bag were present. A copy of the vital signs was printed to be used in case of technical issues during the session. The same setting was used for the follow-up sessions for both groups (Figure 1 (A)).

Intervention group
The same trauma bay room was used with the same equipment present; however, the headset and controllers were set on a table on the side. The virtual environment was initially designed to copy the critical care room at the designer's institution. The participant was placed at the bottom of the stretcher. The vital sign monitor and medication cart were on the patient's left. The airway cart and respiratory technician avatar were at the head of the bed. The nurse and peripheral venous access tray were on the patient's right. The instructor was able to observe the participant's performance on the computer screen ( Figure 1 (B)).

Cases
Two cases were developed for the VR (AiSolve Ltd, UK; BioflightVR, USA). The first is a seven-year-old who experiences anaphylaxis with airway involvement. This case would require medical and airway management. The second is a one-year-old in status epilepticus who requires airway management and escalation of antiepileptic medications. These cases were built under the supervision of two pediatric emergency medicine experts to resemble critical patients in the pediatric emergency room in prior studies [18]. Cases were exactly the same in both the intervention and the control groups. An orientation module was prepared to help users explore the environment and the various functions available.

Debriefing
Debriefing was performed after all sessions following the Promoting Excellence And Reflective Learning in Simulation (PEARLS) framework by Eppich et al. [26]. After exploring the participant's feelings and concerns, the discussion focused on a stepwise approach for patient's evaluation and the steps required for stabilization of the airway, breathing, and circulation with emphasis on addressing any issues before moving on to the next step: for example, evaluating and supporting the airway before requesting peripheral vascular access. This was followed by a detailed review of the management guidelines for anaphylaxis and status epilepticus.

Data collection and analysis
Time (in seconds) to completion of certain critical actions was measured to assess differences between the two groups. These actions were chosen based on the major points focused on during the debriefing of the initial session. These include the rapid assessment and management of acutely ill patients according to the pediatric advanced life support (PALS) guidelines following the ABCs and the concomitant medical management of seizures (airway assessment and intervention, e.g., suction, oxygen via non-rebreather or other delivery devices, establishing intravenous access, ordering an antiepileptic medication, and specifically ordering IV lorazepam). The time was assessed by reviewing the video records of the second session. Since one of the goals was to help the participants develop a stepwise approach, the instructor redirected participants to the PALS algorithm if an issue was not addressed promptly. For example, establishing peripheral vascular access would not be accomplished if the airway was not stabilized.
The initial survey included demographics (gender, residency program, video gaming background, and VR experience) and subjective confidence on managing a patient in status epilepticus. The intervention group completed a section on cybersickness symptom checks, including fatigue, discomfort, sweating, dizziness, headache, eye strain, blurry vision, nausea, and upset stomach. All participants completed a survey on subjective confidence after the initial session. The intervention group completed an extra section on cybersickness symptoms and system usability scale using (which was modified from the work of Bangor et al. 2009 [27]).
Confounding factors that could affect the management skills, including a background in neurology and exposure to seizure cases during the period between sessions, were assessed prior to the second session in addition to the reported subjective confidence before and after the session.

Statistical analysis
Descriptive analyses were performed to compare demographics, clinical, gaming experience, and user satisfaction. Time-to-critical actions and subjective confidence were reported as means and standard deviations. Time-to-critical actions were log-transformed before t-tests were applied to test between intervention and control groups. All data were analyzed using Stata (Strata Statistical Software: Release 13, StataCorp, USA)

Demographics
Of the 44 interns, 42 participated in the first session while two were not able to schedule. Of these, 31% (13/42) were from emergency medicine and 69% (29/42) from pediatrics. Only one had previous experience with VR gaming. During the second session, 76.2% (32/42) of the total participants were able to participate. The remaining 10 were unable to participate due to time constraints (Figure 2). Among the participants in the second session, 15.6% (5/32) had previous EMS experience, 3.1% (1/32) worked as a nurse, and one had a background in neurology (

Time-to-critical actions
Time to clearing the airway secretions with suction was 18.1 seconds (SD 10.5) and 18.9 seconds (SD 15.8) for the intervention group and control group respectively (adjusted p-value 0.9), and administering oxygen with a non-rebreather mask was ordered at 34.1 seconds (SD 17.9), and 34.1 seconds (SD 13.9) for the intervention group and control group respectively (adjusted p-value 0.75) ( Table 2). When the entire group was pooled together, the time to clearing the airway with suction was 18.5 seconds (SD 13.2), and oxygen via nonrebreather was at 34.1 seconds (SD 15.8). (Figure 3).

Subjective confidence
The mean subjective confidence was increased by two points for the intervention group and 1.8 points for the control group ( Table 3).
I feel confident recognizing and providing initial management to patients in status epilepticus 1-5

Cybersickness and system user satisfaction
The VR group included 22 participants. No one reported any symptoms before the VR session. Only one (4.5%) reported mild discomfort and dizziness after completing the VR session, which reportedly resolved without any intervention by the end of the debriefing. Overall, participants rated the user-friendliness of the product as good with a mean of 5.4 out of seven (SD 0.8) ( Table 4).

Discussion
This is a pilot study investigating the efficacy of VR simulation in medical training by comparing it to highfidelity mannequin-based simulation. Furthermore, it looked into how the use of VR was perceived and assessed the incidence of cybersickness. Our data show that VR simulation has comparable efficacy as highfidelity mannequin-based simulation. Both groups in our study had similar performance results and there was no statistical difference in the time required to perform critical actions between them. Additionally, participants in the VR group reported high satisfaction rates when using the VR system and equipment. Their incidence of cybersickness was very low.
The cases used in the study aimed to encourage participants to approach critical patients systematically.
When the time-to-critical action was assessed, participants in both groups had similar results. This shows that VR simulation can at least produce similar effects on learning.
When compared to mannequin-based simulation, VR is relatively less expensive, requires less human resources to perform, and can be carried out in any setting [5,17,21,22]. We were able to perform VR simulations at any time and location based on the resident's availability. Mannequin-based simulations had to be carefully planned and there was less flexibility when performed. Furthermore, we had a high overall satisfaction score for VR showing it was easy to learn and use, which in turn could increase the intention to use it more if available.
There is reported data about the risk of cybersickness with using VR technology [28]. Our study shows a very low risk of developing such symptoms when using modules built for medical training. Only one person reported mild discomfort and dizziness after finishing the modules. These symptoms were reportedly resolved soon after during the debriefing process. Multiple factors play a role in the risk of cybersickness, including personal and technology-related factors. These could be taken into consideration when planning to decrease the risks.
Both VR and mannequin-based simulation technologies have different characteristics and advantages. As they continue to evolve, we think VR should be used as a method to complement mannequin-based simulation. Future studies can explore the efficacy of VR simulation using robust and standardized testing and assessment methods. More studies should attempt to characterize the best uses for VR simulation and how it can be best incorporated in medical training. This includes the type of cases used and the goals that need to be achieved from simulation. The effects of independent use of VR simulation by learners without supervision is an area that needs exploration. Furthermore, the effects of using both VR and mannequinbased simulation in a longitudinal curriculum should be investigated. Debriefing is an important part of simulation training [29,30]; VR simulation allows for various debriefing methods. We performed in-person debriefing one-on-one with the instructor. Participants also received computer-generated performance feedback. Exploring the appropriate debriefing methods that could be used with VR technology and the usefulness of the computer-generated feedback should also be considered.
Our main limitation was the sample size. This was a convenience sample of incoming trainees at a single institution using specific cases. More robust studies evaluating VR simulation are needed with larger sample size and different scenarios. Attempting to prove equivalency using a non-inferior methodology would require a significantly larger sample size. Our sample size allowed us to explore if a difference may be present. Larger studies will be needed to provide evidence. Our learners came from different backgrounds. Given the randomization of participants into both groups, it was assumed that both groups are equivalent, and no pre-assessment tests were utilized. Post-assessment tests were not utilized as well given that all participants were involved in a standardized one-on-one debriefing covering all learning points. Some have seen and actively managed patients with seizures before or during the period of the study. This may have affected their performance in the assessment session. Debriefing was performed on a one-on-one basis. Although all the focus was on one subject, the absence of a team nature and support could be anxietyprovoking to learners.

Conclusions
Virtual reality is a feasible method for clinical simulation and training. Although we did not find a significant difference in learning between VR and mannequin-based simulation, future studies should further evaluate the learning outcomes, explore the utility of VR simulation, its application in clinical training, and its effects on learning.
grant from Oculus from Facebook. Todd completed grant funding from Oculus from FaceBook for the development and pilot evaluation of the VR scenarios used in this manuscript; he did not participate in recruitment nor have access to raw data for this study. Other relationships: All authors have declared that there are no other relationships or activities that could appear to have influenced the submitted work.