The Comparative Effectiveness of Virtual Reality Versus E-Module on the Training of Donning and Doffing Personal Protective Equipment: A Randomized, Simulation-Based Educational Study

Introduction Preventing errors in donning and doffing of personal protective equipment (PPE) is critical for limiting the spread of infectious diseases. Virtual reality (VR) has demonstrated itself as an effective tool for asynchronous learning, but its use in PPE training has not been tested. The objective of this study was to compare donning and doffing performance between VR and e-module PPE training. Methods A prospective randomized open-blinded controlled trial was conducted to determine differences in donning and doffing performance after VR and e-module PPE training among medical staff and medical students at a single institution. The primary outcome was donning and doffing performance with real PPE, assessed using a 64-point checklist. The secondary outcome was participant preparedness and confidence level after training. Results Fifty-four participants were randomized, mostly consisting of medical students (n=24 {44%}) or emergency medicine and otolaryngology residents (n=19 {35%}). The VR group (n=27 {50%}) performed better than the control in the overall PPE scores but this was not statistically significant (mean {SD}, VR: 55.4 {4.4} vs e-module: 53.3 {8.1}; p = 0.40). VR participants also reported higher levels of preparedness and confidence after training. Residents as a subgroup achieved the highest increases after VR training compared to their counterparts in the control training group (mean {SD}, VR: 55.6 {4.9} vs e-module 48.4 {5.5}, p = 0.009). Conclusion In this randomized trial, VR training was found to be non-inferior to e-module for asynchronous PPE training. Our results suggest that in particular residents may benefit most from VR PPE training. Additionally, VR participants felt more confident and prepared to don and doff PPE after training compared to e-module participants. These findings are particularly relevant given the ongoing coronavirus disease 2019 (COVID-19) pandemic. Future studies need to focus on VR integration into residency curriculum and monitoring for long-term skill retention.


Introduction
The coronavirus disease 2019 (COVID-19) pandemic highlighted concerns regarding personal protective equipment (PPE) utilization in hospitals [1,2]. When used correctly, PPE can minimize transmission to COVID-19 [3,4]. However, evidence suggests that up to 90% of PPE procedures like doffing are performed incorrectly [5]. This failure rate has been linked to healthcare workers being more likely to contract COVID-19 compared to the general community, resulting in authorities turning to increased training as a potential solution [6,7]. standard for doing so [8]. Training methods vary, with the conventional approaches being in-person or video presentations [8]. In-person, immersive training with active involvement and feedback tends to be preferred; however, staff shortages, PPE shortages, and social distancing guidelines limit feasibility [9,10]. Online modules and videos are asynchronous methods that are also commonly utilized, but pose problems including lack of student engagement, reduced accountability, and the limitations of teaching hands-on skills online [11].
Virtual reality (VR) is a potential alternative, offering similar benefits to in-person training, such as immersion and feedback, while minimizing barriers related to timing, social distancing, and equipment shortages [12]. VR allows for repetitive practice on an as-needed basis while preserving PPE for clinical interactions. Further, VR headsets continue to grow in availability and affordability. These qualities make VR a viable alternative, although its impact on donning and doffing quality is unknown. Studies regarding PPE training have found in-person and video methods to be comparable [11,13], and computer simulations effectively complement in-person training [14]. However, to our knowledge, there has been no investigation of the use of VR for PPE training.
This study conducted a randomized clinical trial at a tertiary care academic center to evaluate the effectiveness of VR training compared to a control group consisting of electronic-module (e-module) training as it relates to donning and doffing competency. The primary objective was to compare the performance of donning and doffing PPE after VR or control. Secondary objectives included identifying subgroups more likely to benefit from the training and evaluating participants' perceptions of using of VR for PPE training. It was hypothesized that the use of VR training will be superior to the e-module as an asynchronous modality to teach PPE procedures.

Materials And Methods
This study was a parallel, 1:1 randomized trial on a convenience sample conducted at an urban academic medical center with a prospective open blinded endpoint (PROBE) design [15]. The Consolidated Standards of Reporting Trials (CONSORT) were followed [16]. The study was approved by the institutional review board (IRB); verbal and signed consent was obtained from all participants.

Participants
Members of the institution were included. Participants were recruited using email listservs, and the study was primarily conducted at the hospital's simulation center. To meet the needs of participants, we expanded to multiple locations, including residency conferences for emergency medicine and otolaryngology. These residencies were chosen due to invitation from residency directors and their increased risk for infectious exposure during the pandemic.

Randomization
Upon enrollment, participants were assigned an identification study number and were block randomized using a concealed computer-generated random allocation sequence. Randomization was performed on-site in real-time. Participants were separated into control (e-module) or intervention (VR) groups ( Figure 1). After providing consent, participants completed a short anonymous survey regarding demographics, PPE training, and VR experience.

Control
The control group received online PPE training that was completed at the study center. Training included a five-minute instructional video and a PowerPoint presentation. The video included step-by-step instructions and a demonstration of donning and doffing. The presentation contained the same content as the home institution's PPE training, based on Centers for Disease Control and Prevention (CDC) guidelines [7]. Participants were instructed to review the material however they liked, and the duration of their training was recorded. A study member was also present for any assistance that was needed with the PPE training emodule.

Intervention
The intervention group received VR-based PPE training on the Oculus Quest (Menlo Park, CA: Facebook, Inc.) using a PPE training program created by Axonpark, Inc. (Fort Lauderdale, FL). The training included the following: (1) a tutorial of the donning and doffing sequence, based on CDC guidelines [7]; (2) a training mode to practice with stepwise feedback; and (3) a testing mode that repeated until the sequence was completed perfectly (Figure 2, Video 1). A study member was present to assist with the device and record the duration of training. No affiliate of Axonpark, Inc. was present during the course of the study.  Donning and doffing performance was assessed using videos of participants with real PPE. After completing training, participants were brought to a private, recorded room with the necessary PPE and instructed to don and doff. Finally, participants completed a survey regarding reactions to the program, physical complaints, and feedback.

Outcomes
The primary outcome measure was donning and doffing performance -each measured separately -which was evaluated using a checklist based on CDC guidelines (

Raters
Due to the nature of the intervention, participants were not blinded, but outcome raters were. The outcome raters were two independent nurse educators, blinded to the study protocol and interventions. Raters were calibrated using a standardized scoring checklist and assessed through ratings of mock videos. To assess the PPE performance of participants, raters were randomly assigned 35 videos each, with a 16-video overlap to calculate reliability. Recordings were stored in an encrypted and secure folder.

Statistical analysis
All statistical analyses were performed in R version 3.6.3 (Boston, MA). Following data collection, comparisons between study groups were examined at an alpha level = 0.05. The main outcome was the continuous outcome of "overall score" -the summation of donning and doffing scores. A two-way intraclass correlation (ICC) analysis of both rater scores assessed for agreement and consistency in grading both primary and secondary outcomes [17]. Univariate analyses and chi-squared tests with Yates' continuity correction of all demographic variables and study groups determined potential associations and the efficacy of randomization. All continuous outcomes were compared with unpaired t-tests, analysis of variance (ANOVA), or Wilcoxon rank-sum tests based on whether they met appropriate statistical assumptions. Multivariate regression analyses of the primary outcome controlled for demographic associations. Subgroup analyses determined the possible effects of the interventions on the heterogeneous sample.
Ordinal logistic regressions determined the log odds likelihood of reporting different levels of Likert-type survey data. Corrected chi-squared tests compared reported distractibility between groups. The VR group was further analyzed regarding their experience with the VR platform.

Results
The study was conducted from November 16, 2020, to January 27, 2021. Fifty-four participants provided informed consent and were randomized: 27 (50%) to the VR group and 27 (50%) to the e-module group.

Demographics
Participants were on average 30.5 years old (SD = 8.6), female (63%, n = 34), and primarily medical students (44%, n = 24) or residents (35%, n = 19   Fisher's exact test assessed the amount of major or minor steps completed correctly between groups. VR demonstrated significantly more steps that were correctly performed in regard to major doffing steps and minor donning steps (p < 0.05). Similar favorable, but non-significant, trends can be seen for all other categories. Time spent in training was measured with a Wilcoxon rank-sum test and found that the VR group spent significantly more time in training than the e-module (VR, 25.6 min vs e-module, 6.5 min; p < 0.001; 95% CI = -20.0 to 16).

Subgroup analyses
Subgroup analyses were based on the most common occupations listed. The study sample was reduced into three groups: medical students, residents, and all other participants (    An additional variable was created for status as a resident for regression analyses. A robust regression model controlled for age, gender, PPE experience, clinical experience, and status as a resident. Interaction effects were analyzed between status as a resident and the study group [18]. A significant two-way interaction was identified between resident status and study group on the overall score. Residents in the VR group scored 6.98 points higher than residents in the e-module group (p = 0.04). However, for non-residents, there was no difference between VR and e-module groups (p = 0.99) ( Table 3).

Participant experience
Ordinal regression of survey responses measured perceived preparedness and confidence retaining information after training. VR was superior to e-module in both outcomes. In terms of perceived preparedness, log odds of reporting a lower score of 3 compared to 4 or 5 is 1.08 points lower in VR than emodule (p = 0.05). For perceived confidence of retention, the log odds of reporting a lower score of 3 compared to 4 or 5 is 1.55 points lower in VR than for e-module (p = 0.007). A larger, but non-significant, percentage of the E-module group reported being distracted compared to VR (59% vs 48%, p = 0.58).

Feedback on VR platform
No participants in the VR group reported experiencing motion sickness (n = 27). Physical complaints reported included blurry vision due to fogging (n=4), claustrophobia (n=1), and headache (n=1). No participants in the e-module group reported physical complaints. In the VR group, eight participants experienced minor technology malfunctions requiring technical assistance, including repeated attempts to register steps (n=5), random restart (n=2), and defective sound from the device (n=1).

Discussion
In this randomized, intervention-controlled trial, VR training was found to be non-inferior to e-module training, although VR participants reported statistically significant increases in preparedness and confidence in donning and doffing PPE after training.
The study population was heterogeneous and the effects of the intervention differed depending on occupation -importantly, residents compared to non-residents. Residents utilizing VR achieved statistically significant increases in PPE score compared to those utilizing e-modules, while non-residents had nonsignificant increases in VR compared to e-module groups. These observations likely result from differences in training time between residents and non-residents in the e-module group, namely that the residents in the e-module group spent significantly less time in PPE training and thus obtained decreased PPE scores compared to other participants utilizing the e-module (Figure 3). On the contrary, residents in the VR group spent just as much time as medical students in training, lending itself to PPE scores comparable to other participants.
Nonetheless, this study's findings confirm the potential of e-module training in certain motivated groups but suggest it may be less effective with other individuals [19]. Residents represent an important subgroup with regards to PPE competency, as many were deployed to the front lines during the initial COVID-19 response and may have experienced some of the highest risks for exposure [20]. At this very institution, a study found that 42% of internal medicine residents deployed during the initial COVID-19 response presented with COVID-19 or COVID-like illness [21]. This rate is likely comparable to emergency medicine and otolaryngology residents, who held similar roles at the time. Of note, while 68% of residents in the study reported treating more than 40 COVID-19 patients in the last year, 47% of residents reported never receiving PPE training and 21% of residents had it more than one year prior, highlighting the importance of finding the right training for the right population at the right time.
Although the results of this study cannot support the hypothesis that VR PPE training is superior to emodule, they do suggest that VR may be an effective training tool in specific groups, such as residents. Several factors may explain this effect. Most importantly, residents may have had difficulty engaging with the e-module, as shown by the significantly decreased e-module training time in residents compared to medical students versus the similar training times between these two groups in VR training. Increased stressors during residency may influence the quality of learning devoted to self-paced instruction [22]. Residents may be more susceptible to distractions in non-immersive environments, and stressors can encourage quicker completion of training [23]. Additionally, given the experimental design, residents may have been less vulnerable to the Hawthorne effect compared to their medical student counterparts [24]. At earlier stages of their careers, medical students may consciously or subconsciously perform differently in the presence of superiors. Therefore, resident effort and time towards e-module training may be more realistic to the manner e-modules are completed without supervision. Finally, residents who went through a more engaging and interactive simulation-based environment (e.g., VR training) may have performed significantly better due to the high correlation between learner engagement and the effectiveness of teaching [25]. It has been found that engaging methods of learning can help maximize retention, decision making, and psychomotor skills of medical tasks -each of which is critical in stressful emergency step-by-step procedures like PPE donning and doffing [26][27][28].

Limitations and future directions
The current study recruited from a heterogeneous convenience sample of participants within a single hospital system, and selection bias may have influenced results. As a single institution with already limited study power, it was not feasible to execute numerous multi-factorial randomized studies to study this intervention in separate groups. Additionally, given the recent pandemic, there was an urgent need for healthcare workers to learn donning and doffing. Therefore, to understand which populations benefit from VR-based training and also control for selection biases, numerous subgroup analyses were conducted.
An inherent limitation of most VR-based programs such as our own is the additional time required to complete the necessary training. VR programs include additional tutorial and training features that do not allow skipping through materials, but such features may actually provide additional benefits in the retention of the learned material [29].
Additionally, both training modalities were not true asynchronous modes of learning since the research team was present to assist the participants if needed. The participants of the VR group had more assistance than the e-module since the VR game had more technological issues. Further studies should evaluate these modalities in a more asynchronous setting.
Lastly, the donning and doffing performance checklist, although based on CDC guidelines, has not been validated. To our knowledge, there is only one validated rubric from 2014, which does not emphasize the correct sequence of steps [30]. The research team worked with selected experts to create a performance checklist that prioritized sequence and allowed for analysis of individual steps as well as general performance. This checklist allowed for rigorous analysis of PPE performance differences between different training modalities. However, we encourage future researchers to refine the rubric to understand the generalizability and reliability of its results.

Conclusions
In this randomized trial, VR training was found to be non-inferior to an e-module for asynchronous PPE training in a heterogeneous group of medical students, residents, and faculty. In residents alone, VR training led to improved PPE performance compared to the e-module training. Overall, VR participants felt more confident and prepared to don and doff PPE after training compared to their e-module counterparts. These findings are particularly relevant given the ongoing COVID-19 pandemic and the increasing need to provide effective PPE training to healthcare providers. Future studies should evaluate how to integrate the VR program into the medical curriculum and identify further subgroups that may benefit from the program. Additionally, long-term retention of VR donning and doffing PPE training must be monitored in the future to further pursue a golden standard for PPE training.

Additional Information Disclosures
Human subjects: Consent was obtained or waived by all participants in this study. Montefiore Medical Center issued approval #NCT04937257. Animal subjects: All authors have confirmed that this study did not involve animal subjects or tissue. Conflicts of interest: In compliance with the ICMJE uniform disclosure form, all authors declare the following: Payment/services info: All authors have declared that no financial support was received from any organization for the submitted work. Financial relationships: All authors have declared that they have no financial relationships at present or within the previous three years with any organizations that might have an interest in the submitted work. Other relationships: All authors have declared that there are no other relationships or activities that could appear to have influenced the submitted work.