"Never doubt that a small group of thoughtful, committed citizens can change the world. Indeed, it is the only thing that ever has."

Margaret Mead
Original article
peer-reviewed

Virtual Reality for Pediatric Sedation: A Randomized Controlled Trial Using Simulation



Abstract

Introduction: Team training for procedural sedation for pediatric residents has traditionally consisted of didactic presentations and simulated scenarios using high-fidelity mannequins. We assessed the effectiveness of a virtual reality module in teaching preparation for and management of sedation for procedures.

Methods: After developing a virtual reality environment in Second Life® (Linden Lab, San Francisco, CA) where providers perform and recover patients from procedural sedation, we conducted a randomized controlled trial to assess the effectiveness of the virtual reality module versus a traditional web-based educational module. A 20 question pre- and post-test was administered to assess knowledge change. All subjects participated in a simulated pediatric procedural sedation scenario that was video recorded for review and assessed using a 32-point checklist. A brief survey elicited feedback on the virtual reality module and the simulation scenario.

Results: The median score on the assessment checklist was 75% for the intervention group and 70% for the control group (P = 0.32). For the knowledge tests, there was no statistically significant difference between the groups (P = 0.14). Users had excellent reviews of the virtual reality module and reported that the module added to their education.

Conclusions: Pediatric residents performed similarly in simulation and on a knowledge test after a virtual reality module compared with a traditional web-based module on procedural sedation. Although users enjoyed the virtual reality experience, these results question the value virtual reality adds in improving the performance of trainees. Further inquiry is needed into how virtual reality provides true value in simulation-based education.

Introduction

Pediatric procedural sedation is a common intervention performed in multiple areas of the hospital. Competency assessment and sedation credentialing are largely determined by individual hospitals and their sedation committees, which are often chaired by an anesthesiologist whose reference standard may be limited to the operating room [1]. With procedural sedation occurring across hospital settings, training should reflect additional patient care areas.

In training health care providers in procedural sedation, as with all other clinical areas, multiple challenges exist, ranging from the new evidence to be incorporated into existing time-limited curricula to the need to improve not only knowledge but also technical skills and critical reasoning capabilities. Physician and nurses have historically been trained under an apprenticeship model in which they wait for an opportunity to perform a particular skill under the guidance of a senior faculty member [2]. This approach can result in far fewer practice occasions than desirable to achieve competence and may result in unnecessary discomfort for patients while trainees perform procedures they are not adept in.

In the past decade, simulation has emerged as a technique for practicing high-stakes, yet low-frequency tasks, in medical education curricula, including pediatric procedural sedation [3-5]. Many studies have demonstrated that virtual reality is well accepted, and some have demonstrated its validity for skills training, with particularly good evidence in surgical fields [6-11]. However, studies assessing its effectiveness in team training and patient care scenarios has been limited [12-14]. A recent meta-analysis by Cook indicated that virtual reality performed better than no intervention, but not better than other instructional designs [15]. The aim of this study was to develop a virtual reality–based procedural sedation experience and assess its effectiveness compared to an existing web-based didactic training module. Educational outcome measures included high-fidelity simulation and knowledge assessments.

Materials & Methods

Study design

We conducted a randomized controlled trial to assess the effectiveness of the virtual reality educational module compared with the traditional web-based module for pediatric sedation training. We chose a high-fidelity simulation for assessment to have participants show how they perform procedural sedation. The intervention group used the virtual reality module, and the control group completed the web-based module. The study was granted exempt status by the Institutional Review Board at Children’s National Medical Center (#Pro00001633) with a waiver of written informed consent. The authors provided an information sheet to each participant prior to participation. We invited all postgraduate year two and three pediatrics residents to participate in the study during the 2012-2013 academic year, arranging to complete enrollment during their Emergency Department rotation. Postgraduate year one residents were excluded to minimize variation in experience. An information sheet was provided to all residents explaining the study. Once enrolled and scheduled, the resident was randomized to the intervention or control group by an administrative assistant. The participant completed the pretest in advance, spent 20 to 30 minutes to complete the intervention or control module, and then proceeded to the Simulation Center to complete the simulation assessment and post-test.

Educational modules

In the first phase of this project, in partnership with a technology consultant (SRA International), we developed a virtual reality program in which participants engage in the complete process of providing pediatric procedural sedation in two unique patient scenarios. At the time of development, the team chose Second Life® (Linden Lab, San Francisco, CA) as the software environment for this project. The virtual reality module provided an anteroom to orient the user to basic functions of movement and interaction in the Second Life environment. Then, the user proceeded to one room where a child required sedation for a fracture reduction and another room where a child required sedation for a CT scan with contrast. Each room was constructed similarly to the actual rooms at our hospital (Figure 1).

The user then responded to a series of questions to prepare the patient for sedation, worked with a pre-programmed nurse to administer the sedation, and needed to intervene and treat a complication related to the sedation. With a resolution of the complication, the scenario ended and the user was directed to the other room.

The control module was a web-based learning module with a series of slides with overlaid audio, intermittent video clips, and knowledge-check questions (Figure 2). The slides covered definitions, indications, and contraindications, a review of the medications, competencies of the providers, complications, equipment needed, and the process for sedation. This module was developed locally by the Department of Anesthesiology and has been the standard at our institution for credentialing in procedural sedation for many years.

Simulation scenario

After completing the intervention or control module, all residents then immediately participated in a simulated pediatric procedural sedation scenario. All simulations occurred in the Simulation Center with an infant patient simulator (SimBaby, Laerdal Medical, Stavanger, Norway). A standard scenario and script were created and piloted before enrolling participants with a pediatric emergency medicine attending and a fellow. The scenario was standardized with two team members (AD, PZ) interacting with the participants following an explicit script. The Simulation Center was used to ensure a standard setup and availability of equipment, as well as familiarity with the mannequin and arrangement by the residents. The scenario involved a 12-month-old infant with bilateral lower-extremity burns requiring debridement. The scenario required answering several questions to set the stage followed by a preparation phase and complication phase. Once the requested medication was delivered, the patient would develop laryngospasm (if given ketamine) or hypopnea (if given a narcotic/benzodiazepine), both of which were programmed to resolve once positive-pressure ventilation was delivered.

Assessment instruments

The preparation and event were video and audio recorded in the Simulation Center. The video review was completed with the Laerdal Debrief Viewer (Laerdal Medical, Stavanger, Norway), which allowed three camera views, a monitor review, and an event log to be viewed simultaneously. Each performance video was reviewed by one or two team members (KO, EW) blinded to the group allocation. Performance on preparation and management of a complication was assessed using a 32-point checklist, adapted for this sedation scenario from a previously published checklist [16]. The initial checklist was determined by a consensus from a panel of experts in pediatric emergency medicine and pediatric procedural sedation [16]. The clinical performance checklist was first modified for the delivery of care as it occurs at our hospital. Then, in a modified Delphi format, the revised checklist was sent to a panel of experts in simulation, emergency medicine, and pediatric emergency medicine to ensure content validity in the simulated setting as well as clarity and applicability. With three rounds, a consensus was achieved on the checklist (Figure 3).

A twenty question test was administered to assess knowledge change. This test is the required assessment for the control module, which is currently required as part of credentialing at our hospital. The same test was used as the pre-test and the post-test with only a change in the order of the questions.

In addition to knowledge and performance assessment, feedback on the virtual reality module and the simulation scenario was obtained from each participant through a brief five-item survey.

Statistical analysis

To demonstrate a 20% difference between scores on the performance checklist of the intervention and control groups, each group required 15 participants. A Mann-Whitney U test was performed to compare the intervention and control group scores on the checklist, the pre-tests, and the post-tests. A Wilcoxon signed rank test compared pre- versus post-test improvements for each group. Descriptive results of the feedback surveys on the virtual reality module and simulation were also reviewed. An intraclass coefficient was determined to assess the reliability of the video review process between the two reviewers.

Study data were collected and managed using REDCap electronic data capture tools hosted at Children's National Medical Center. REDCap is a secure, web-based application designed to support data capture for research studies [17]. Statistical analysis was performed using SPSS version 21 (SPSS, Inc., Chicago, IL).

Results

We enrolled 32 residents in this study from May to December 2013. Due to various conflicts, only 10 residents in the intervention group and 12 residents in the control group completed the simulation (Figure 4). Due to eight recording failures, simulation data for assessment of the primary outcome were obtained for only seven residents in each group. The groups were fairly similar in their level of training and experience with no statistically significant differences between the groups (Table 1).

Characteristic Intervention (N = 7) Control (N = 7)*
PGY 2 4 5
Sedations participated in (median, IQR) 1 (1-5) 2 (2-2)
Total sedations performed as primary provider 4 7
Total sedation complications managed 2 2

Assessment

Results of the pre-test, post-test, and simulation assessment are provided in Table 2. The intervention group had a median score of 75% for the assessment checklist versus 70% for the control with no statistically significant difference (P = 0.32). Reliability of the video review was confirmed with an intraclass coefficient of 0.69.

  Median (Interquartile Range) P
Value*
Intervention Group Control Group
Knowledge pre-test 14/20 (13.5 to 15) 13/20 (11.75 to 14.25) 0.25
Assessment checklist 24/32 (22.5 to 25) 22.5/32 (20 to 25) 0.32
Knowledge post-test 15/20 16/20  
Difference, post-test – pre-test 1 (–1 to 2.25) P = 0.13+ 3 (1 to 4) P = 0.007+ 0.14

For the knowledge test, there were no statistically significant differences between the groups for both pre- and post-test scores. The intervention group improved its score by a median of 1 point with an interquartile range of –1 to 1.25, whereas the control group improved its score by a median of 3 points with an interquartile range of 1 to 4 points.

Usability of virtual reality module

In the brief survey of usability, eight of 12 participants who completed the virtual reality module agreed or strongly agreed that the module was easy to use and navigate. All residents who did the virtual reality module felt it added to their education regarding sedation. Residents further commented:

“I enjoyed the interactive display, allowed me to critically think and get instant feedback.”

“Overall, enjoyed the experience.”

“Lots of fun! Made learning easy, active, and interactive.”

“Great experience. Would love more of these throughout training.”

“Great module, would recommend it to all trainees.”

“Allowed for step-by-step thinking process in regards to approaching sedating a patient.”

Discussion

This study demonstrated that residents performed similarly after review of a standard web-based module or a virtual reality module. Development of the module was a time- and fund-intensive investment occurring over two years of interactions with a software consultant and a $40,000 grant to create two scenarios. The assessment did not demonstrate any difference between the traditional web-based and virtual reality modules aside from a positive participant experience, although limited by an inadequate sample.

In looking at the value of virtual reality, insight is provided by analyzing studies by the level reached on the Kirkpatrick hierarchy. Kirkpatrick described level 1 of evaluation as reactions; level 2, learning; level 3, behavior and performance; and level 4, results—which would be patient outcomes in the clinical setting. Most studies have reported excellent results when assessing reaction at Kirkpatrick’s level 1 [6-7, 18-20], but few studies have shown behavioral outcomes at Kirkpatrick level 3 for virtual reality. Our study adds to the literature in showing positive findings at Kirkpatrick level 1 in that the participants enjoyed the experience; however, performance at levels 2 and 3 was no different than our control module, and, thus, it is unclear if we can improve performance beyond the control module through virtual reality.

Studies evaluating virtual reality to teach common procedures, such as intravenous catheter placement, fiberoptic intubation, and bronchoscopy, have reported improvement in performance success with decreased errors and time to completion [7-8, 11, 21]. In a review of the neurosurgical literature using virtual reality, most studies demonstrated positive user feedback with only a few able to show improvements in time to completion and one with performance improvement on a checklist [6]. When evaluating performance on specific surgical procedures after training with virtual reality, several studies demonstrated modest improvements on procedural checklists without any difference in time to completion [9-10, 22]. Kundhal, et al. correlated a task-based checklist from a virtual reality performance with actual operating room performance, an example starting to reach Kirkpatrick level 4 assessment [23]. These studies have shown that virtual reality training can effect change at Kirkpatrick level 3 and occasionally at level 4, patient outcomes, achieving demonstrable value to education.

However, those studies evaluating an interactive procedure, such as mass casualty triage, have had more mixed results, not offering enough argument to suggest widespread use and investment [12, 24-28]. Although most participants enjoy the experience, the performance measures do not uniformly improve over alternative modalities, such as live simulation [12, 24] or card sort triage [27]. Such results continue to call into question the true value of virtual reality.

Studies specifically evaluating Second Life as the platform for virtual reality have demonstrated feasibility, with positive participant feedback related to the modality, but have not researched performance outcomes [14, 18-20]. When Kidd, et al. used Second Life to teach nursing students how to do mental health assessments, the students enjoyed the safety of simulation and the interactivity afforded but dressing and maneuvering the avatars and a lack of realism limited full engagement [19]. At another nursing school, the faculty implemented Second Life for one curriculum, where again reactions were positive but limited by lack of orientation and difficulty with the technology [20]. Finally, in a third report, as part of a $1.6 million grant, the faculty developed a simulation instruction experience in Second Life and reported that it was feasible based on user feedback, but did not demonstrate any outcomes on assessment [14]. In contrast, Cohen, et al.’s development in Second Life demonstrated positive participant feedback and face validity, yet called for “robust assessment metrics,” highlighting the challenges in showing outcomes improvement [18].

Similar to our study where a scenario was to be managed, Nyssen, et al. demonstrated that the virtual reality environment can result in improved diagnosis time and treatment scores when repeating a virtual reality scenario, but can fall short in covering the full range of patient scenario management compared with high-fidelity simulation scenarios. The specific area of improvement was in technical skills, which prompts the question of whether the virtual reality was necessary or a reading or didactic may have sufficed [29]. In our study, looking at preparation for sedation and management of an adverse event, the virtual reality module did not demonstrate superior outcomes compared with the traditional web-based module.

The mixed results of studies evaluating the efficacy of virtual reality training on participant performance bring to the forefront the question of the value of virtual reality in medical education. Educators need to carefully assess the balance of cost, time, availability of outcome data, and generalizability to various settings and audiences before deciding to build or invest in virtual reality. Virtual reality as a modality to flip the classroom and immerse the learner has yet to be proven as creating value for the participant and the educator.

There were several limitations to this study. First, our study had a small number of participants in each group due to inadequate enrollment numbers and technical failures. Our planned recruitment numbers did not account for this unforeseen complication. This raises the possibility of our study failing to identify the effect of virtual reality due to an inadequate sample size. Second, we encountered funding shortages and were able to design only half of the intended Second Life scenarios. As others have shared in their descriptions of virtual reality development, the design and creation aspect can be quite expensive [14, 18, 20]. This can create significant challenges in creating a sustainable product and modifying it as users provide feedback. Fortunately, newer software for virtual reality function in common web browsers allows wider adaptability and easier modification. 

Conclusions

Our study demonstrated high participant satisfaction with the virtual reality module, but we were unable to show a difference in sedation performance for those trained using virtual reality compared with the traditional web-based module. Participants felt the virtual reality environment added positively to their education and was easy to use. We acknowledge that our sample size for this research project was small and encourage further research to investigate the value of virtual reality modules in medical education. 


References

  1. Shankar V, Deshpande JK: Procedural sedation in the pediatric patient. Anesthesiol Clin North America. 2005, 23:635–54. 10.1016/j.atc.2005.07.001
  2. Lenchus JD: End of the "see one, do one, teach one" era: the next generation of invasive bedside procedural instruction. J Am Osteopath Assoc. 2010, 110:340–46.
  3. Tobin CD, Clark CA, McEvoy MD, Reves JG, Schaefer JJ, Wolf BJ, Reeves ST: An approach to moderate sedation simulation training. Simul Healthc. 2013, 8:114–23. 10.1097/SIH.0b013e3182786209
  4. Jendrusch J, Lampotang S, Lizdas D, Gravenstein N, Ham D, Lok B, Quarles J: Virtual humans for inter-ethnic variability training in sedation and analgesia. Medicine Meets Virtual Reality 21. Westwood J, Westwood SW, Felländer-Tsai, L, Fidopiastis CM, Haluck RS, Robb RA, Senger S, Vosburgh KG (ed): IOS Press, Fairfax, VA; 2014. 196:175–81. 10.3233/978-1-61499-375-9-175
  5. Shavit I, Keidan I, Hoffmann Y, Mishuk L, Rubin O, Ziv A, Steiner IP: Enhancing patient safety during pediatric sedation: the impact of simulation-based training of nonanesthesiologists. Arch Pediatr Adolesc Med. 2007, 161:740–43. 10.1001/archpedi.161.8.740
  6. Alaraj A, Lemole MG, Finkle JH, Yudkowsky R, Wallace A, Luciano C, Banerjee PP, Rizzi SH, Charbel FT: Virtual reality training in neurosurgery: Review of current status and future applications. Surg Neurol Int. 2011, 2:52. 10.4103/2152-7806.80117
  7. Loukas C, Nikiteas N, Kanakis M, Georgiou E: Evaluating the effectiveness of virtual reality simulation training in intravenous cannulation. Simul Healthc. 2011, 6:213–17. 10.1097/SIH.0b013e31821d08a9
  8. Rowe R, Cohen RA: An evaluation of a virtual reality airway simulator. Anesth Analg. 2002, 95:62–66. 10.1097/00000539-200207000-00011
  9. Shirai Y, Yoshida T, Shiraishi R, Okamoto T, Nakamura H, Harada T, Nishikawa J, Sakaida I: Prospective randomized study on the use of a computer-based endoscopic simulator for training in esophagogastroduodenoscopy. J Gastroenterol Hepatol. 2008, 23:1046–50. 10.1111/j.1440-1746.2008.05457.x
  10. Taffinder N, Sutton C, Fishwick RJ, McManus IC, Darzi A: Validation of virtual reality to teach and assess psychomotor skills in laparoscopic surgery: results from randomised controlled studies using the MIST VR laparoscopic simulator. Medicine Meets Virtual Reality. Westwood JD, Hoffman HM, Stredney D, Weghorst SJ (ed): IOS Press, Oxford; 1998. 50:124–30. 10.3233/978-1-60750-894-6-124
  11. Colt HG, Crawford SW, Galbraith O 3rd: Virtual reality bronchoscopy simulation: a revolution in procedural training. Chest. 2001, 120:1333–39. 10.1378/chest.120.4.1333
  12. Luigi Ingrassia P, Ragazzoni L, Carenzo L, Colombo D, Ripoll Gallardo A, Della Corte F: Virtual reality and live simulation: a comparison between two simulation tools for assessing mass casualty triage skills. Eur J Emerg Med. 2015, 22:121-27. 101097/MEJ?0000000000000132
  13. Medina LS, Racadio JM, Schwid HA: Computers in radiology. The sedation, analgesia, and contrast media computerized simulator: a new approach to train and evaluate radiologists' responses to critical incidents. Pediatr Radiol. 2000, 30:299–305. 10.1007/s002470050744
  14. Weiner E, McNew R, Trangenstein P, Gordon J: Using the virtual reality world of Second Life to teach nursing faculty simulation management. MEDINFO 2010: Proceedings of the 13th World Congress on Medical Informatics, Part 1. Safran C, Reti S, Marin HF (ed): IOS Press, CA; 2010. 160:615–19. 10.3233/978-1-60750-588-4-615
  15. Cook DA, Erwin PJ, Triola MM : Computerized virtual patients in health professions education: a systematic review and meta-analysis. Acad Med. 2010, 85:1589–602. 10.1097/ACM.0b013e3181edfe13
  16. Schinasi DA, Nadel FM, Hales R, Boswinkel JP, Donoghue AJ: Assessing pediatric residents' clinical performance in procedural sedation: a simulation-based needs assessment. Pediatr Emerg Care. 2013, 29:447–52. 10.1097/PEC.0b013e31828b6552
  17. Harris PA, Taylor R, Thielke R, Payne J, Gonzalez N, Conde JG: Research electronic data capture (REDCap)--a metadata-driven methodology and workflow process for providing translational research informatics support. J Biomed Inform. 2009, 42:377–81. 10.1016/j.jbi.2008.08.010
  18. Cohen D, Sevdalis N, Taylor D, Kerr K, Heys M, Willett K, Batrick N, Darzi A: Emergency preparedness in the 21st century: Training and preparation modules in virtual environments. Resuscitation. 2013, 84:78–84. 10.1016/j.resuscitation.2012.05.014
  19. Kidd LI, Knisley SJ, Morgan KI: Effectiveness of a Second Life® simulation as a teaching strategy for undergraduate mental health nursing students. J Psychosoc Nurs Ment Health Serv. 2012, 50:28–37. 10.3928/02793695-20120605-04
  20. Schmidt B, Stewart S: Implementing the virtual reality learning environment: Second Life. Nurse Educ. 2009, 34:152–55. 10.1097/NNE.0b013e3181aabbe8
  21. Goldmann K, Steinfeldt T : Acquisition of basic fiberoptic intubation skills with a virtual reality airway simulator. J Clin Anesth. 2006, 18:173–78. 10.1016/j.jclinane.2005.08.021
  22. Park J, MacRae H, Musselman LJ, Rossos P, Hamstra SJ, Wolman S, Reznick RK: Randomized controlled trial of virtual reality simulator training: transfer to live patients. Am J Surg. 2007, 194:205–11. 10.1016/j.amjsurg.2006.11.032
  23. Kundhal PS, Grantcharov TP: Psychomotor performance measured in a virtual environment correlates with technical skills in the operating room. Surg Endosc. 2009, 23:645–49. 10.1007/s00464-008-0043-5
  24. Andreatta PB, Maslowski E, Petty S, Shim W, Marsh M, Hall T, Stern S, Frankel J: Virtual reality triage training provides a viable solution for disaster-preparedness. Acad Emerg Med. 2010, 17:870–76. 10.1111/j.1553-2712.2010.00728.x
  25. Cone DC, Serra J, Kurland L: Comparison of the SALT andSmart triage systems using a virtual reality simulator with paramedic students. Eur J Emerg Med. 2011, 18:314–21. 10.1097/MEJ.0b013e328345d6fd
  26. Heinrichs WL, Youngblood P, Harter P, Kusumoto L, Dev P: Training healthcare personnel for mass-casualty incidents in a virtual emergency department: VED II. Prehosp Disaster Med. 2010, 25:424–32.
  27. Knight JF, Carley S, Tregunna B, Jarvis S, Smithies R, de Freitas S, Dunwell I, Mackway-Jones K: Serious gaming technology in major incident triage training: a pragmatic controlled trial. Resuscitation. 2010, 81:1175–79. 10.1016/j.resuscitation.2010.03.042
  28. Vincent DS, Sherstyuk A, Burgess L, Connolly KK: Teaching mass casualty triage skills using immersive three-dimensional virtual reality. Acad Emerg Med. 2008, 15:1160–65. 10.1111/j.1553-2712.2008.00191.x
  29. Nyssen AS, Larbuisson R, Janssens M, Pendeville P, Mayné A: A comparison of the training value of two types of anesthesia simulators: computer screen-based and mannequin-based simulators. Anesth Analg. 2002, 94:1560-65. 10.1097/00000539-200206000-00035
Original article
peer-reviewed

Virtual Reality for Pediatric Sedation: A Randomized Controlled Trial Using Simulation


Author Information

Pavan P. Zaveri Corresponding Author

Division of Emergency Medicine, Children's National Health System

Aisha B. Davis

Division of Hospital Medicine, Children's National Health System

Karen J. O'Connell

Division of Emergency Medicine, Children's National Health System

Emily Willner

Division of Emergency Medicine, Children's National Health System

Dana A. Aronson Schinasi

Division of Emergency Medicine, Lurie Children's Hospital

Mary Ottolini

Division of Hospital Medicine, Children's National Health System


Ethics Statement and Conflict of Interest Disclosures

Human subjects: Children's National Health System issued approval Pro00001633. This study was deemed exempt by the Institutional Review Board. As an educational study, the only link to the participant would be the name on the consent form, so written informed consent was waived. An information sheet was provided to each participant prior to participating in the study. Animal subjects: This study did not involve animal subjects or tissue.

Acknowledgements

The authors thank Eileen Searson and Maria Lewis for their assistance in data management, Todd P. Chang, MD, MAcM for his assistance in module development, and Paul Santoro and Terrell Chandler of SRA International for their collaboration in developing the virtual reality module. The development of the virtual reality module was supported by a grant from the Children’s National Board of Visitors.


Original article
peer-reviewed

Virtual Reality for Pediatric Sedation: A Randomized Controlled Trial Using Simulation


Figures etc.

PDF Print Share
Edward J. Rovera
February 09, 2016 at 08:41 AM
Edward J. Rovera

Given that the bulk of the sizable cost of developing the two scenarios has already be spent, it would be good to see this study replicated with a larger sample size.

Original article
peer-reviewed

Virtual Reality for Pediatric Sedation: A Randomized Controlled Trial Using Simulation

  • Author Information
    Pavan P. Zaveri Corresponding Author

    Division of Emergency Medicine, Children's National Health System

    Aisha B. Davis

    Division of Hospital Medicine, Children's National Health System

    Karen J. O'Connell

    Division of Emergency Medicine, Children's National Health System

    Emily Willner

    Division of Emergency Medicine, Children's National Health System

    Dana A. Aronson Schinasi

    Division of Emergency Medicine, Lurie Children's Hospital

    Mary Ottolini

    Division of Hospital Medicine, Children's National Health System


    Ethics Statement and Conflict of Interest Disclosures

    Human subjects: Children's National Health System issued approval Pro00001633. This study was deemed exempt by the Institutional Review Board. As an educational study, the only link to the participant would be the name on the consent form, so written informed consent was waived. An information sheet was provided to each participant prior to participating in the study. Animal subjects: This study did not involve animal subjects or tissue.

    Acknowledgements

    The authors thank Eileen Searson and Maria Lewis for their assistance in data management, Todd P. Chang, MD, MAcM for his assistance in module development, and Paul Santoro and Terrell Chandler of SRA International for their collaboration in developing the virtual reality module. The development of the virtual reality module was supported by a grant from the Children’s National Board of Visitors.


    Article Information

    Published: February 09, 2016

    DOI

    10.7759/cureus.486

    Cite this article as:

    Zaveri P P, Davis A B, O'connell K J, et al. (February 09, 2016) Virtual Reality for Pediatric Sedation: A Randomized Controlled Trial Using Simulation . Cureus 8(2): e486. doi:10.7759/cureus.486

    Publication history

    Received by Cureus: October 30, 2015
    Peer review began: November 09, 2015
    Peer review concluded: January 28, 2016
    Published: February 09, 2016

    Copyright

    © Copyright 2016
    Zaveri et al. This is an open access article distributed under the terms of the Creative Commons Attribution License CC-BY 3.0., which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.

    License

    This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.

Abstract

Introduction: Team training for procedural sedation for pediatric residents has traditionally consisted of didactic presentations and simulated scenarios using high-fidelity mannequins. We assessed the effectiveness of a virtual reality module in teaching preparation for and management of sedation for procedures.

Methods: After developing a virtual reality environment in Second Life® (Linden Lab, San Francisco, CA) where providers perform and recover patients from procedural sedation, we conducted a randomized controlled trial to assess the effectiveness of the virtual reality module versus a traditional web-based educational module. A 20 question pre- and post-test was administered to assess knowledge change. All subjects participated in a simulated pediatric procedural sedation scenario that was video recorded for review and assessed using a 32-point checklist. A brief survey elicited feedback on the virtual reality module and the simulation scenario.

Results: The median score on the assessment checklist was 75% for the intervention group and 70% for the control group (P = 0.32). For the knowledge tests, there was no statistically significant difference between the groups (P = 0.14). Users had excellent reviews of the virtual reality module and reported that the module added to their education.

Conclusions: Pediatric residents performed similarly in simulation and on a knowledge test after a virtual reality module compared with a traditional web-based module on procedural sedation. Although users enjoyed the virtual reality experience, these results question the value virtual reality adds in improving the performance of trainees. Further inquiry is needed into how virtual reality provides true value in simulation-based education.

Introduction

Pediatric procedural sedation is a common intervention performed in multiple areas of the hospital. Competency assessment and sedation credentialing are largely determined by individual hospitals and their sedation committees, which are often chaired by an anesthesiologist whose reference standard may be limited to the operating room [1]. With procedural sedation occurring across hospital settings, training should reflect additional patient care areas.

In training health care providers in procedural sedation, as with all other clinical areas, multiple challenges exist, ranging from the new evidence to be incorporated into existing time-limited curricula to the need to improve not only knowledge but also technical skills and critical reasoning capabilities. Physician and nurses have historically been trained under an apprenticeship model in which they wait for an opportunity to perform a particular skill under the guidance of a senior faculty member [2]. This approach can result in far fewer practice occasions than desirable to achieve competence and may result in unnecessary discomfort for patients while trainees perform procedures they are not adept in.

In the past decade, simulation has emerged as a technique for practicing high-stakes, yet low-frequency tasks, in medical education curricula, including pediatric procedural sedation [3-5]. Many studies have demonstrated that virtual reality is well accepted, and some have demonstrated its validity for skills training, with particularly good evidence in surgical fields [6-11]. However, studies assessing its effectiveness in team training and patient care scenarios has been limited [12-14]. A recent meta-analysis by Cook indicated that virtual reality performed better than no intervention, but not better than other instructional designs [15]. The aim of this study was to develop a virtual reality–based procedural sedation experience and assess its effectiveness compared to an existing web-based didactic training module. Educational outcome measures included high-fidelity simulation and knowledge assessments.

Materials & Methods

Study design

We conducted a randomized controlled trial to assess the effectiveness of the virtual reality educational module compared with the traditional web-based module for pediatric sedation training. We chose a high-fidelity simulation for assessment to have participants show how they perform procedural sedation. The intervention group used the virtual reality module, and the control group completed the web-based module. The study was granted exempt status by the Institutional Review Board at Children’s National Medical Center (#Pro00001633) with a waiver of written informed consent. The authors provided an information sheet to each participant prior to participation. We invited all postgraduate year two and three pediatrics residents to participate in the study during the 2012-2013 academic year, arranging to complete enrollment during their Emergency Department rotation. Postgraduate year one residents were excluded to minimize variation in experience. An information sheet was provided to all residents explaining the study. Once enrolled and scheduled, the resident was randomized to the intervention or control group by an administrative assistant. The participant completed the pretest in advance, spent 20 to 30 minutes to complete the intervention or control module, and then proceeded to the Simulation Center to complete the simulation assessment and post-test.

Educational modules

In the first phase of this project, in partnership with a technology consultant (SRA International), we developed a virtual reality program in which participants engage in the complete process of providing pediatric procedural sedation in two unique patient scenarios. At the time of development, the team chose Second Life® (Linden Lab, San Francisco, CA) as the software environment for this project. The virtual reality module provided an anteroom to orient the user to basic functions of movement and interaction in the Second Life environment. Then, the user proceeded to one room where a child required sedation for a fracture reduction and another room where a child required sedation for a CT scan with contrast. Each room was constructed similarly to the actual rooms at our hospital (Figure 1).

The user then responded to a series of questions to prepare the patient for sedation, worked with a pre-programmed nurse to administer the sedation, and needed to intervene and treat a complication related to the sedation. With a resolution of the complication, the scenario ended and the user was directed to the other room.

The control module was a web-based learning module with a series of slides with overlaid audio, intermittent video clips, and knowledge-check questions (Figure 2). The slides covered definitions, indications, and contraindications, a review of the medications, competencies of the providers, complications, equipment needed, and the process for sedation. This module was developed locally by the Department of Anesthesiology and has been the standard at our institution for credentialing in procedural sedation for many years.

Simulation scenario

After completing the intervention or control module, all residents then immediately participated in a simulated pediatric procedural sedation scenario. All simulations occurred in the Simulation Center with an infant patient simulator (SimBaby, Laerdal Medical, Stavanger, Norway). A standard scenario and script were created and piloted before enrolling participants with a pediatric emergency medicine attending and a fellow. The scenario was standardized with two team members (AD, PZ) interacting with the participants following an explicit script. The Simulation Center was used to ensure a standard setup and availability of equipment, as well as familiarity with the mannequin and arrangement by the residents. The scenario involved a 12-month-old infant with bilateral lower-extremity burns requiring debridement. The scenario required answering several questions to set the stage followed by a preparation phase and complication phase. Once the requested medication was delivered, the patient would develop laryngospasm (if given ketamine) or hypopnea (if given a narcotic/benzodiazepine), both of which were programmed to resolve once positive-pressure ventilation was delivered.

Assessment instruments

The preparation and event were video and audio recorded in the Simulation Center. The video review was completed with the Laerdal Debrief Viewer (Laerdal Medical, Stavanger, Norway), which allowed three camera views, a monitor review, and an event log to be viewed simultaneously. Each performance video was reviewed by one or two team members (KO, EW) blinded to the group allocation. Performance on preparation and management of a complication was assessed using a 32-point checklist, adapted for this sedation scenario from a previously published checklist [16]. The initial checklist was determined by a consensus from a panel of experts in pediatric emergency medicine and pediatric procedural sedation [16]. The clinical performance checklist was first modified for the delivery of care as it occurs at our hospital. Then, in a modified Delphi format, the revised checklist was sent to a panel of experts in simulation, emergency medicine, and pediatric emergency medicine to ensure content validity in the simulated setting as well as clarity and applicability. With three rounds, a consensus was achieved on the checklist (Figure 3).

A twenty question test was administered to assess knowledge change. This test is the required assessment for the control module, which is currently required as part of credentialing at our hospital. The same test was used as the pre-test and the post-test with only a change in the order of the questions.

In addition to knowledge and performance assessment, feedback on the virtual reality module and the simulation scenario was obtained from each participant through a brief five-item survey.

Statistical analysis

To demonstrate a 20% difference between scores on the performance checklist of the intervention and control groups, each group required 15 participants. A Mann-Whitney U test was performed to compare the intervention and control group scores on the checklist, the pre-tests, and the post-tests. A Wilcoxon signed rank test compared pre- versus post-test improvements for each group. Descriptive results of the feedback surveys on the virtual reality module and simulation were also reviewed. An intraclass coefficient was determined to assess the reliability of the video review process between the two reviewers.

Study data were collected and managed using REDCap electronic data capture tools hosted at Children's National Medical Center. REDCap is a secure, web-based application designed to support data capture for research studies [17]. Statistical analysis was performed using SPSS version 21 (SPSS, Inc., Chicago, IL).

Results

We enrolled 32 residents in this study from May to December 2013. Due to various conflicts, only 10 residents in the intervention group and 12 residents in the control group completed the simulation (Figure 4). Due to eight recording failures, simulation data for assessment of the primary outcome were obtained for only seven residents in each group. The groups were fairly similar in their level of training and experience with no statistically significant differences between the groups (Table 1).

Characteristic Intervention (N = 7) Control (N = 7)*
PGY 2 4 5
Sedations participated in (median, IQR) 1 (1-5) 2 (2-2)
Total sedations performed as primary provider 4 7
Total sedation complications managed 2 2

Assessment

Results of the pre-test, post-test, and simulation assessment are provided in Table 2. The intervention group had a median score of 75% for the assessment checklist versus 70% for the control with no statistically significant difference (P = 0.32). Reliability of the video review was confirmed with an intraclass coefficient of 0.69.

  Median (Interquartile Range) P
Value*
Intervention Group Control Group
Knowledge pre-test 14/20 (13.5 to 15) 13/20 (11.75 to 14.25) 0.25
Assessment checklist 24/32 (22.5 to 25) 22.5/32 (20 to 25) 0.32
Knowledge post-test 15/20 16/20  
Difference, post-test – pre-test 1 (–1 to 2.25) P = 0.13+ 3 (1 to 4) P = 0.007+ 0.14

For the knowledge test, there were no statistically significant differences between the groups for both pre- and post-test scores. The intervention group improved its score by a median of 1 point with an interquartile range of –1 to 1.25, whereas the control group improved its score by a median of 3 points with an interquartile range of 1 to 4 points.

Usability of virtual reality module

In the brief survey of usability, eight of 12 participants who completed the virtual reality module agreed or strongly agreed that the module was easy to use and navigate. All residents who did the virtual reality module felt it added to their education regarding sedation. Residents further commented:

“I enjoyed the interactive display, allowed me to critically think and get instant feedback.”

“Overall, enjoyed the experience.”

“Lots of fun! Made learning easy, active, and interactive.”

“Great experience. Would love more of these throughout training.”

“Great module, would recommend it to all trainees.”

“Allowed for step-by-step thinking process in regards to approaching sedating a patient.”

Discussion

This study demonstrated that residents performed similarly after review of a standard web-based module or a virtual reality module. Development of the module was a time- and fund-intensive investment occurring over two years of interactions with a software consultant and a $40,000 grant to create two scenarios. The assessment did not demonstrate any difference between the traditional web-based and virtual reality modules aside from a positive participant experience, although limited by an inadequate sample.

In looking at the value of virtual reality, insight is provided by analyzing studies by the level reached on the Kirkpatrick hierarchy. Kirkpatrick described level 1 of evaluation as reactions; level 2, learning; level 3, behavior and performance; and level 4, results—which would be patient outcomes in the clinical setting. Most studies have reported excellent results when assessing reaction at Kirkpatrick’s level 1 [6-7, 18-20], but few studies have shown behavioral outcomes at Kirkpatrick level 3 for virtual reality. Our study adds to the literature in showing positive findings at Kirkpatrick level 1 in that the participants enjoyed the experience; however, performance at levels 2 and 3 was no different than our control module, and, thus, it is unclear if we can improve performance beyond the control module through virtual reality.

Studies evaluating virtual reality to teach common procedures, such as intravenous catheter placement, fiberoptic intubation, and bronchoscopy, have reported improvement in performance success with decreased errors and time to completion [7-8, 11, 21]. In a review of the neurosurgical literature using virtual reality, most studies demonstrated positive user feedback with only a few able to show improvements in time to completion and one with performance improvement on a checklist [6]. When evaluating performance on specific surgical procedures after training with virtual reality, several studies demonstrated modest improvements on procedural checklists without any difference in time to completion [9-10, 22]. Kundhal, et al. correlated a task-based checklist from a virtual reality performance with actual operating room performance, an example starting to reach Kirkpatrick level 4 assessment [23]. These studies have shown that virtual reality training can effect change at Kirkpatrick level 3 and occasionally at level 4, patient outcomes, achieving demonstrable value to education.

However, those studies evaluating an interactive procedure, such as mass casualty triage, have had more mixed results, not offering enough argument to suggest widespread use and investment [12, 24-28]. Although most participants enjoy the experience, the performance measures do not uniformly improve over alternative modalities, such as live simulation [12, 24] or card sort triage [27]. Such results continue to call into question the true value of virtual reality.

Studies specifically evaluating Second Life as the platform for virtual reality have demonstrated feasibility, with positive participant feedback related to the modality, but have not researched performance outcomes [14, 18-20]. When Kidd, et al. used Second Life to teach nursing students how to do mental health assessments, the students enjoyed the safety of simulation and the interactivity afforded but dressing and maneuvering the avatars and a lack of realism limited full engagement [19]. At another nursing school, the faculty implemented Second Life for one curriculum, where again reactions were positive but limited by lack of orientation and difficulty with the technology [20]. Finally, in a third report, as part of a $1.6 million grant, the faculty developed a simulation instruction experience in Second Life and reported that it was feasible based on user feedback, but did not demonstrate any outcomes on assessment [14]. In contrast, Cohen, et al.’s development in Second Life demonstrated positive participant feedback and face validity, yet called for “robust assessment metrics,” highlighting the challenges in showing outcomes improvement [18].

Similar to our study where a scenario was to be managed, Nyssen, et al. demonstrated that the virtual reality environment can result in improved diagnosis time and treatment scores when repeating a virtual reality scenario, but can fall short in covering the full range of patient scenario management compared with high-fidelity simulation scenarios. The specific area of improvement was in technical skills, which prompts the question of whether the virtual reality was necessary or a reading or didactic may have sufficed [29]. In our study, looking at preparation for sedation and management of an adverse event, the virtual reality module did not demonstrate superior outcomes compared with the traditional web-based module.

The mixed results of studies evaluating the efficacy of virtual reality training on participant performance bring to the forefront the question of the value of virtual reality in medical education. Educators need to carefully assess the balance of cost, time, availability of outcome data, and generalizability to various settings and audiences before deciding to build or invest in virtual reality. Virtual reality as a modality to flip the classroom and immerse the learner has yet to be proven as creating value for the participant and the educator.

There were several limitations to this study. First, our study had a small number of participants in each group due to inadequate enrollment numbers and technical failures. Our planned recruitment numbers did not account for this unforeseen complication. This raises the possibility of our study failing to identify the effect of virtual reality due to an inadequate sample size. Second, we encountered funding shortages and were able to design only half of the intended Second Life scenarios. As others have shared in their descriptions of virtual reality development, the design and creation aspect can be quite expensive [14, 18, 20]. This can create significant challenges in creating a sustainable product and modifying it as users provide feedback. Fortunately, newer software for virtual reality function in common web browsers allows wider adaptability and easier modification. 

Conclusions

Our study demonstrated high participant satisfaction with the virtual reality module, but we were unable to show a difference in sedation performance for those trained using virtual reality compared with the traditional web-based module. Participants felt the virtual reality environment added positively to their education and was easy to use. We acknowledge that our sample size for this research project was small and encourage further research to investigate the value of virtual reality modules in medical education. 

References

  1. Shankar V, Deshpande JK: Procedural sedation in the pediatric patient. Anesthesiol Clin North America. 2005, 23:635–54. 10.1016/j.atc.2005.07.001
  2. Lenchus JD: End of the "see one, do one, teach one" era: the next generation of invasive bedside procedural instruction. J Am Osteopath Assoc. 2010, 110:340–46.
  3. Tobin CD, Clark CA, McEvoy MD, Reves JG, Schaefer JJ, Wolf BJ, Reeves ST: An approach to moderate sedation simulation training. Simul Healthc. 2013, 8:114–23. 10.1097/SIH.0b013e3182786209
  4. Jendrusch J, Lampotang S, Lizdas D, Gravenstein N, Ham D, Lok B, Quarles J: Virtual humans for inter-ethnic variability training in sedation and analgesia. Medicine Meets Virtual Reality 21. Westwood J, Westwood SW, Felländer-Tsai, L, Fidopiastis CM, Haluck RS, Robb RA, Senger S, Vosburgh KG (ed): IOS Press, Fairfax, VA; 2014. 196:175–81. 10.3233/978-1-61499-375-9-175
  5. Shavit I, Keidan I, Hoffmann Y, Mishuk L, Rubin O, Ziv A, Steiner IP: Enhancing patient safety during pediatric sedation: the impact of simulation-based training of nonanesthesiologists. Arch Pediatr Adolesc Med. 2007, 161:740–43. 10.1001/archpedi.161.8.740
  6. Alaraj A, Lemole MG, Finkle JH, Yudkowsky R, Wallace A, Luciano C, Banerjee PP, Rizzi SH, Charbel FT: Virtual reality training in neurosurgery: Review of current status and future applications. Surg Neurol Int. 2011, 2:52. 10.4103/2152-7806.80117
  7. Loukas C, Nikiteas N, Kanakis M, Georgiou E: Evaluating the effectiveness of virtual reality simulation training in intravenous cannulation. Simul Healthc. 2011, 6:213–17. 10.1097/SIH.0b013e31821d08a9
  8. Rowe R, Cohen RA: An evaluation of a virtual reality airway simulator. Anesth Analg. 2002, 95:62–66. 10.1097/00000539-200207000-00011
  9. Shirai Y, Yoshida T, Shiraishi R, Okamoto T, Nakamura H, Harada T, Nishikawa J, Sakaida I: Prospective randomized study on the use of a computer-based endoscopic simulator for training in esophagogastroduodenoscopy. J Gastroenterol Hepatol. 2008, 23:1046–50. 10.1111/j.1440-1746.2008.05457.x
  10. Taffinder N, Sutton C, Fishwick RJ, McManus IC, Darzi A: Validation of virtual reality to teach and assess psychomotor skills in laparoscopic surgery: results from randomised controlled studies using the MIST VR laparoscopic simulator. Medicine Meets Virtual Reality. Westwood JD, Hoffman HM, Stredney D, Weghorst SJ (ed): IOS Press, Oxford; 1998. 50:124–30. 10.3233/978-1-60750-894-6-124
  11. Colt HG, Crawford SW, Galbraith O 3rd: Virtual reality bronchoscopy simulation: a revolution in procedural training. Chest. 2001, 120:1333–39. 10.1378/chest.120.4.1333
  12. Luigi Ingrassia P, Ragazzoni L, Carenzo L, Colombo D, Ripoll Gallardo A, Della Corte F: Virtual reality and live simulation: a comparison between two simulation tools for assessing mass casualty triage skills. Eur J Emerg Med. 2015, 22:121-27. 101097/MEJ?0000000000000132
  13. Medina LS, Racadio JM, Schwid HA: Computers in radiology. The sedation, analgesia, and contrast media computerized simulator: a new approach to train and evaluate radiologists' responses to critical incidents. Pediatr Radiol. 2000, 30:299–305. 10.1007/s002470050744
  14. Weiner E, McNew R, Trangenstein P, Gordon J: Using the virtual reality world of Second Life to teach nursing faculty simulation management. MEDINFO 2010: Proceedings of the 13th World Congress on Medical Informatics, Part 1. Safran C, Reti S, Marin HF (ed): IOS Press, CA; 2010. 160:615–19. 10.3233/978-1-60750-588-4-615
  15. Cook DA, Erwin PJ, Triola MM : Computerized virtual patients in health professions education: a systematic review and meta-analysis. Acad Med. 2010, 85:1589–602. 10.1097/ACM.0b013e3181edfe13
  16. Schinasi DA, Nadel FM, Hales R, Boswinkel JP, Donoghue AJ: Assessing pediatric residents' clinical performance in procedural sedation: a simulation-based needs assessment. Pediatr Emerg Care. 2013, 29:447–52. 10.1097/PEC.0b013e31828b6552
  17. Harris PA, Taylor R, Thielke R, Payne J, Gonzalez N, Conde JG: Research electronic data capture (REDCap)--a metadata-driven methodology and workflow process for providing translational research informatics support. J Biomed Inform. 2009, 42:377–81. 10.1016/j.jbi.2008.08.010
  18. Cohen D, Sevdalis N, Taylor D, Kerr K, Heys M, Willett K, Batrick N, Darzi A: Emergency preparedness in the 21st century: Training and preparation modules in virtual environments. Resuscitation. 2013, 84:78–84. 10.1016/j.resuscitation.2012.05.014
  19. Kidd LI, Knisley SJ, Morgan KI: Effectiveness of a Second Life® simulation as a teaching strategy for undergraduate mental health nursing students. J Psychosoc Nurs Ment Health Serv. 2012, 50:28–37. 10.3928/02793695-20120605-04
  20. Schmidt B, Stewart S: Implementing the virtual reality learning environment: Second Life. Nurse Educ. 2009, 34:152–55. 10.1097/NNE.0b013e3181aabbe8
  21. Goldmann K, Steinfeldt T : Acquisition of basic fiberoptic intubation skills with a virtual reality airway simulator. J Clin Anesth. 2006, 18:173–78. 10.1016/j.jclinane.2005.08.021
  22. Park J, MacRae H, Musselman LJ, Rossos P, Hamstra SJ, Wolman S, Reznick RK: Randomized controlled trial of virtual reality simulator training: transfer to live patients. Am J Surg. 2007, 194:205–11. 10.1016/j.amjsurg.2006.11.032
  23. Kundhal PS, Grantcharov TP: Psychomotor performance measured in a virtual environment correlates with technical skills in the operating room. Surg Endosc. 2009, 23:645–49. 10.1007/s00464-008-0043-5
  24. Andreatta PB, Maslowski E, Petty S, Shim W, Marsh M, Hall T, Stern S, Frankel J: Virtual reality triage training provides a viable solution for disaster-preparedness. Acad Emerg Med. 2010, 17:870–76. 10.1111/j.1553-2712.2010.00728.x
  25. Cone DC, Serra J, Kurland L: Comparison of the SALT andSmart triage systems using a virtual reality simulator with paramedic students. Eur J Emerg Med. 2011, 18:314–21. 10.1097/MEJ.0b013e328345d6fd
  26. Heinrichs WL, Youngblood P, Harter P, Kusumoto L, Dev P: Training healthcare personnel for mass-casualty incidents in a virtual emergency department: VED II. Prehosp Disaster Med. 2010, 25:424–32.
  27. Knight JF, Carley S, Tregunna B, Jarvis S, Smithies R, de Freitas S, Dunwell I, Mackway-Jones K: Serious gaming technology in major incident triage training: a pragmatic controlled trial. Resuscitation. 2010, 81:1175–79. 10.1016/j.resuscitation.2010.03.042
  28. Vincent DS, Sherstyuk A, Burgess L, Connolly KK: Teaching mass casualty triage skills using immersive three-dimensional virtual reality. Acad Emerg Med. 2008, 15:1160–65. 10.1111/j.1553-2712.2008.00191.x
  29. Nyssen AS, Larbuisson R, Janssens M, Pendeville P, Mayné A: A comparison of the training value of two types of anesthesia simulators: computer screen-based and mannequin-based simulators. Anesth Analg. 2002, 94:1560-65. 10.1097/00000539-200206000-00035

Community Discussion

Edward J. Rovera
February 09, 2016 at 08:41 AM
Edward J. Rovera

Given that the bulk of the sizable cost of developing the two scenarios has already be spent, it would be good to see this study replicated with a larger sample size.

Pavan P. Zaveri, M.D., Assistant Professor

Division of Emergency Medicine, Children's National Health System

For correspondence:
pzaveri@childrensnational.org

Aisha B. Davis

Division of Hospital Medicine, Children's National Health System

Karen J. O'Connell

Division of Emergency Medicine, Children's National Health System

Emily Willner

Division of Emergency Medicine, Children's National Health System

Dana A. Aronson Schinasi

Division of Emergency Medicine, Lurie Children's Hospital

Mary Ottolini

Division of Hospital Medicine, Children's National Health System

Pavan P. Zaveri, M.D., Assistant Professor

Division of Emergency Medicine, Children's National Health System

For correspondence:
pzaveri@childrensnational.org

Aisha B. Davis

Division of Hospital Medicine, Children's National Health System

Karen J. O'Connell

Division of Emergency Medicine, Children's National Health System

Emily Willner

Division of Emergency Medicine, Children's National Health System

Dana A. Aronson Schinasi

Division of Emergency Medicine, Lurie Children's Hospital

Mary Ottolini

Division of Hospital Medicine, Children's National Health System