A Fully Online Research Practicum Curriculum for Undergraduate Medical Students: A Protocol Paper

Introduction: Research is a critically important skill for medical trainees that helps physicians to advance the state of science and provide optimal healthcare to patients. The number of physician-scientists is decreasing. Competing priorities, limited research faculty, shrinking research budgets, and meeting accreditation standards limit the amount of time and effort needed to train undergraduate medical students sufficiently for them to engage in independent research activities. The purpose of this paper is to describe the development and implementation of a fully online research practicum as part of the medical school curriculum. Methods: A fully online research curriculum for preclinical medical students was developed and implemented by the researchers as a mandatory component of student training. This curriculum addressed research education challenges in medical schools with limited resources and/or for situations for which face-to-face research may be impractical or expensive. Students were also encouraged to practice critical thinking and critical appraisal of the evidence. Student-initiated research projects were conducted in teams and completed with support from faculty research mentors and librarians. During the first two cycles, 86 student team projects were completed. During the third cycle, 45 team projects were conducted. Results: Two complete cycles of a two-semester research practicum were completed. The majority of these studies were presented at regional, national, and international conferences, published as posters, and/or published as articles in peer-reviewed journals. Faculty mentor publications also increased as a result of engagement in the practicum. Discussion: The results of this research practicum provide evidence of the usefulness of a hands-on tailored research program delivered online for increasing research literacy, promoting interest among medical students in research, and motivation for future engagement in research as verified by student project outcomes during and beyond the practicum experience. The practicum can be used for individual students or as a team-based approach. A research practicum such as this also has the potential to improve the prospects of a successful application for post-graduate training, grants, and high-impact publications.


Introduction
Physician-scientists are MDs or DOs who undergo additional training in research and dedicate a significant portion of their time to pursue new knowledge about the delivery of patient care, disease, and health through research. Research is a critically important skill for medical trainees that helps physicians to advance the state of science and provide optimal healthcare to patients. Expediting physician-scientist training has thus become critical given the stagnation of clinical investigators in practice [1], regardless of the growing total number of physicians in practice, with the projected demand exceeding the expected supply [2][3][4].
Medical education strives to prepare students to practice medicine and provides the tools to do so successfully. Competing priorities, limited research faculty, shrinking research budgets, and meeting accreditation standards limit the amount of time and effort needed to train undergraduate medical students sufficiently for them to engage in independent research activities during graduate training and beyond. At most medical schools, a research "gap" year can be done after either the second or third year but taking the year off may not be logistically or economically feasible. Undergraduate research fellowships may also be available (the year between preclinical and clinical training) but are awarded to only a few students per class per year. These gaps delay graduation and separate the student from their known cohort support system.

Medical students' exposure to research
Undergraduate medical students are incorporated into research in a variety of ways with no apparent consistency. Students may participate in summer research electives (one-two months), mandatory coursework, and extracurricular research activities. Students may be reluctant to take time out for research, particularly if they are not receiving school credit.
Nonetheless, even students not interested in a research career will obtain skills transferable to clinical practice [5]. Something as cursory as writing a literature review can help students develop information literacy, library searching skills, critical thinking skills, and critical appraisal of evidence skills [6].
Another path for engagement in research is faculty-initiated studies. Students often "assist" the faculty in their ongoing research such as helping with lab work, writing up case reports, or conducting literature reviews for future papers. For several reasons, students may not be added as authors on publications related to projects they have worked on (e.g., they were not given substantive tasks that warranted authorship, the project was put on hold or abandoned by the faculty, the student moved into their clinical rotations, authorship was not negotiated up front). However, some medical schools do not have ample faculty with (1) expertise in research and/or (2) time to devote to student research mentoring.

Factors affecting medical student research
There are published reports that assessed the competencies, interests, barriers, and facilitators of research participation among undergraduate medical students [7][8][9][10][11][12][13][14][15][16][17][18][19][20]. Common barriers include lack of knowledge, lack of time (competing priorities), insufficient funding, and lack of mentoring (e.g., limited access to research mentors in their areas of interest). Students may lack the confidence and/or self-efficacy, and thus the inclination or motivation to conduct independent research while in medical school [7,[21][22][23][24][25]. These factors combined may define "research literacy" as a unique entity affecting a person's ability to conduct research. Research literacy in this aspect goes beyond memorization and recall of facts.
Another factor that impedes medical students from being involved in research includes not having good role models or mentors [7,26]. Moreover, matching the right mentor with a student is central to ensuring that students are not just providing assistantship in lab work or data collection, but rather providing an intellectual contribution to research projects [7,26].
The college's influence, including the type and length of research opportunities offered, may also be relevant factors regarding students' choice to engage in research. Additionally, ethical approval processes (whether in terms of time or paperwork) are often a chief difficulty facing faculty advisors and students alike. Moreover, the relatively short duration of the undergraduate research experience places limits on its publication or citation potential.

The rationale for the development of the research practicum
The number of students who report interest in research may be higher than that of students who are actually involved in research projects [14]. Reasons for this phenomenon may be due, in part, to the shortage of opportunities to meet students' interests, lack of funding, good mentorship, or inflexible curricula that leave little or no time for research [14,25,26]. The acknowledgment that barriers exist combined with students' desire to learn research led to the creation of a curriculum-based, comprehensive research training for preclinical undergraduate medical students that is delivered fully online.
There is a demand for a unique balance between scientific inquiry and outstanding clinical training. With this in mind, the research practicum (herein referred to as the "Practicum") contains activities to transform the manner in which students learn and prepare for careers in academic medicine or clinical practice in which scientific investigation is involved. These innovative efforts in any medical education curriculum can provide students with a more unique and flexible educational experience and accelerate the preparation of physician-scientists to become leaders in biomedical research.
In this paper, the development and implementation of a Practicum for all second-year students enrolled in the Nova Southeastern University Dr. Kiran C. Patel College of Osteopathic Medicine (NSU-KPCOM) Research Practicum are described. These activities include a series of didactic lectures combined with a mentor-driven, structured yet flexible Practicum to promote students' research literacy and promote critical thinking skills. The Practicum can be done exclusively online using internet-based platforms typically utilized in modern medical schools such as Canvas and Zoom. Further, the Practicum does not incur any costs from students, faculty, or medical school administrators (aside from faculty percent effort) and thus no extracurricular budget is required. No textbooks are required.
To our knowledge, the NSU-KPCOM Research Practicum is the only one of its kind to provide a mentordriven, comprehensive, and tailored research program delivered fully online for increasing research literacy, promoting interest among medical students in research, yielding publishable products, and increasing motivation for future engagement in research. The Practicum can be used for individual students or as a team-based approach. A Practicum such as this also has the potential to improve prospects of a successful application for post-graduate training, grants, and high-impact publications.

Program overview
In August 2020, the inaugural Practicum was delivered fully online as part of mandatory curricula programming for second-year students enrolled in the NSU-KPCOM. The Practicum provided mentordriven, structured, interactive tailored training with student access to free statistical and data management software. All students participated in team research projects with an average of 8-10 students per team. Approximately 800 students have successfully completed the Practicum between 2020 and 2022.

The practicum protocol
A bottom-up approach to student participation in research (e.g., student-run, faculty-facilitated student research teams) is embraced by providing opportunities for research projects with a continued focus on sustained mentorship that includes advancing communication and clinical skills as well as professionalism -in tandem with their academic studies. All second-year students are required, as part of the medical school program, to enroll in the Practicum that (1) involves students in the practicalities of research, from planning to execution, by providing opportunities for them to participate in team-based research projects; (2) incorporates the teaching of key research methodologies learned in didactic courses provided in the curriculum for students to apply to tangible team-based research projects; (3) facilitates the generation of new research questions by students; (4) provides research mentors from the college and the wider research community at the university or affiliated hospitals to help facilitate their research activities; and (5) supports students to present and publish their research.
The Practicum addresses key elements in designing and conducting health and epidemiological research, including (1) understanding research theory and practice; (2) evaluating research through reading, reviewing, summarizing, and assessment; (3) formulating good research questions; (4) conducting literature reviews; (5) selecting research approaches; (6) documenting, presenting, and writing the research topic and results; (7) exploring ethical issues in research design (plagiarism, human subject ethics, interpreting results); and (8) conducting a team research project.
The Practicum helps develop students' abilities by increasing their research literacy, i.e., the capacity to obtain, process, and understand basic information needed to make informed decisions about conducting research. Research literacy in this aspect goes beyond memorization and recall of facts. By involving students in the logistics of research, from planning to implementation, their scientific acumen as well as their self-efficacy increases. In this way, by promoting research readiness in students transitioning to their clinical phase of training, graduates can begin residency, a generally stressful time, feeling confident if not masterful in their ability to conduct research.
The student-initiated research projects were conducted in teams with guidance and support from faculty mentors. The faculty mentors were particularly instrumental in promoting the academic medicine mission of training the next generation of physicians to engage in research to inform advancement and innovation in healthcare delivery. Due to specific circumstances, however (e.g., large class size, pandemics), the course directors needed to adapt the Practicum in the following ways.

Large Class Sizes
While small student research teams may be optimal (2-5 students), the medical college has approximately 400 students in each cohort and encompasses two campuses in Florida nearly 300 miles apart (i.e., Fort Lauderdale and Clearwater). Due to the large student body, students carried out team science with a faculty research mentor, with each group consisting of 3-10 students.

COVID-19 Pandemic
While the onset of the COVID-19 pandemic offered new opportunities for research, the logistical challenges in complying with coronavirus safety restrictions called for creative modifications of proposed research projects. What might have been a study conducted with patients face to face became one that utilized Internet-based technology such as Zoom for interviews, REDCap data management software for online surveys, and Rayyan systematic review collaboration software.

Flexible Design Based on Student Preferences
Based on student feedback, the curriculum was revised to teach only systemized review studies (e.g., systematic, meta-analysis, and scoping reviews) and is the focus of this protocol. These adaptations yielded excellent results while simultaneously highlighting the importance of critical thinking, flexibility, and creativity in scientific endeavors. This gave students more time to work on projects without having to wait for IRB approval and to finalize data collection, which could take several months. Moreover, data collection efforts sometimes yielded insufficient sample sizes leading to non-publishable papers. Lastly, there was a scarcity of faculty members with statistical analysis training to help analyze the results. The resulting change left more time to focus on publications of students' work before the Practicum ended.

Getting Started with Faculty
After faculty research mentors were identified, they attended an online synchronous orientation session that explained the goals and objectives of the Practicum, expectations of mentors, course content, student assignments, grading procedures, and a tour of the online learning platform (e.g., Canvas). Mentors were expected to attend all synchronous student sessions, but all sessions were recorded for future viewing for mentors who were unable to attend.

Librarians
Finding good, reliable research sources is often challenging in academic health professions settings. Librarians provided instrumental help to students, particularly in developing a search strategy for systemized reviews, working with reference management software packages (e.g., EndNote), and citing good sources for their introductory literature reviews.

Statistical Consulting
For the meta-analysis, faculty members with statistical expertise were utilized as secondary mentors for projects in which the primary mentor did not have quantitative statistical skills.

Administrative
The medical college administration offers the use of (at no charge to students or faculty) Rayyan (https://www.rayyan.ai/), an online software platform to organize, manage, and accelerate collaborative systematic and scoping reviews, and EndNote, a reference manager.

The curriculum
The Practicum outlines the fundamentals of conducting research for medical students at the preclinical level. The sessions include the following general topic areas related to conducting systemized review studies (e.g., scoping reviews, systematic reviews, meta-analyses): choosing a research topic, developing a good research question, writing a literature review, developing a research protocol, collecting data, conducting the analysis, reporting the results, writing the discussion, scientific writing skills (abstracts, posters, manuscripts), and using research tools (Rayyan, EndNote referencing manager, online library databases).

Getting Started with Students
In Canvas, a module labeled "Start Here" was created for students. Here is where students found dates and links to synchronous Zoom sessions (six total per Practicum), research faculty mentor bios with their research experience and interests, assignments and grading system, grading rubrics, the syllabus, information on plagiarism and academic misconduct, faculty virtual office hours, and other logistical information. Students were instructed to self-assign to pre-constructed groups in Canvas based on their research interests and mentor preference; each group was labeled by mentor's name and had a limit on the number of students who could self-assign predetermined by the course directors in coordination with the mentors.

Getting Started with Mentors
A mandatory orientation session with mentors before the start of the Practicum was accompanied by a Canvas module designed to offer training and resources for mentors and was appropriately labeled "Mentor Center." The module was visible only to the faculty research mentors. Sample topics included: helping students define their review question and inclusion criteria, Rayyan video tutorial, navigating the Practicum using an online learning platform (i.e., Canvas, Blackboard), course requirements, how to grade group assignments, and grading rubrics. Other logistical information (e.g., assignments and due dates, student emails by group) were also provided as well as PowerPoint presentations and recordings from synchronous Practicum sessions with all students. The Mentor Center was updated periodically as needed.

Sample learning modules for meta-analysis, systematic, and scoping reviews
This curriculum was conducted over two semesters (15 weeks each semester). Each module covered 2-3 weeks of material, with each week covering a different topic that builds on itself. At the end of each module is an exercise (graded as a group assignment) focused on completing the steps for conducting the review study. To ensure engagement on the individual level, there was an article screen assignment in Rayyan and a series of quizzes throughout the semester that tested students' critical thinking skills related to research methods and appraisal of the scientific literature. Students were also given access to plagiarism detection software to use before submitting assignments if applicable. Grading rubrics were developed and made accessible so both students and mentors understood the expectations for all deliverables. Table 1 delineates the outline of modules pertaining to conducting systematic/scoping reviews and meta-analysis studies.

Module
Main Topic

Module Objectives
A concise list of the learning objectives of the module that are reflected in the information and activities throughout the module.

Asynchronous Virtual Seminar
This page included the module's learning objectives and information in written form on the module's topic. It included a module overview and week-by-week detailed explanations, relevant attachments, definitions, examples, and other information pertinent to the module topic. This was an economic and effective instructional tool for teaching students who are at a distance from their instructor.

Supplemental Learning Materials
Included in each module were video tutorials (from reputable lectures found online or pre-recorded lessons developed by the course director, links to relevant websites), annotated PowerPoint presentations, and relevant scientific articles.

Links to Assignments
Links to the module's quiz and assignment (including examples of completed assignments) were included in each module.

Communication
An online discussion forum labeled "Ask Your Professor" was available for students to post questions pertaining to the corresponding module. Students were also provided the course directors' emails and an online scheduling assistant, Calendly, with a Zoom meeting link to set up appointments with the directors (individually or as a group).

Evaluations
At the midpoint and conclusion of the Practicum, students completed a survey evaluating their mentor. Feedback provided by students was instrumental in evaluating mentor performance and engagement. Mentors who were not available to meet with their research team(s) regularly or who did not provide adequate guidance were replaced (after attempts at remediation) and not asked to serve the following year. Course evaluations were conducted at the end of the Practicum (end of the second semester) via the institution in which students provided quantitative and qualitative feedback regarding the course directors and the course itself.

Results
The Practicum cultivated engagement in research throughout the second year of medical school by integrating a curriculum that provides not only education in research methodology, but tangible and tailored opportunities for student research. It was cost-effective, with the allocation of faculty effort as the only financial expense.
Student team projects from years 2020-2022 consisted of collecting data through questionnaires and interviews, conducting quasi-experiments, preparing systematic and scoping reviews, retrospective medical chart reviews, and analyzing large healthcare data sets. These adaptations yielded excellent results while simultaneously highlighting the importance of critical thinking, flexibility, and creativity in scientific endeavors.  The student projects were disseminated as published abstracts, published posters, articles in peer-reviewed scientific journals, international conference proceedings, and presentations at regional and national scientific conferences. Many of the posters were presented at local or school-initiated student poster competitions, with several taking away awards. For each publication, faculty mentors were listed as senior (last) authors, increasing faculty research output in addition to teaching load or service (i.e., student mentoring or advising) to the college.

Discussion
The Practicum incorporated the teaching of key research methodologies into the curriculum and emphasizes the importance of evidence-based medicine (EBM) and the need to broach new questions. Expediting physician-scientist training such as this might help address the stagnation of physician-scientists in the face of the growing total number of clinically oriented physicians who practice [1][2][3][4]. The platform described in this paper involved students in the practicalities of research, from planning to execution. As with all new educational programs, strengths were acknowledged, and lessons were learned.

More Opportunities for Collaboration
There were multiple advantages to a fully online research program. It provided many opportunities for students to participate in research within their undergraduate curriculum. Students from different regional campuses from the same school seamlessly collaborated on team projects. Research mentors from outside the college or university were enlisted.

Fiscal/Logistical Advantages
There was no need to schedule classroom time, and money was saved on overhead expenses. Additional expenses, such as transportation costs, were eliminated for both mentors and students. Also, no textbooks were required as all learning materials were provided online.

Flexible
The practicum was implemented during the second year of medical school but can be implemented for firstyear students, depending on the particular school's preclinical curriculum. Students had online access to the learning materials and mentors during off hours. The online Practicum was convenient, offered flexibility, and provided more individual attention.

Skills Acquired Through the Practicum Experience
Students developed research self-efficacy and confidence to conduct autonomous research [23]. The Practicum promoted time-management skills, making it easier to achieve optimal school-life balance. As a fully online program with minimal synchronous meetings with faculty, it helped students develop selfmotivation and self-discipline while promoting teamwork and providing maximum flexibility for students to simultaneously complete their coursework.

Student Research Is Supported
The Practicum exposed students to the wider research community at the university. It encouraged them to appraise and respond to current research. Many published studies have reported on the importance of supporting student research in undergraduate medical school programs [7,15,17,23].
The Practicum aimed to support students in presenting and publishing their research, highlighting the potential career pathways within academic medicine. Students obtained skills such as writing a literature review and manuscript as well as library research skills [6].
would not allow the student to complete the preclinical requirements and thus could not advance to the clinical phase of their training (i.e., rotations).
To help ensure sustainability, the following issues need to be considered and several commitments from the administration need to be ensured. First, course directors must be flexible due to the competing interests of medical students (e.g., other course exams, natural disasters). Directors must also periodically check the mentors' work for quality assurance of projects they are mentoring. Second, the school must be able to provide capable mentors in numbers that promote optimal ratios of mentors to students. Mentors must have time allotted and sanctioned by the administration, as much of the work occurs outside of synchronous class sessions. Mentor to student ratio may not be ideal in large classes or schools with few research faculty. Third, the school infrastructure, including technology provisions, must provide support in the way of free software for students (e.g., SPSS, REDCap), librarians, statistical faculty/staff, and an amendable IRB. Fourth, the school should provide sufficient venues (e.g., school-initiated conferences) for students to present their work. Ideally, the school should provide research funding to students for conference attendance and for students to present and/or publish their work in open-access journals, though this is not always feasible with the competing demands and strains on present-day college budgets. Last, offering students mentoring in different types of research (e.g., surveys, quasi-experiments, secondary data studies, and systemized reviews), as we had done with the first cohort, is time intensive. It requires creating additional learning materials and tailored didactic lessons for each type of study. Moreover, mentors with expertise in these research areas need to be accessible.

Implications for medical education
It is important to encourage participation in research throughout the preclinical years by integrating a Practicum that offers education and opportunities for research. When students understand the practicalities of research, they can feel more confident in their abilities [22,25] which could in turn promote research activity among medical students. Medical school faculty members should promote the advancement of the scholastic medicine mission of preparing future physicians to engage in research. This will help ensure advancement and innovation in healthcare delivery. Cultivating medical students' engagement in research early on in their training should be a major goal for medical educators. There is a place for input across the course of the medical degree and that stepwise encouragement that is tailored to the interests, time resources, and career aspirations of the student that we believe will prove most beneficial.

Conclusions
Undergraduate medical education strives to instill the competencies required for graduates to deliver safe, effective, and high-quality patient care. However, competing priorities, limited research faculty, and meeting accreditation standards have traditionally limited the amount of time and effort allocated to train medical students sufficiently for them to engage in research activities. As board exams have transitioned to pass/fail, student research is likely to become an increasingly important component of well-rounded medical student portfolios to increase their chance of successfully matching to competitive residency programs. One way to foster this process is to implement a fully online Practicum that is economical and easy to administrate such as the one described here.

Additional Information Disclosures
Human subjects: All authors have confirmed that this study did not involve human participants or tissue. Animal subjects: All authors have confirmed that this study did not involve animal subjects or tissue.

Conflicts of interest:
In compliance with the ICMJE uniform disclosure form, all authors declare the following: Payment/services info: All authors have declared that no financial support was received from any organization for the submitted work. Financial relationships: All authors have declared that they have no financial relationships at present or within the previous three years with any organizations that might have an interest in the submitted work. Other relationships: All authors have declared that there are no other relationships or activities that could appear to have influenced the submitted work.