In recent years online educational content, efforts at quality appraisal, and integration of online material into institutional teaching initiatives have increased. However, medical education has yet to develop large-scale online learning centers. Crowd-sourced curriculum development may expedite the realization of this potential while providing opportunities for innovation and scholarship. This article describes the current landscape, best practices, and future directions for crowdsourced curriculum development using Kern’s framework for curriculum development and the example topic of core content in emergency medicine.
A scoping review of online educational content was performed by a panel of subject area experts for each step in Kern’s framework. Best practices and recommendations for future development for each step were established by the same panel using a modified nominal group consensus process.
The most prevalent curriculum design steps were (1) educational content and (2) needs assessments. Identified areas of potential innovation within these steps included targeting gaps in specific content areas and developing underrepresented instructional methods. Steps in curriculum development without significant representation included (1) articulation of goals and objectives and (2) tools for curricular evaluation.
By leveraging the power of the community, crowd-sourced curriculum development offers a mechanism to diffuse the burden associated with creating comprehensive online learning centers. There is fertile ground for innovation and scholarship in each step along the continuum of curriculum development. Realization of this paradigm’s full potential will require individual developers to strongly consider how their contributions will align with the work of others.
Introduction & Background
Online education is rapidly expanding across many disciplines, and graduate medical education is no exception [1-6]. Disruptive innovations such as massive open online courses have been heralded as pillars in a future paradigm of collaborative online learning for medical education . While in recent years we have seen an increase in online educational content, quality appraisal initiatives [8-13], and integration of online material into institutional teaching initiatives [9, 14], medical education remains far from realizing large-scale online learning centers such as those described by Mehta, et al. .
One clear barrier to making this vision a reality is the enormous amount of work involved in developing such comprehensive resources. While many have succeeded in creating a wide variety of high-quality online educational resources, no single or curated collection of open-access resources boasts the scope to cover an entire subject, specialty, or discipline. Strides are being made in both the creation and curation of content, however, and the potential for this type of comprehensive resource is real [9, 14, 15]. While this progress is encouraging, it is important to note that educational content is only one piece of the puzzle and another less discussed but significant barrier remains: the integration of educational content into a purposeful curriculum.
Rigorous curriculum design is not foreign to the realm of open-access online education; there are many examples of successful small-scale curricular interventions that have used this platform [16-19]. However, examples of large-scale curricular interventions are lacking. Despite the lack of current examples, evidence of a movement toward the formation of such resources is mounting as multiple independent sources develop components of traditional curriculum design for large-scale online curricula [9, 14, 20, 21].
With multiple individuals contributing to curriculum development, the education community stands to benefit from relief of the burden of completing each curriculum design step alone. This “crowdsourced curriculum development” is an exciting prospect for medical educators interested in robust online curriculum design and provides opportunities for scholarship throughout the continuum of curriculum creation. In this article, through a review of online medical education resources and consensus process for establishing recommendations, we will illustrate current examples and future directions for crowdsourced curriculum development using Kern’s framework and the topic of core content in emergency medicine .
The scoping review process was conducted by an expert panel composed of the authors, each of whom possesses knowledge and experience in the realm of online medical education. Each step in Kern’s framework (problem identification, needs assessment, goals and objectives, educational strategies, implementation, measuring outcomes) was explored individually. The search process was conducted from September to November of 2016. All authors participated in the search process. The search included a scientific database (PubMed) and a general search engine (Google). Search terms including the titles of each of Kern's steps, 'emergency medicine,' 'intern,' 'curriculum,' 'core content,' 'FOAM,' and 'FOAMed.' In addition, resources known to the authors and those identified by snowball sampling were included.
A modified nominal group consensus process was used to determine the final list of resources and recommendations for further development of each step. The consensus process continued until all authors agreed on the final list of resources to be included. This study did not involve human subjects.
Step 1: Problem identification
In Kern’s model, the “problem” to be identified may be thought of as the difference between the way something is currently taught and the ideal solution .
“The better a problem is defined, the easier it will be to design an appropriate curriculum to address the problem.” (Kern, p. 11)
This process of problem analysis is also referred to as a general needs assessment.
In the example of a curriculum on core content for emergency medicine, the most common model relies on didactic sessions, required textbook readings, and on-shift teaching. These traditional education methods have well-described weaknesses: didactic sessions suffer from variable attendance (due to off-service rotations and other clinical and wellness obligations) , millennial residents have largely eschewed the textbook [21, 24, 25], and on-shift teaching is highly dependent on patient pathology and lacks the consistency to assure a suitable foundation in core concepts [26-28]. These limitations could be addressed with an online curriculum that is always available, utilizes educational methods that the learners value, and is designed with the rigor and consistency required to assure a suitable foundation in core concepts.
Educators should explicitly define the problems that educational interventions are designed to address. This step is largely under-represented in the paradigm of large-scale online medical education. Educators interested in advancing this paradigm may begin by clearly articulating a problem that is used to align stakeholders and assimilate resources.
Step 2: Targeted needs assessment
The targeted needs assessment begins with the identification of target learners (defined as “the group most likely, with further learning, to contribute to the solution of the problem” ), followed by an evaluation of the specific needs of these learners relative to the identified problem.
In our example of a curriculum for core content in emergency medicine, the target population would include emergency medicine residents and senior medical students bound for a career in emergency medicine. Those responsible for emergency medicine training programs and certifying bodies would be crucial stakeholders to include in the targeted needs assessment. To assess the needs of this broad and diverse population, multiple assessments for each stakeholder group may be required.
There are several existing examples of targeted needs assessments that align with a web-based curriculum on core content in emergency medicine outlined in Table 1.
Existing targeted needs assessments may be applicable to large-scale online medical education; however, confounding factors in the translation of existing evidence to this new paradigm must be addressed. Formulation of a large-scale needs assessment (either de novo or through assimilation and adaptation of existing resources) would meaningfully contribute to the realization of this new paradigm. Once large-scale needs assessments are conducted, local education stakeholders may wish to conduct additional needs assessments at the program level to determine whether the curriculum appropriately addresses the needs of local learners.
Step 3: Goals and objectives
Goals provide a broad overview of the intent of the curriculum. Objectives provide specific and measurable educational directives that build toward the broad educational goal of the curriculum. These constructs communicate the purpose and proper usage of the curriculum, in addition to informing the design of assessments (see Step 6). The importance of this step is reflected in the Accreditation Council for Graduate Medical Education (ACGME) common program requirement that curricula include “Competency-based goals and objectives for each assignment at each educational level” .
Unfortunately, despite the large amount of quality online educational content, rarely are resources accompanied by goals and objectives that clarify the desired outcomes from use of the resource. We identified two online resources that consistently outline goals and objectives for online core content in emergency medicine: EM Fundamentals  and the Clerkship Directors in Emergency Medicine (CDEM) website .
Content generated without a clear measure of learner success is of limited use to instructors and leaves self-assessing learners directionless. Clearly outlined goals and objectives provide transparency across stakeholder groups and a benchmark for measurement of learners.
Writing clear objectives can be a difficult task for curriculum designers. We suggest using the SMART framework to assist when designing educational objectives . These criteria for objectives were originally described in business literature and have been adapted to medical education (Table 2). When using the SMART framework, all objectives must hold these four elements: 1) “Who will do,” 2) “How much or how well,” 3) “Of what” and 4) “By when?”
Goals and objectives are intimately linked to educational strategies and assessment, therefore developing these steps in parallel will help ensure alignment. Existing content need not be overhauled by originators to include this language; goals and objectives for high-quality content can be crowdsourced. Mapping existing content to nationally recognized goals and objectives (e.g., those of specialty organizations, ACGME competencies and/or milestones) will likely be more interesting to outside stakeholders than retrofitting content to custom goals and objectives.
Step 4: Educational strategies
Educational strategies are the vehicle for learner progression. Multiple methods may be used to reinforce a single concept or to target different levels of objectives within a single curriculum.
To date, the ad hoc nature of online content production has, unsurprisingly, led to an imbalance in coverage of various topics . However, a small but growing number of resources have focused on foundational knowledge (Table 3).
These foundationally-focused resources are mostly text- and video-based, which are reasonable methods for objectives pertaining to basic knowledge. While most blogs permit some level of interaction through comments and responses, objectives related to higher-level functions (such as application and analysis) may be better targeted using social media tools as forums for discussion (e.g., Twitter, Slack, Google hangouts, Reddit). If classroom time is available, higher-level functions may also be targeted by using a blended or flipped classroom approach, leveraging online content for preparation and interactive small group activities for the live sessions.
While online educational content is arguably the most developed step in Kern’s framework at present, significant gaps still exist. Educators motivated to advance this area should focus on underrepresented content areas and/or underrepresented methods of instruction (e.g., interactive online experiences).
Step 5: Implementation
The implementation phase focuses on the logistics of resource development, deployment, and sustainability.
There are two ways in which educators deliver online content: (1) contributing to an existing resource and (2) implementing a new online resource. The simplest way to deliver content is by contributing to an existing resource with an established product and reputation. This process is akin to journal submissions: many existing web resources will review submitted content and, if deemed to meet an appropriate standard, publish the submission. Existing resources that allow third-party submissions include CanadiEM , EMDocs , EMSimCases , and R.E.B.E.L. EM .
There are also many formats for deployment of new online resources, each with variable start-up costs and diverse hosting options. For podcasts, start-up costs may include a microphone, audio processing software, and a hosting service (e.g., Libsyn, Soundcloud). For video creation, costs may include a camera and video processing software. Free video hosting services include YouTube and Vimeo. For websites, costs may include hosting fees, web development software, and potentially hiring professional web developers. There are also many options for web hosts, some of which are free (e.g., Wordpress.com, Wix, Blogger). Each of these steps also carries non-financial requirements (e.g., basic technical skills).
In addition to start-up costs, financial support to enable upkeep will be necessary. Currently used models for ongoing financial support include local and national organizational funding [34, 40, 41], ad revenue, the freemium model , and subscription services .
Creating a new web resource can require significant investments of time and money. New resources will also need to build a reputation with learners. When starting out, consider modeling after a known successful resource or contributing to an existing host in order to conserve resources while gaining experience. New resources should include a sustainability plan that addresses sources of funding for continuing costs.
Step 6: Measuring outcomes (assessment and evaluation)
Assessment of learners and evaluation of usefulness are the methods by which outcomes data pertaining to the curriculum are obtained. Without robust mechanisms for assessment and evaluation, educators will lack meaningful information regarding the successes or shortcomings of the educational intervention. Data from these steps may be used to highlight areas for improvement in the iterative process of curriculum development.
Example: Assessment of Learning
Arguably the most robust online resource for assessment of learners on core content in emergency medicine is the suite developed by Academic Life in Emergency Medicine (ALiEM) . These include ALiEMU, ALiEM Capsules, ALiEM AIR, and ALiEM AIR PRO, which track the learner’s progress using multiple-choice quizzes. ALiEM also hosts the Medical Education in Cases (MEdIC) series, in which participants contribute to an open discussion forum via blog comments . This format allows instructors to monitor uptake and understanding of a topic. An existing tool for self-assessment is the flashcard bank developed by the CRACKCast podcast . Finally, there are several online resources that could be used to facilitate learner assessment at the local level. EM Sim Cases is an online repository of peer-reviewed simulation cases that are presented in a standardized format , and the McMaster Modular Assessment Program (McMAP) has three e-books that contain numerous competency-based assessment tools . Online question banks may also be used locally as readiness assurance tests in the flipped-classroom model.
Example: Evaluation of Usefulness
Our search of existing resources has found that robust evaluation tools are lacking. Most online resources provide contact information for the authors that may be used to submit feedback as a form of evaluation, however, this mechanism is inefficient and unreliable. Participation as a proxy for usefulness, however, can be easily quantified in the online environment. For websites, it is possible to embed code that measures usage with metrics such as page views, unique users, and time the users spent on a specific page. For podcasts, it is possible to track analytics such as number of downloads and subscribers. An amalgam of metrics has been developed in an effort to measure website impact . It must be noted, however, that these metrics only measure engagement with the resource and are limited in their ability to truly measure usefulness.
For the online environment, we propose educators use an adaptation of Kirkpatrick’s framework for program evaluation (Figure 1) . The dearth of robust mechanisms for formal evaluation of online resources as curricular tools makes this another area ripe for innovation and scholarship. Table 4 outlines recommendations for gathering and communicating evaluation data in the absence of more robust intrinsic evaluation tools.
Development of comprehensive online learning experiences is a daunting task. This approach has been successful in other fields, however, and medical education stands to similarly benefit from this model. Crowdsourced curriculum development offers a mechanism to diffuse the burden associated with realization of this promising new paradigm. While this article has used the topic example of core content in emergency medicine to review the state of online resources relative to Kern’s framework, the same principles can be applied to any number of topics. As described above, this method of curriculum design begins with the notion that not all scholarly contributions to the field need to include new educational resources. Rather, there is fertile ground for novel scholarship in each step along the continuum of curriculum development and in curating existing resources to align with a specific educational initiative.
We note that some of Kern’s steps in curriculum development are already well-represented, including educational content. Many existing problems and needs assessments may align with these existing resources. Innovation in this area is still possible and may target gaps in content or instructional methods, or perhaps synthesize existing material to note or address a specific need.
Underrepresented steps in Kern’s framework include the articulation of goals and objectives and tools for curricular evaluation. Each of these areas is clear targets for educators interested in contributing scholarship to advance this new paradigm.
This study limited focus to the example topic of core content in emergency medicine. It is possible that the existing landscape in other content areas varies significantly; however, the approach to curriculum development in these areas would remain the same. Also, while this study used multiple modalities to identify content relevant to each of the highlighted areas (e.g., scholarly search engines, general search engines, and snowball sampling), it is possible that some relevant resources that are earlier in their development were still not discovered. As these resources mature and gain prominence, they may play important roles in the framework described.
By leveraging the power of the community, crowd-sourced curriculum development offers a mechanism to diffuse the burden associated with creating comprehensive online learning centers. There is fertile ground for innovation and scholarship in each step along the continuum of curriculum development. Realization of this paradigm’s full potential will require individual developers to consider how their contributions will align with the work of others to be most impactful for the community.
- Khan S: The One World Schoolhouse: Education Reimagined. Hodder & Stoughton, London, England; 2012.
- Radford AW: Learning at a distance: undergraduate enrollment in distance education courses and degree programs. United States Department of Education. 2011, Accessed: November 22, 2017: https://nces.ed.gov/pubs2012/2012154.pdf.
- Mallin M, Schlein S, Doctor S, et al.: A survey of the current utilization of asynchronous education among emergency medicine residents in the United States. Acad Med. 2014, 89:598–601. 10.1097/ACM.0000000000000170
- Purdy E, Thoma B, Bednarczyk J, et al.: The use of free online educational resources by Canadian emergency medicine residents and program directors. CJEM. 2015, 17:101-106. 10.1017/cem.2014.73
- Folkl A, Chan T, Blau E: Use of free, open access medical education and perceived emergency medicine educational needs among rural physicians in Southwestern Ontario. Cureus. 2016, 8:796. 10.7759/cureus.796
- Thurtle N, Banks C, Cox M, et al.: Free open access medical education resource knowledge and utilisation amongst emergency medicine trainees: a survey in four countries. African J Emerg Med. 2016, 6:12–17. 10.1016/j.afjem.2015.10.005
- Mehta NB, Hull AL, Young JB, et al.: Just imagine: new paradigms for medical education. Acad Med. 2013, 88:1418–1423. 10.1097/ACM.0b013e3182a36a07
- Lin M, Thoma B, Trueger NS, et al.: Quality indicators for blogs and podcasts used in medical education: modified Delphi consensus recommendations by an international cohort of health professions educators. Postgrad Med J. 2015, 91:546–550. 10.1136/postgradmedj-2014-133230
- Lin M, Joshi N, Grock A, et al.: Approved instructional resources series: a national initiative to identify quality emergency medicine blog and podcast content for resident education. J Grad Med Educ. 2016, 8:219–225. 10.4300/JGME-D-15-00388.1
- Cadogan M, Thoma B, Chan TM, et al.: Free open access meducation (FOAM): the rise of emergency medicine and critical care blogs and podcasts (2002-2013). Emerg Med J. 2014, 31:76–77. 10.1136/emermed-2013-203502
- Sidalak D, Purdy E, Luckett-Gatopoulos S, et al.: Coached peer review: developing the next generation of authors. Acad Med. 2017, 92:201-204. 10.1097/ACM.0000000000001224
- Thoma B, Chan T, Desouza N, et al.: Implementing peer review at an emergency medicine blog: bridging the gap between educators and clinical experts. CJEM. 2015, 17:188–191. 10.2310/8000.2014.141393
- Chan TM, Thoma B, Krishnan K, et al.: Derivation of two critical appraisal scores for trainees to evaluate online educational resources: a METRIQ study. West J Emerg Med. 2016, 17:574-584. 10.5811/westjem.2016.6.30825
- Shappell E, Ahn J: IDEA series: a novel flipped-classroom approach to intern conference education featuring EM fundamentals. Academic Life in Emergency Medicine. 2016, Accessed: November 22, 2017: https://www.aliem.com/2016/idea-novel-flipped-classroom-approach-intern-conference-education/.
- New advanced “AIR Professional” series: AIR-Pro. (2015). Accessed: November 22, 2017: https://www.aliem.com/2015/09/new-advanced-air-profesional-series-air-pro-series/.
- Lewin LO, Singh M, Bateman BL, et al.: Improving education in primary care: development of an online curriculum using the blended learning model. BMC Med Educ. 2009, 9:33. 10.1186/1472-6920-9-33
- Satterwhite T, Son J, Carey J, et al.: Microsurgery education in residency training: validating an online curriculum. Ann Plast Surg. 2012, 68:410–414. 10.1097/SAP.0b013e31823b6a1a
- Metcalf MP, Tanner TB, Buchanan A: Effectiveness of an online curriculum for medical students on genetics, genetic testing and counseling. Med Educ Online. 2010, 15:4856. 10.3402/meo.v15i0.4856
- Lindeman BM, Law JK, Lipsett PA, et al.: A blended online curriculum in the basic surgery clerkship: a pilot study. Am J Surg. 2015, 209:145–151. 10.1016/j.amjsurg.2014.10.003
- Stuntz R, Clontz R: An evaluation of emergency medicine core content covered by free open access medical education resources. Ann Emerg Med. 2016, 67:649–653. 10.1016/j.annemergmed.2015.12.020
- Shappell E, Ahn J: A needs assessment for a longitudinal emergency medicine intern curriculum. West J Emerg Med. 2017, 18:31–34. 10.5811/westjem.2016.9.31493
- Kern DE, Thomas PA, Howard DM: Curriculum Development for Medical Education: A Six-Step Approach. Johns Hopkins University Press, Baltimore, MD; 2009.
- Austin PE, Brown CK, Dunn KA: Conference attendance: do we meet the new residency review committee requirements?. Ann Emerg Med. 1995, 25:325–327. 10.1016/S0196-0644(95)70288-1
- Twenge JM: Generational changes and their impact in the classroom: teaching Generation Me. Med Educ. 2009, 43:398–405. 10.1111/j.1365-2923.2009.03310.x
- Roberts DH, Newman LR, Schwartzstein RM: Twelve tips for facilitating Millennials' learning. Med Teach. 2012, 34:274–278. 10.3109/0142159X.2011.613498
- Langdorf MI, Montague BJ, Bearie B, et al.: Quantification of procedures and resuscitations in an emergency medicine residency. J Emerg Med. 1998, 16:121–127. 10.1016/S0736-4679(97)00252-7
- Langdorf MI, Strange G, Macneil P: Computerized tracking of emergency medicine resident clinical experience. Ann Emerg Med. 1990, 19:764–773. 10.1016/S0196-0644(05)81700-7
- Deveau JP, Lorenz JE, Hughes MJ: Emergency medicine resident work productivity and procedural accomplishment. J Am Osteopath Assoc. 2003, 103:291–296.
- M4 curriculum. (2017). Accessed: November 22, 2017: https://cdemcurriculum.com/m4/.
- Manthey DE, Ander DS, Gordon DC, et al.: Emergency medicine clerkship curriculum: an update and revision. Acad Emerg Med. 2010, 17:638-643. 10.1111/j.1553-2712.2010.00750.x
- Kessler CS, Tolia V, Singh N: Targeted needs assessment of off-service residents in emergency medicine. West J Emerg Med. 2010, 11:470-473.
- Bleeding and clotting: a SoMe-based needs assessment. (2016). Accessed: November 22, 2017: https://canadiem.org/bleeding-clotting-based-needs-assessment/.
- ACGME common program requirements. (2017). Accessed: November 22, 2017: http://www.acgme.org/Portals/0/PFAssets/ProgramRequirements/CPRs_2017-07-01.pdf.
- EM fundamentals. (2017). Accessed: November 22, 2017: http://emfundamentals.com/.
- Bogue R: Use S.M.A.R.T. goals to launch management by objectives plan. TechRepublic. 2005, Accessed: November 22, 2017: https://www.techrepublic.com/article/use-smart-goals-to-launch-management-by-objectives-plan/.
- CanadiEM. (2016). Accessed: November 22, 2017: http://canadiem.org/welcome-to-canadiem/.
- EMDocs. (2017). Accessed: November 22, 2017: http://www.emdocs.net/.
- EM sim cases. (2017). Accessed: November 22, 2017: https://emsimcases.com/.
- R.E.B.E.L. EM. (2017). Accessed: November 22, 2017: http://rebelem.com.
- Council of emergency medicine residency directors blog. (2017). Accessed: November 22, 2017: https://cordemblog.wordpress.com/.
- Academic life in emergency medicine. (2017). Accessed: November 22, 2017: https://www.aliem.com/.
- EMCrit. (2017). Accessed: November 22, 2017: http://emcrit.org/.
- Emergency medicine: reviews and perspectives. (2017). Accessed: November 22, 2017: https://www.emrap.org/.
- Chan TM, Thoma B, Lin M: Creating, curating, and sharing online faculty development resources: the medical education in cases series experience. Acad Med. 2015, 90:785–789. 10.1097/ACM.0000000000000692
- Core Rosen's and clinical knowledge. (2017). Accessed: November 22, 2017: https://canadiem.org/crackcast/.
- EM sim cases. (2017). Accessed: November 22, 2017: https://emsimcases.com/.
- Chan T: The McMAP Project | #FOAMed Assessment Resources. Academic Life in Emergency Medicine. 2015, Accessed: November 22, 2017: https://www.aliem.com/2015/06/mcmap-project/.
- Thoma B, Sanders JL, Lin M, et al.: The social media index: measuring the impact of emergency medicine and critical care websites. West J Emerg Med. 2015, 16:242–249. 10.5811/westjem.2015.1.24860
- Kirkpatrick DL, Kirkpatrick JD: Evaluating Training Programs: The Four Levels. Berrett-Koehler Publishers, Oakland, CA; 2006.
Crowdsourced Curriculum Development for Online Medical Education
Ethics Statement and Conflict of Interest Disclosures
Conflicts of interest: In compliance with the ICMJE uniform disclosure form, all authors declare the following: Payment/services info: All authors have declared that no financial support was received from any organization for the submitted work. Financial relationships: Seth Trueger declare(s) personal fees from Emergency Physicians Monthly. Seth Trueger reports receiving a stipend as social media editor for Emergency Physicians Monthly. Teresa Chan, Brent Thoma declare(s) non-financial support from CanadiEM.org. Teresa Chan and Brent Thoma operate and contribute to CanadiEM.org. Teresa Chan, Brent Thoma declare(s) non-financial support from Academic Life in Emergency Medicine. Teresa Chan and Brent Thoma operate and contribute to CanadiEM.org. Eric Shappell declare(s) Owner from EM Fundamentals. Eric Shappell owns and operates the EM Fundamentals website. James Ahn declare(s) non-financial support from EM Fundamentals. James Ahn is a contributor and adviser for EM Fundamentals. Robert Cooney declare(s) non-financial support from Council of Emergency Medicine Residency Directors. Robert Cooney contributes to the CORD blog (cordemblog.wordpress.com). Other relationships: All authors have declared that there are no other relationships or activities that could appear to have influenced the submitted work.
Cite this article as:
Shappell E, Chan T M, Thoma B, et al. (December 08, 2017) Crowdsourced Curriculum Development for Online Medical Education. Cureus 9(12): e1925. doi:10.7759/cureus.1925
Received by Cureus: November 25, 2017
Peer review began: November 28, 2017
Peer review concluded: December 06, 2017
Published: December 08, 2017
© Copyright 2017
Shappell et al. This is an open access article distributed under the terms of the Creative Commons Attribution License CC-BY 3.0., which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.