Resources That Improve Medical Board Licensing Examination Performance

Purpose Examine the factors improving performance on national medical licensing board examinations. Rationale Accreditation Council for Graduate Medical Education (ACGME) accredited residency programs report the United States Medical Licensing Examination (USMLE) Step 1 and Comprehensive Osteopathic Licensing Examination-USA (COMLEX-USA) Level 1 scores as the most important criteria in selecting candidates to interview. Hypotheses (1) Certain resources are superior for exam preparation. (2) Certain practice tests better assess exam preparedness. (3) USMLE performance will correlate with the COMLEX-USA. Methods One-hundred and two (102) medical students were surveyed regarding preparation for and performance on COMLEX-USA Level 1 and USMLE Step 1. Results USMLE-specific question banks were positively correlated with performance on COMLEX-USA Level 1 and USMLE Step 1 while COMLEX-specific question banks showed no correlation. National Board of Medical Examiners (NBME) Comprehensive Basic Science Self Assessment (CBSSA) and National Board of Osteopathic Medical Examiners (NBOME) Comprehensive Osteopathic Medical Self-Assessment Examination (COMSAE) practice examinations were positively correlated with performance on the USMLE Step 1 and the COMLEX-USA Level 1. Scores on USMLE Step 1 and COMLEX-USA Level 1 were highly correlated. Students who took USMLE Step 1 performed better on COMLEX-USA Level 1 than those who did not. Conclusion COMLEX-specific resources may not adequately prepare students for COMLEX-USA Level 1. Students studying for COMLEX-USA Level 1 may benefit by preparing for USMLE Step 1.


Introduction
Each year, both the number of U.S. medical schools and their respective class sizes continue to grow, dramatically outpacing the expansion and related availability of postgraduate year (PGY)-1 medical residency positions [1]. Since 1993, there have been fewer available positions in American Council of Graduate Medical Education (ACGME) accredited post-graduate training programs than there were applicants seeking to match [2]. In the 2017 National Residency Matching Program (NRMP) match, a total of 35,969 applicants competed for only 28,849 PGY-1 positions, representing a ratio of applicants to positions of 0.8 [2]. Despite the upcoming merger of ACGME and American Osteopathic Association (AOA) accredited postgraduate training positions by 2020, the move would only increase the number of accredited ACGME PGY-1 positions by 3,109, assuming that all AOA programs participating in the 2017 National Matching Service (NMS) match were incorporated into a joint match [3].
This trend of continued growth in the domestic medical graduate pool has served as a catalyst for the increasing competition that all applicants face in attempting to match into a residency program. The literature is replete with surveys and analyses of criteria that are predictive of success in both interviewing and matching at prospective residency programs. Consistent throughout much of this material is the conclusion that performance on the first component of national medical licensing board examinations -either United States Medical Licensing Examination (USMLE) Step 1, published by the National Board of Medical Examiners (NBME), for US-MD students, international medical graduates, and osteopathic medical students, or the Comprehensive Osteopathic Medical Licensing Examination of the United States (COMLEX-USA) Level 1, published by the National Board of Osteopathic Medical Examiners (NBOME) for osteopathic medical students -is a standardized comparison tool that proves vital in both the screening and ranking of candidates for residency positions in the United States [4][5]. Additional incentives for bolstering student performance on standardized board examinations result from the importance that residency directors place on passing these examinations in a single attempt [4], the accreditation requirement that allopathic medical schools use these rates in program quality improvement [6], and the accreditation requirement of the Commission on Osteopathic College Accreditation (COCA) that osteopathic medical schools publicly report these rates [7].
In consideration of this necessity to excel in medical licensing board examinations, medical students are inclined to be rigorous, persistent, and strategic in their approach to preparation. Recent research in the fields of learning and cognition has generated evidence to better inform students and educators about the methodology most likely to lead to success in taking highstakes examinations. In particular, a great deal of work has been conducted on assessing the use of retrieval practice (i.e., formative testing) as a more efficacious manner in which to encode long-term memory when compared to elaborative processes (often consisting of passive repetition). While traditional schools of thought have presupposed that the retrieval of information such as that completed during testing scenarios only served to assess previously acquired knowledge, recent work suggests that retrieval practice can serve as a mnemonic enhancer and generate greater long-term retention than would elaboration [8][9]. Further, studies suggest that retrieval practice may promote knowledge acquisition that can be applied in novel contexts [9]. Retrieval practice may also contribute to a phenomenon known as the "forward effect," a process through which new information presented after conducting retrieval practice is more effectively consolidated into long-term memory [10].
The aim of the present study was to examine several factors in national medical licensing board examination preparation that have the potential to directly influence performance outcomes. We sought to evaluate the association between the use of retrieval practice examination preparation methods (i.e., the use of question banks) and performance on licensing examinations, in addition to evaluating retrieval practice efficacy amongst various commercial preparation products that were marketed to target preparation for either the USMLE or the COMLEX-USA. Finally, we explored the independent correlation between performance on the USMLE and COMLEX-USA examinations.

Materials And Methods
In 2017, the authors solicited survey participation from the 171 third-year osteopathic medical students then enrolled in the Class of 2019 at the University of New England College of Osteopathic Medicine. Of those solicited, 102 (60%) students elected to complete the authors' survey. This population of students both prepared and sat for the COMLEX-USA Level 1 while 58 also both prepared and sat for the USMLE Step 1 during the 2016-2017 academic year. Survey participation was solicited via e-mail, with voluntary participation. The survey was open during the months of July and August of 2017. Responses were de-identified and no additional demographic information was collected in order to maintain the anonymity of the participants.
The survey was composed of five sections. Section 1 of the survey sought to elucidate the month the student began preparation, question banks used, question mode (tutor, timed, or both tutor and timed), question organization (randomized or organ system-specific), number of questions completed in each question bank, and number of questions repeated in each question bank. Section 2 was composed of questions pertaining to the tools used for study plan organization, flashcard use (complete or partial), video resource use (complete or partial), and book use (complete or partial). Section 3 consisted of questions about Comprehensive Osteopathic Self-Assessment Examinations (COMSAE) Forms A-E (date taken and corresponding score) and COMLEX-USA Level 1 (date taken, goal score, and actual score). The final question of this section inquired as to whether or not the student prepared for USMLE Step 1. If this question was answered in the affirmative, participants were directed to Section 4 while those participants answering in the negative were directed to Section 5. Section 4 was composed of questions regarding NBME Comprehensive Basic Science Self-Assessments (CBSSA) 12-13 and 15-19 (date taken and score) and USMLE Step 1 (date taken, goal score, and actual score). Section 5 consisted of inquiries into residency specialty interest and how well the college curriculum prepared students for COMLEX-USA Level 1 (Likert scale of 1-7, with 1 being "very poorly" and 7 being "very well").
Linear regression was used to evaluate the relationships of continuous variables (USMLE Step 1 and COMLEX-USA Level 1 performance, the number of questions completed in the various question banks, and performance on practice exams). Survey responses were also analyzed using unpaired t-tests comparing parametric variables. Statistical analysis was performed using the R Statistical Programming Package [11]. This study was reviewed by the university institutional review board (IRB) and its protocol was determined to be exempt from further IRB review and oversight.

USMLE and COMLEX-USA relationship
Fifty-eight students (57% of participants) reported taking both USMLE Step 1 and the COMLEX-USA Level 1. Performance outcomes for these examinations were strongly correlated (r = 0.78) and can be found in Figure 1, and students who reported taking the USMLE Step 1 performed better than those who did not, with those taking USMLE with a mean of 633 on the COMLEX-

Question banks
The total number of questions that students completed was positively correlated with performance on the USMLE Step 1 (r = 0.30) and COMLEX-USA Level 1 (r = 0.29). The number of questions answered in banks marketed for preparation for the USMLE, such as UWorld, USMLE Rx, and Kaplan USMLE, was positively correlated with performance on both USMLE Step 1 (r = 0.38) and COMLEX-USA Level 1 (r = 0.42). The number of questions answered in banks marketed toward preparation for the COMLEX, such as Truelearn COMBANK (question bank for COMLEX-USA), COMQUEST (question bank for COMLEX-USA), and Kaplan COMLEX, was poorly related to performance on the USMLE Step 1 (r = -0.03) and negatively correlated with performance on COMLEX-USA Level 1 (r = -0.22) ( Table 1).   Practice examinations NBME CBSSA practice tests were consistently positively correlated with performance outcomes of USMLE Step 1 (r = 0.53-0.80) and COMLEX-USA Level 1 (r = 0.55-0.73). Students' highest performing NBME CBSSA was correlated with both the USMLE Step 1 (r = 0.82) and COMLEX-USA Level 1 scores (r = 0.74). Students' performance on a COMSAE taken nearest to their USMLE Step 1 and COMLEX-USA Level 1 examination was positively correlated with the USMLE Step 1 (r = 0.41) and COMLEX-USA Level 1 score (r = 0.61) (

Video resources
Prior to adjusting for multiple comparisons, Sketchy Micro (marketed by Sketchy Group, LLC, California, US) was the only resource whose total completion was related to higher scores on the COMLEX-USA (t = -2.17, p = 0.04). Total completion of video resources was not related to performance outcomes on the USMLE Step 1 ( Table 4).

Discussion
National medical licensing board examination performance is crucial to both student success in attaining post-graduate residency training positions and in medical school accreditation. In consideration of reports that licensing board examination performance is a prime factor in selecting applicants for residency interviews [4], and the importance of examination pass rates in the accreditation of COCA-accredited medical schools [12], improving performance on these high-stakes examinations is in the interest of both students and their respective institutions. There is, presently, a paucity of published data demonstrating evidence of modifiable factors that bolster examination performance. Rather, much of the literature centers on the topic of admission criteria such as the Medical College Admission Test (MCAT) score and the undergraduate or post-baccalaureate grade-point average (GPA) [13]. There is also limited data available to the public analyzing the relationship between performance on practice examinations, such as the NBME CBSSA and NBOME COMSAE, and the eventual outcomes of the correlated high-stakes licensing examinations.
US-MD students and international medical graduates (IMGs) are required to take USMLE Step 1 while all osteopathic medical students are required to take COMLEX-USA Level 1, with the option to also take USMLE Step 1 in order to apply for medical licensure in the United States. Certain medical residency programs require that USMLE Step 1 and occasionally USMLE Step 2 CK be taken by osteopathic medical student applicants, while others will accept COMLEX-USA Level 1 and 2CE scores alone. This had led to an interest in the correlation between scores on the two examinations. A previous small-scale study determined the Pearson correlation coefficient between COMLEX-USA Level 1 and USMLE Step 1 scores to be 0.83 (p<0.001) [14].
Our study showed a similar correlation of 0.78 (p < 0.001), supporting the hypothesis that a strong positive correlation exists between COMLEX-USA Level 1 and USMLE Step 1 examination performance.
Several third-party companies offer practice questions for medical student board examination studying. The present study assessed which question banks students used, the number of questions completed, and the outcome of their performance on both USMLE Step 1 and COMLEX-USA Level 1. Consistent with our expectations, it was found that the number of practice questions students completed correlated positively with scores on both USMLE Step 1 and COMLEX-USA Level 1. Additionally, these results demonstrated that students who used question banks in timed mode performed better than students who completed these question banks in tutor mode or timed-tutor mode. It is worth noting that in this study, questions banks marketed toward preparation for USMLE Step 1 had a modestly stronger correlation with COMLEX-USA Level 1 scores than with USMLE Step 1 scores (r = 0.42 and r = 0.38, respectively). Interestingly, the data collected in this study demonstrated a negative correlation between the number of questions completed in banks marketed toward preparation for the COMLEX-USA Level 1 (r = -0.22) and COMLEX-USA Level 1 scores. This finding was particularly noteworthy given that some products, such as Truelearn COMBANK, advertise the applicability of its product to COMLEX-USA Level 1 in much of their marketing materials and student testimonials [15][16]. Osteopathic medical students who only sit for COMLEX-USA Level 1 may select these resources over those marketed for preparation for USMLE Step 1 as a result of such marketing statements and testimonials.
Practice examinations and self-assessments designed by the NBME and NBOME are tools used by medical schools and medical students in order to assess preparedness for the licensing examinations. While this study was not designed to assess the differences between NBME CBSSA and NBOME COMSAE, the strength of the correlation between scores on NBME CBSSA and the performance outcomes on COMLEX-USA Level 1 suggests that further research is needed in order to identify those self-assessments that will most reliably predict preparedness for these licensing examinations. While the COMSAE is described as a self-assessment for osteopathic medical students, the present data suggest that osteopathic medical students may not be best served in assessing their future examination performance as a direct function of their COMSAE scores.
Sketchy Micro, a microbiology review resource, was the only video resource that was related to an increase in score on COMLEX-USA Level 1. The completion of video resources did not appear to be related to the USMLE Step 1 score. Despite this finding, the present study may not have been adequately powered to reliably observe the relationship between the use of video resources and USMLE Step 1 and COMLEX-USA Level 1 performance.
This study was limited by being a convenience sample rather than a randomized trial of different resources. Due to the study population and the subject matter, it would be difficult to design such a trial where students would be encouraged to complete one resource and avoid others. Additionally, the study was limited by being self-reports of exam scores and exam preparation rather than information gathered through a data pull. This was done specifically to ensure the consent of participants in what data was and was not available, however, it remains a limitation of the above work.

Conclusions
This study demonstrates that resources written for the USMLE Step 1 exam are effective at preparing students for the COMLEX-USA Level 1 exam, and that performance on USMLE Step 1 is highly correlated with COMLEX-USA Level 1. This study also demonstrates the need for both medical schools and students to evaluate the resources they purchase for board examination preparation, especially those that claim to be designed for COMLEX-USA Level 1. Additionally, a collaboration between LCME-and COCA-accredited medical schools and longitudinal studies is needed in order to evaluate the resources and study methods used by medical students to optimize their performance on these examinations.

Appendices
This data is included and presented in the question bank subsection of the Results section, specifically as part of Table 1. The following figures are presented for additional reference.