Standard-Setting of Multidisciplinary Objective Structured Practical Examination

Purpose The present work applies the standard-setting in the multidisciplinary Objective Structured Practical Examination (OSPE) of the MusculoSkeletal and Integument systems (MSK) module using Modified Ebel’s method to differentiate the competent students from the non-competent ones. Materials and methods One hundred fifty-six students participated in the multidisciplinary OSPE. The MSK-OSPE consists of mid-module and final. According to the blueprint of the OSPE, the mid-module OSPE tested the knowledge and skills of the upper limb, and the final OSPE verified the knowledge and skills of the lower limb. Modified Ebel’s method was used to identify the Minimum Pass Level (MPL) in each station and the whole exam accordingly. Results Fifty-seven percent (57%) of the students passed both exams, while 25.6% did not pass the mid-module exam and 31.4% did not pass the final exam, 17.9% did not pass both exams and 25% did not pass one exam. The MPL for most of the stations in both exams using modified Ebel’s method of the standard-setting was more than 50% which is the conventional pass mark. However, the MPL for stations 4, and 6 in the mid-module exam (ulna and arteries of the upper limb) and stations 7, 9, and 14 (muscles of the lower limb, anatomy of ankle joint, physiology of nerve) was < 50%. While the total pass mark of the mid-module OSPE was 66% and the pass mark for the final OSPE was 60% Conclusion The minimal pass level (MPL) in mid module and final OSPE were 66% and 60% respectively which are more than the conventional cut off point (50%) that indicating that the standard-setting was effective in identifying poor performers who cannot be identified by the conventional method that led to enhance the quality of OSPE as an assessment tool. Moreover, students developed the skills to deal with standardized patients in clinical stations. However, some defects and areas of improvement were identified in some physiological and anatomical stations. The organizing committee recommended identifying the poor performers and conducting extra-tutorial sessions on the defective topics.


Introduction
Most medical schools assess the practical parts through the standard spotter tests and objective structured practical examination (OSPE) [1]. Some schools use the oral (viva voce) examination [1]. OSPE is the widespread method of assessing practical basic science in the early phase of the medical curriculum [2]. OSPEs evaluate the theoretical, applied, and technical skills instantaneously [3,4]. OSPEs differ from standard spotter examinations in many aspects. The station in OSPE has one specific objective, tests a higher level of the knowledge domain, and is structured according to the exam's blueprint [5].
Two different disciplinary OSPEs are usually used to assess the students, the classical multi-disciplinary OSPE and mono-disciplinary OSPE. Most institutes use the mono-disciplinary OSPE where the examination runs in one discipline [6]. Fakeeh College for Medical Sciences is a newly developing private college that offers four undergraduate programs. One of them is the Bachelor of Medicine, Bachelor of Surgery program (MBBS). The MBBS program is a system-based, spiral-integrated program. The program is delivered as modules in the preclinical phase, each module has theoretical and practical examination. The program utilized the classic multi-disciplinary OSPE as this type of examination allows the integration between different basic science and clinical sciences.
The musculoskeletal and Integument system (MSK) module commenced in the academic year 2018-2019. The 2nd year medical undergraduates undertake this module. The module is four credit hours, delivered in five weeks and divided into two theoretical hours and two practical hours.
Based on the course specification of the module, the practical assessment of the MSK module is organized through two multidisciplinary OSPEs (mid-module and final). The interval period between the two examinations is three weeks. The current study aimed to apply the standard-setting using Modified Ebel's methods to the multidisciplinary OSPE to ensure high-quality OSPE as an assessment tool differentiating between competent and non-competent students.

Design
The current study was a descriptive observational study.

Sampling
The sample was comprehensive, with 156 students (100 female, 56 male) in the 2nd year enrolled in the current study. All students completed the MSK course and participated in the multidisciplinary OSPE.

Description of multidisciplinary OSPE
Based on the course specification of the MSK module, the module has two multidisciplinary OSPEs (midmodule and final). Both OSPEs were conducted in May 2021. the examinations were designed in alignment with the intended learning outcomes of the module. The OSPE stations were assessed by a multidisciplinary faculty committee and an Examination Reliability analysis was conducted.
The organizing committee was formulated of thirteen members, two members from each discipline and one medical educationist. The disciplines represented in this committee are Anatomy, Physiology, Histology, Pathology, Radiology, and Orthopedic. All committee members are Ph.D. holders and highly qualified in their area of expertise with more than three years of teaching experience. In addition to that, all the committee members were familiar with the students and their performance. The questions were created following the blueprint [7]. The answer key was prepared before the examination.
There were 12 stations in the mid-module OSPE and 17 stations in the final OSPE. The time allowed in each station was 3-5 minutes. Each station was marked out of 5. In alignment with the integrated nature of the MBBS program, the multi-disciplinary OSPE included nine anatomical, one histological, one radiological, and one clinical examination (Figure 1a).

FIGURE 1: Mid-module and final OSPE stations
While, the final OSPE stations were nine anatomical, one histological, one radiological, two physiological, two pathological, and two clinical examinations (Figure 1b). The detailed questions and figures of each 2022          You are requested to examine the patient's elbow and summarize the findings to him.  1-Which one of the marked nerves is affected (A or B or C)? Mention its name -----------A, the sciatic nerve. 2-Name two muscles directly supplied by the affected nerve-----Biceps femoris, semitendinosus, and semimembranosus. 6 3-f A 16 -year-old student was seen in the ER with a stab wound injury to the inguinal region. The femoral artery was injured.
1-Which one of the marked vessels was affected in this case (A or B or C)? A 2-Name two branches given by the affected artery---Superficial epigastric, superficial external pudendal.      You are requested to examine the patient's knee and summarize the findings to the patient.

TABLE 2: OSPE final stations
The clinical examination stations tested the psychomotor skills in OSPE. These observed stations with standardized patients are very helpful to test skills of joint examination. The other stations included cadaveric specimens, plastic models, plastinated models, light microscopy, or images [5]. A clinical scenario was given at each station, and students were asked integrated questions. The questions were in two parts, the first part was the identification of the labeled structure, and the second part was relating that structure to its function [5]. Such a way of design tests the basic science competency [8].

Description of the standard-setting method (modified Ebel's method)
Modified Ebel's method was used to identify the minimum pass level (MPL) of the OSPEs [9]. In this method, the subject matter expert of each discipline categorizes each station based on the difficulty level and relevance. An expected score at each station was calculated by multiplying the raw score for relevance and difficulty with the proportion of minimally competent students ( Table 3).  To set MPL for the whole mid-module and final OSPE, the average score of all stations was calculated by the organizing committee. The chi-square test was used to compare the number of students who scored less than pass marks in conventional methods and modified Ebel's method.

Ethical considerations
The Institutional Review Board at Fakeeh College for Medical Sciences (FCMS) approved the study. The approval number is 43/IRB/2018. Permissions were taken from the undergraduate curriculum and examination committee.

Results
The reliability analysis of mid-module and final multidisciplinary OSPE was performed, and the results were 0.834 in mid module and 0.880 in final OSPE which reflected very good reliability as shown in Table 4. In addition to that, all stations contributed to overall reliability in mid-module and final OSPE.  The mid-module OSPE tested the knowledge, skills, and attitudes learning outcomes of the upper limb, and the final OSPE verified the learning outcomes of the lower limb. The MPL for the mid-module stations was as the following: stations 1-4 (anatomy of the bones) 93%,70%, 74%,41%; stations 5-7 (anatomy of the nerves, arteries, muscles) 64%, 34%, 59%; stations 8-9 (anatomy of the joints) 68 %, 58%; station 10 (X-ray station) 71%; station 11 (histological of the bone) 83%; and station 12 (clinical examination station) 82%. Accordingly, the pass mark of the mid-module exam was calculated to be 66%.

5: MPL for the mid-module and final OSPE stations
The MPL for stations 4 and 6 in the mid-module exam (ulna and arteries of the upper limb) and stations 7, 9, and 14 (muscles of the lower limb, anatomy of ankle joint, physiology of nerve) was < 50%. No ambiguity was noted on the questions of these stations. So, these two stations were not dropped.
One hundred and fifty-six students (100 female, 56 male) were participating in the MSK module. All students completed the course. According to the conventional pass score, 25.6% (18.6% female,7.1% male) did not pass the mid-module exam.   The average pass scores in mid-module and final multidisciplinary OSPE using Modified Ebel's method were 66% and 60% accordingly. The OSPE organizing committee believed this achieved performance to be satisfactory. In addition to that, using the bell curve, the distribution of both OPSEs scores through applying modified Ebel's method seems to take the normal distribution.
The percentage of students who scored >50% and >66% in mid-module OSPE were (74.4%) and (69.9%) respectively. This difference was significant (p < 0.001). In addition to that, The chi-square test showed that there is a significant difference between the percentage of passed students using the conventional pass mark (74.5%) and passed students percentage using Modified Ebel's Method (68.6%) in the final OSPE ( Table  7 above).

Discussion
Traditional practical examination (TPE) was the basis of assessment of practical skills and knowledge for many years, and much was adjusted (MCQ, spot) to overcome the TPE flaws [10]. Recently, many authors favored the replacement of TPE with OSPE as OSPE has no inadequacies of the TPE, and deals with the shifting scenario of medical education, which entails problem-based learning, early clinical exposure, and inter-professional collaboration and education [6,11].
MBBS program at FCMS is a system-based, spirally integrated curriculum with an early clinical experience component, thereafter multidisciplinary OSPE was the aligned method of assessment utilized in such type of curriculum.
The musculoskeletal module is a multidisciplinary module where the teaching and learning process is delivered in an integrated manner. Thereafter the multidisciplinary OSPE is applied in this module. The main advantage of applying the multidisciplinary OSPE is that medical students should have a holistic picture of the disease starting from the anatomy, physiology, and pathology of the affected organ to the clinical aspect [8].
The mid-module OSPE assessed the learning outcomes of the upper limb, and the final OSPE verified the learning outcomes of the lower limb. The first nine stations assessed the anatomical knowledge and station 10 checked the applied radiological knowledge. We also analyzed pathological and physiological information in the final exam (stations 12-16) and the histology of bone and muscle in both exams.
The mid module and final established an examination of joints (station 12 in the mid-module exam and stations 16 and 17 in the final exam) to assess psychomotor skills and verify the impact of anatomical knowledge on clinical skills. Schoeman and Chandratilake showed a weak relationship between anatomy competence and clinical skills among junior medical students [12]. However, in the current study, there was a good relationship between all the stations as reflected in the two OSPES' reliabilities.
The reliability of mid-module and final OSPE were 0.83 and 0.88 respectively that reflecting very good internal consistency of the examination and this result comes in accords with a study conducted at Manipal University to assess the validity of OSPE in the pharmacology department [13].
Modified Ebel's method was applied to the multidisciplinary OSPE and MPL is calculated for each station by subject matter experts, the MPL in mid-module and final ranged between 34 and 93. This result is similar to a study conducted at Al-Faisal University, Saudi Arabia [14]. This indicates that the multi-disciplinary OSPE stations have various difficulties and relevance.
There was statistical significance between the percentage of passed students using the conventional method and Modified Ebel's method in mid-module and final OSPE. This result is consistent with the result of a study conducted in Melaka Manipal Medical College, India, aiming to set standards for OSPE [15]. This reflects that standard-setting methods help to differentiate between competent and incompetent students and this enhances the quality of OSPE as an assessment tool.
Feedback was sent to all participating departments to take all necessary actions for next year. The staff members in various departments advised more practical sessions for shutting any pertinent flaws.

Limitation of the study
The method was applied to the musculoskeletal module only. In addition to that, more than two experts are needed in each discipline.

Conclusions
Teaching is an energetic process. We need constant assessment of our evaluation tools. The practical assessment of the MSK module is structured in two OSPEs (mid-module and final). The MPLs for midmodule and final multidisciplinary OSPE were 66% and 60% accordingly. The OSPE organizing committee believed this achieved performance to be satisfactory. The students developed the skills to deal with standardized patients through the clinical examination stations. Defects in some physiological and anatomical stations were observed. The organizing committee recommended identifying the poor performers and conducting extra-tutorial sessions on the defective topics.

Additional Information Disclosures
Human subjects: Consent was obtained or waived by all participants in this study. Institutional Review Board at Fakeeh College for Medical Sciences issued approval 43/IRB/2018. The Institutional Review Board at Fakeeh College for Medical Sciences approved the study. The approval number is 43/IRB/2018. Permissions were taken from the undergraduate curriculum and examination committee. Animal subjects: All authors have confirmed that this study did not involve animal subjects or tissue. Conflicts of interest: In compliance with the ICMJE uniform disclosure form, all authors declare the following: Payment/services info: All authors have declared that no financial support was received from any organization for the submitted work. Financial relationships: All authors have declared that they have no financial relationships at present or within the previous three years with any organizations that might have an interest in the submitted work. Other relationships: All authors have declared that there are no other relationships or activities that could appear to have influenced the submitted work.