Aim: The Triage and Action (T&A) minor emergency course was developed to improve the clinical skills of Japanese non-specialist physicians for minor emergent problems. Currently, the course quality is evaluated only by a self-reported satisfaction questionnaire. This study described a new clinical skills and confidence questionnaire to evaluate its validity and reliability.
Methods: The web-based questionnaire was evaluated by 103 physicians identified from a mailing list as having taken the T&A minor emergency course. The clinical experience and confidence (CEC) questionnaire was prepared, and its content and contextual validity were validated using a clinical sensibility test (CST). Reliability was assessed by the interclass correlation coefficient after two weeks via a follow-up CEC questionnaire.
Results: Of the 103 physicians contacted 44 (42.7%) responded to the questionnaire, 36 (40.8%) to the follow-up CEC questionnaire, and 33 (32.0%) to both questionnaires; 28 (27.2%) participants took the clinical sensibility test. Five questions which asked the total number of patients treated within six months showed fair agreement on the reliability test. All answers to the questions in the CST were favorable.
Conclusion: We removed every question which asked the total number of patients treated for various minor emergencies within six months from CEC. Consequently, the new questionnaire was shown to be contextually well validated and reliable. We will use the CEC questionnaire to improve our course, which we hope to demonstrate improved primary care for selected minor conditions.
The practice of emergency medicine in Japan is different from that in western countries . Adequate emergency physicians are unavailable, and non-emergency physicians are required to manage patients with minor emergent problems in the emergency room. These deficits have led to ambulances not being able to find hospitals with appropriate resources to treat patients. A frequent reason given for refusal is “without a specialist,” in case of minor emergencies .
We developed a Triage and Action (T&A) minor emergency course (http://minoremergency.club/) in Japan and began offering it in 2015 with simulation training to improve clinical knowledge and skills; simulation training has been reported to have clinical context validity in various studies [3-5]. The course has been held 21 times, and 461 physicians have completed it as of December 31, 2018. The course aims to improve clinical knowledge and skills for managing minor emergencies. The training sessions comprised five minor emergencies (epistaxis, ear and nose foreign body, sprain or fracture, ocular surface foreign bodies, and burns) with a lecture- and simulation-based training conducted by dedicated multidisciplinary instructors. The simulation-based training uses real-case scenarios, and participants decide how to treat each training-simulated patient with a minor emergency. The instructors described patients with various diseases and injury scenarios and assessed the participants’ decisions and skills. This is a one-day course (7 h), and two T&A minor emergency instructors supervise five attendants. Instructors for the T&A minor emergency course are selected by a T&A principal member after taking the course. All 441 participants who answered the post-course-paper-based “satisfaction” questionnaire responded positively when asked, “Do you think you can use lessons learned from this course in practice?” 80.1% responded “Strongly agree” and 19.9% responded “Agree” [responses from the Likert scale (strongly agree, agree, neither agree nor disagree, disagree, and strongly disagree)]. However, information regarding the physicians’ clinical practice in treating minor emergencies after taking the course has been limited.
Improvements in clinical skills have usually been evaluated by performing the same simulation task or in a real clinical situation more than three weeks after the simulation . However, T&A minor emergency course participants from all over Japan have participated in this course, making it difficult to evaluate the change in a particular physician’s skill level. Therefore, we considered that information regarding a physician’s confidence level and the total number of minor emergencies treated could be used to show the change in the physicians’ practice and reduce their refusals to treat minor emergencies.
Some simulation courses have used questionnaires to evaluate knowledge before and after the simulation course [7-9]. To ensure that the questionnaire closely matches the real-world experience and ability, the post-course questionnaire should be completed within a few months and should contain information about self-confidence and real clinical practice experience (e.g., change in the total number of treatments without referrals to a specialist within the specific period).
A questionnaire was used to evaluate the change of confidence or clinical skills among physicians . However, to our knowledge, articles on the validity and reliability of questionnaires and changes in physicians’ confidence and attitude before and after few months of taking the course are limited. Herein, we aimed to develop a clinical experience and confidence (CEC) questionnaire to evaluate its validity and reliability.
Materials & Methods
This study was approved by the Ethics Committee of the Tokyo Bay Urayasu/Ichikawa Hospital (approval number: 385).
We conducted this study in accordance with the method proposed by Burns et al. . The survey used a questionnaire and the clinical sensibility test (CST). We had previously developed the questionnaire to evaluate how participants change after taking a T&A minor emergency course. We tested the reliability of the questionnaire and used a CST to validate its content.
The participants responded to the first CEC questionnaire to validate its content using a CST, followed by responding to the follow-up CEC questionnaire within two weeks after the test to evaluate reliability (test-retest exam). This survey was conducted in November and December 2018.
The web-based questionnaire was evaluated by 103 physicians identified from a mailing list as having taken the T&A minor emergency course. There were two inclusion criteria. (1) To evaluate reliability, we included the responders who answered the first and follow-up CEC questionnaires. (2) To evaluate the validity, we included physicians who answered the first questionnaire and took the CST.
Two principal developers of the T&A minor emergency course and one physician-researcher created a new questionnaire, called the CEC questionnaire. The questionnaire comprised 32 questions related to two major factors: the physicians’ background and practice experience and self-confidence in treating minor emergencies (Supporting Information 1). The responses were provided by checking boxes or by ranking responses.
In the first section, the data collected included age, sex, post-graduate year, specialty (residents, 19 basic specialists from the Japanese Society of Internal Medicine, and others), and the number of hospital beds (clinic and 20-49, 50-99, 100-199, 200-299 beds, 300-499, and ≥500 beds) .
The second section of the CEC questionnaire had questions about the physicians’ experience and confidence in treating each type of minor emergency. We chose five minor emergencies (epistaxis, ear and nose foreign body, sprain or fracture, burn, and ocular surface foreign bodies) that were presented via simulation-based training in the T&A minor emergency course. To evaluate the participant’s hospital for its ability to provide minor emergency treatment, it was required that the hospital has a specialist for the disease; for example, the specialist for epistaxis and ear and nose foreign body was otolaryngologists [13,14], for sprain or fracture were orthopedic physicians , for burns were dermatologists and plastic surgeons , and for ocular surface foreign bodies were ophthalmologists . The following questions were created: (1) “The total number of patients seen within each specialty in a month,” (2) “With or without specialists in the respondent’s hospital,” (3) “Total number of minor emergencies treated within six months,” (4) “Confidence,” (5) Experience of each minor emergency without a specialist’s support within the last six months?,” and (6) “Total number of patients treated for each minor emergency without specialist support within the last six months?”
The web-based follow-up CEC questionnaire was modified and resent to participants two weeks after completion of the first CEC questionnaire. The modification was the removal of the physician’s background in the first questionnaire to reduce the effort needed to answer the follow-up CEC questionnaire.
To evaluate the reproducibility of the questionnaire, the participant’s name was used to match the first and follow-up CEC questionnaires’ responses.
Clinical sensibility test (CST)
The CST was performed to assess the comprehensiveness, clarity, and contextual validity of the web-based CEC questionnaire’s content (Supporting Information 2) : (1) “Important issues pertaining to the T&A course,” (2) “Missing items,” (3) “Simplicity and ease in understanding,” (4) “Information about physician’s knowledge and experience,” (5) Inappropriate or redundant items,” (6) “Issues in the physician’s knowledge and experiences of minor emergencies,” and (7) “Answering time (minutes).” The participants answered questions 1-4 and 6 by selecting a response from the Likert scale (e.g., very unlikely, unlikely, neutral, likely, and very likely). The answers to question 5 were “Yes” or “No.” Questions 2, 4, 5, and 6 had a free-entry column about each question. A free-entry column was also provided for the participants to suggest ideas on how to improve the questionnaire.
Primary data analysis
We used the STATA/MP 15.1 software (StataCorp LLC, Texas, USA) for data analyses and interclass correlation coefficient (ICC) analysis to assess reliability. We decided that the reliability coefficient could be qualitatively categorized as follows: ICC < 0.4 is poor, 0.4 ≤ ICC < 0.6 is fair, 0.6 ≤ ICC < 0.75 is good, and 0.75 ≤ ICC ≤ 1 is excellent .
Characteristics of study participants
During the study period, 44 (42.7%) participants responded to the first CEC questionnaire and 36 (40.8%) responded to the follow-up CEC questionnaire; 33 (32.0%) answered both questionnaires; 28 (27.2%) took the CSC, and no responders replied only to the CSC. Table 1 presents the backgrounds of the physicians who answered both CEC questionnaires.
The first CEC and the follow-up CEC
Table 2 presents the results of the participants who took the first CEC and the follow-up CEC questionnaires. The mean time period in which the two questionnaires were completed was 35.4 days (SD = 12.1). We found that five questions regarding “The total number of epistaxis patients treated without an otolaryngologist within 6 months”; “The total number of ear and nose foreign body patients treated without an otolaryngologist within 6 months”; “The total number of burn patients treated within 6 months”; “The total number of burn patients treated without a dermatologist or plastic surgeon within 6 months”; and “The total number of patients with ocular surface foreign bodies treated without an ophthalmologist within 6 months” achieved a fair ICC (0.4 ≤ ICC < 0.6).
Clinical sensibility test (CST)
Table 3 presents the results of the CST. For questions 1, 3, and 4, >80% of the respondents answered: “Fair to a large extent.” For question 2, 96.5% of the respondents opted for “Minor to insignificant gaps.” For question 5, 92.9% of the respondents stated that no items were inappropriate or redundant in the questionnaire. For question 6, 92.6% of the respondents answered “Likely to very likely.” The median time to answer the questionnaire was 5 minutes [interquartile range (IQR): 5-10 minutes].
The study results showed that the CEC questionnaire about the T&A minor emergency course was valid and reliable.
In the CEC and follow-up CEC questionnaire analysis, five questions about the total number of patients treated for various minor emergencies alone or with a specialist within six months gave a fair ICC value (0.4 ≤ ICC < 0.6). Morita et al. reported that it was difficult to show reliable results for questions involving human emotions or knowledge because of changes in the subjects’ subjective conditions . This fair ICC may have been caused by random error or because the participants saw different number of patients during the study periods. The results showed that total number of patients treated for various minor emergencies within six months was small and some participants did not experience them at all. Therefore, we considered questions asking for the total number of patients treated for various minor emergencies within six months were not informative enough, and the binominal questions which asked whether the participants experienced treating minor emergencies alone within six months were sufficient. Finally, we removed every question which asked the total number of patients treated for various minor emergencies alone or with a specialist within six months.
All answers to the questions in the CST were favorable, which we interpreted as strong indication of the validity of the questionnaire’s content and clinical context . Regarding question 2, some responders mentioned that this survey did not include a question about “satisfaction rate.” The lack of this question may cause less confidence in the questionnaire among some physicians. However, our course already collected information regarding the “satisfaction rate” on paper-based questionnaire without the physicians’ names before starting the study. Because we thought that the participants’ answers may be affected if the questionnaire was not anonymous, no name should be associated with the “satisfaction rate” question . Therefore, a question about the “satisfaction rate” was not included in our web-based questionnaire. Galesic et al. reported in their web-based study that their online questionnaire’s length (10, 20, and 30 min) was longer than the length and number of participants in our study and fewer respondents started and completed the questionnaire . Our study showed that the median time to answer the questionnaire was 5 (IQR, 5-10) min, which we considered to be very reasonable.
One of the participants commented in the free-entry column that “This questionnaire should contain questions about the availability of a specialist at night or when the primary physician has a day off.” Therefore, we added a question asking “Do you have specialist support at night or on holidays?,” and the answers were “Any time as needed,” “Sometimes,” and “Not at all.”
This study has several limitations. The first concerns the external validity because we chose T&A minor emergency mailing list members to serve as participants; therefore, the risk of selection bias should be considered. Furthermore, the limited sample size should be considered. The second is that we used two questionnaires over a two-week period. Therefore, the actual practice pattern may have changed during the two weeks. Third, the sample size was small, and the response rate was low, which can be attributed to the fact that the respondents were volunteers.
This newly developed questionnaire was developed to evaluate the change in clinical skills and confidence of clinical practice after completing a T&A minor emergency course. In the future, we plan to send a pre-web-based questionnaire before the course and a post-web-based questionnaire six months after the course. The post-web-based questionnaire contains the same question as those in the pre-questionnaire, except the addition of the question “Did you change your place of work after answering the pre-questionnaire?” to check this possibility. Our study shows the validity and reliability of the questionnaire. Future research should focus on administering the questionnaire to participants in the T&A minor emergency course. These results will provide us information regarding ways to improve the T&A minor emergency course.
- Hori S: Emergency medicine in Japan. Keio J Med. 2010, 59:131-9. 10.2302/kjm.59.131
- Tanigawa K, Tanaka K: Emergency medical service systems in Japan: past, present, and future. Resuscitation. 2006, 69:365-70. 10.1016/j.resuscitation.2006.04.001
- Friedman Z, Siddiqui N, Katznelson R, Devito I, Bould MD, Naik V: Clinical impact of epidural anesthesia simulation on short- and long-term learning curve: High- versus low-fidelity model training. Reg Anesth Pain Med. 2009, 34:229-32. 10.1097/AAP.0b013e3181a34345
- Gauger PG, Hauge LS, Andreatta PB, et al.: Laparoscopic simulation training with proficiency targets improves practice and performance of novice surgeons. Am J Surg. 2010, 199:72-80. 10.1016/j.amjsurg.2009.07.034
- Wayne DB, Didwania A, Feinglass J, Fudala MJ, Barsuk JH, McGaghie WC: Simulation-based education improves quality of care during cardiac arrest team responses at an academic teaching hospital: a case-control study. Chest. 2008, 133:56-61. 10.1378/chest.07-0131
- Cherry RA, Ali J: Current concepts in simulation-based trauma education. J Trauma. 2008, 65:1186-93. 10.1097/TA.0b013e318170a75e
- Siaulys MM, da Cunha LB, Torloni MR, Kondo MM: Obstetric emergency simulation training course: experience of a private-public partnership in Brazil. Reprod Health. 2019, 16:24. 10.1186/s12978-019-0689-6
- Kirkpatrick AJ, Cantrell MA, Smeltzer SC: Relationships among nursing student palliative care knowledge, experience, self-awareness, and performance: an end-of-life simulation study. Nurse Educ Today. 2019, 73:23-30. 10.1016/j.nedt.2018.11.003
- Tantavisut S, Tofte JN, Westerlind BO, Karam MD, Phisitkul P, Marsh JL: Efficacy and validation of a simulation-based compartment syndrome instructional course. Iowa Orthop J. 2018, 38:113-21.
- Morgan J, Green V, Blair J: Using simulation to prepare for clinical practice. Clin Teach. 2018, 15:57-61. 10.1111/tct.12631
- Burns KE, Duffett M, Kho ME, Meade MO, Adhikari NK, Sinuff T, Cook DJ: A guide for the design and conduct of self-administered surveys of clinicians. CMAJ. 2008, 179:245-52. 10.1503/cmaj.080372
- Muraoka A: Renovation of medical specialty education system in Japan (current status and future perspectives). Rinsho Shinkeigaku. 2013, 53:1142-4. 10.5692/clinicalneurol.53.1142
- Kucik CJ, Clenney T: Management of epistaxis. Am Fam Physician. 2005, 71:305-11.
- Heim SW, Maughan KL: Foreign bodies in the ear, nose, and throat. Am Fam Physician. 2007, 76:1185-9.
- Tiemstra JD: Update on acute ankle sprains. Am Fam Physician. 2012, 85:1170-6.
- Lloyd EC, Rodgers BC, Michener M, Williams MS: Outpatient burns: prevention and care. Am Fam Physician. 2012, 85:25-32.
- Gelston CD: Common eye emergencies. Am Fam Physician. 2013, 88:515-9.
- Cicchetti DV, Sparrow SA: Developing criteria for establishing interrater reliability of specific items: applications to assessment of adaptive behavior. Am J Ment Defic. 1981, 86:127-37.
- Morita Y, Miyamoto Y, Takano A, Kawakami N, Coulombe S: Reliability and validity of the Japanese version of the Mental Health Self-management Questionnaire among people with mental illness living in the community. BMC Psychol. 2019, 7:30. 10.1186/s40359-019-0301-4
- Burns KE, Kho ME: How to assess a survey report: a guide for readers and peer reviewers. CMAJ. 2015, 187:E198-205. 10.1503/cmaj.140545
- Futrell CM, Stem DE, Fortune BD: Effects of signed versus unsigned internally administered questionnaires for managers. J Bus Res. 1978, 6:91-8.
- Galesic M, Bosnjak M: Effects of questionnaire length on participation and indicators of response quality in a web survey. Public Opin Q. 2009, 73:349-60. 10.1093/poq/nfp031
Development of Clinical Skills and Confidence Questionnaire for Triage and Action Minor Emergency Course: Test-Retest Exam
Ethics Statement and Conflict of Interest Disclosures
Human subjects: Consent was obtained or waived by all participants in this study. The Ethics Committee of Tokyo Bay Urayasu/Ichikawa Hospital issued approval 385. The protocol for this research project was approved by the Ethics Committee of Tokyo Bay Urayasu/Ichikawa Hospital (approval number: 385) and it conforms to the provisions of the Declaration of Helsinki. Animal subjects: All authors have confirmed that this study did not involve animal subjects or tissue. Conflicts of interest: In compliance with the ICMJE uniform disclosure form, all authors declare the following: Payment/services info: All authors have declared that no financial support was received from any organization for the submitted work. Financial relationships: All authors have declared that they have no financial relationships at present or within the previous three years with any organizations that might have an interest in the submitted work. Other relationships: All authors have declared that there are no other relationships or activities that could appear to have influenced the submitted work.
The authors would like to thank all the participants for answering the questionnaire and Enago (www.enago.jp) for the English language review.
Cite this article as:
Numata K, Matsubara T, Okumura Y, et al. (September 10, 2021) Development of Clinical Skills and Confidence Questionnaire for Triage and Action Minor Emergency Course: Test-Retest Exam. Cureus 13(9): e17864. doi:10.7759/cureus.17864
Peer review began: August 27, 2021
Peer review concluded: September 10, 2021
Published: September 10, 2021
© Copyright 2021
Numata et al. This is an open access article distributed under the terms of the Creative Commons Attribution License CC-BY 4.0., which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.