"Never doubt that a small group of thoughtful, committed citizens can change the world. Indeed, it is the only thing that ever has."

Original article
peer-reviewed

## Quantitative Readability Assessment of the Internal Medicine Online Patient Information on Annals.org

### Abstract

#### Background

Approximately 90% of Americans have access to the internet with the majority of people searching online for medical information pertaining to their health, or the health of loved ones. The public relies immensely on online health information to make decisions related to their healthcare. The American Medical Association (AMA) and the National Institute of Health (NIH) recommend that publicly available health-related information be written at the level of the sixth-seventh grade.

#### Results

Eighty-nine articles were analyzed generating an average score as follows: FRE 62.8, FKGL 7.0, GFI 8.6, CLI 9.6 and SMOG 9.8. Overall, 87.6% of the articles were written at a level higher than the 7th-grade level, which is recommended by the AMA and NIH.

#### Conclusion

In an era of increased reliance on the internet for medical information pertaining to patients’ health, materials written at a higher grade than recommended has the potential to negatively impact patients’ well-being, in addition to tremendous ramifications on the healthcare system. Potentially redrafting, these articles can prove beneficial to patients who rely on these resources for making healthcare-related decisions.

### Introduction

In an era of widespread internet availability, it is estimated that approximately 90% of Americans have access to the internet, with more than 80% of internet users searching online for medical information. This translates to more than eight million Americans searching the internet for health-related information, on any given day [1-2]. Only a third of these users discuss this health information with their healthcare providers [1]. Moreover, 53% of health seekers report that the information obtained had an impact on how they take care of themselves or someone else [1]. This is in part because of the ease by which information could be found online and the fact that more than three-quarters of Americans own a smartphone and have access to the internet at the tip of their fingers [3].

Unfortunately, the 2003 National Assessment of Adult Literacy (NAAL) showed that 14% of Americans could not read, or understand text written in English and were only able to comprehend very basic, simple text [4]. Furthermore, almost half of Americans lack sufficient literacy required to appropriately comprehend and implement medical treatment and preventive health care, with grave economic consequences [4-6]. According to the NAAL report, uninsured adults have lower health literacy than insured adults [4]. In addition, limited health literacy is prevalent and associated with lower socioeconomic status, comorbidities, and poor health care access, which suggests that limited health literacy can be considered an independent risk factor for the disparities in health faced by the older population. Studies have shown that adults older than 65 years with lower health literacy were more likely to utilize the emergency department and experience higher costs during those visits [7].

Therefore, it is of utmost importance to define health literacy and eHealth literacy in this era. The Institute of Medicine defines health literacy as “the degree to which individuals have the capacity to obtain, process, and understand basic health information and services needed to make appropriate health decision” [8]. Multiple other definitions are proposed by various other entities, with varying degrees of emphasis on eHealth literacy, where eHealth literacy is defined as “a set of skills and knowledge that are essential for productive interactions with technology-based health tools” [9]. One of the crucial qualities of health literacy is the readability of health literature that is directed towards laypeople, with readability defined as “the ease with which written materials are read” [10]. Readability is paramount, as increased readability correlates with increased comprehension [10]. Multiple readability assessment tools are in place to assist authors in addressing the public with materials that are comprehensible and more effortlessly understood by readers. Yet, online patient information is often written at a level that is beyond the comprehensibility of the majority of the population [6,11-13]. Internal medicine ailments encompass a wide variety of illnesses, especially in the older generations. Hence, the requirement of easily readable materials to facilitate understanding and potentially improve adherence and outcomes.

### Materials & Methods

Online patient education materials available to the public from “Annals of Internal Medicine: Patient Information” website, http://annals.org/aim/pages/patient-information [15], were retrieved in September 2018. This website, sponsored by the American College of Physicians (ACP), offers healthcare-related information in the form of brief summaries of studies and clinical guidelines published in the Annals of Internal Medicine journal, targeting patients and interested lay people.

 Readability Scale Variables Formula Flesch Reading Ease (FRE) Average number of syllables (B), average number of words per sentence (W), average number of sentences (S) 206.835 – (84.6 x (B/W)) – (1.015 x (W/S)) Flesch Kincaid Grade Level (FKGL) Average number of syllables per word (SY) and the average number of words per sentence (W) (0.39 x W) + (11.8 x SY) – 15.59 Gunning Fog Index (GFI) Number of sentences (S), number of words (W), number of words with three or more syllables (C) 0.4 x (W/S + ((C/W) x 100)) Coleman-Liau Index (CLI) Average number of letters per 100 words (L) and the average number of sentences per 100 words (S) (0.0588 x L) – (0.296 x S) – 15.8 Simple Measure of Gobbledygook (SMOG) Average number of words with 3 or more syllables (C) and the average number of sentences (S) 1.043 x √ (C x (30/S)) + 3.1291

### Results

The readability grades for the 89 articles obtained from Annal.org online patient information were analyzed. Each article was analyzed individually using five readability scales. The average readability score using the FRE for the collective articles was 62.8, indicating a level of readability that is of average difficulty, per the USDHHS, which is higher than the recommended readability level by the USDHHS, NIH, and AMA [5,10,14]. Table 2 presents the readability scores for the 89 individual articles, in addition to the final column displaying the average grade level for each article.

 Topic Flesch Reading Ease Flesch Kincaid Grade Level Gunning Fog Index Coleman Liau Index SMOG Index Average Grade Level Abdominal Aortic Ultrasound 77.9 5.2 6.8 5.7 8 6.4 Acne 78.9 5.2 6.8 6.2 7.6 6.5 Acute Colonic Diverticulitis 58.7 7.2 7.6 9.6 9 8.4 Acute Gastrointestinal Bleeding 52.1 9.1 10.8 12.5 11.4 11.0 Acute Kidney Injury 65.9 5.8 7.3 7.7 8.7 7.4 Acute Pancreatitis 53.3 9.1 11 11.8 11.2 10.8 Acute Sinusitis 49.7 9.6 11 11 11.4 10.8 Alcohol Use 57.2 6.9 8 9.4 10.1 8.6 Aortic Stenosis 67.4 6.2 8.4 8.5 9.5 8.2 Asthma 69.9 6.8 8.6 9.7 9.2 8.6 Atopic Dermatitis (Eczema) 58.8 7.6 9.4 11.1 10.4 9.6 Atrial Fibrillation 79 4.7 6.8 6.9 8.4 6.7 Breast Cancer Screening and Prevention 74.2 5.3 7.2 8.9 8.8 7.6 Care of the Adult Cancer Survivor 48.6 9.5 11 13.1 11.3 11.8 Care of Returning Military Personnel 34.5 10.3 11.1 14.3 11.4 11.2 Carpal Tunnel Syndrome 69.9 6.1 7.4 9.1 8.5 7.8 Celiac Disease 49.4 10 12.2 12 12.1 11.6 Chlamydia and Gonorrhea Infection 39.7 10.2 12.4 13.2 12 12.0 Chronic Kidney Disease 67.4 6 6.4 9.6 8.1 7.5 Chronic Obstructive Pulmonary Disease 61.1 7.2 9.3 10.8 9.9 9.3 Clostridium difficile Infection 50.9 9.5 12.2 10.9 12.1 11.2 Common Cutaneous Parasites 67.2 6.7 8.5 9 9.3 8.4 Community-Acquired Pneumonia 65 6.7 8.7 8.4 9.9 8.4 Concussion 66.1 6.5 9 8.3 10 8.5 Constipation 62.9 6.6 7.6 9.9 9 8.3 Contraception 46.1 9.3 12 13.7 12 11.8 Deep Venous Thrombosis 83.8 4.1 5.9 6.5 7.7 6.1 Delirium 29.1 11.1 13 13.4 11.6 12.3 Dementia 57 7.7 9.6 10.9 10.8 9.8 Depression 67.7 5.9 8.6 8.2 9.6 8.1 Diabetic Ketoacidosis 66.8 7.6 10 8.2 10.5 9.1 Dyslipidemia 70.1 5.5 8 7.5 9.1 7.5 Eating Disorders 37.8 9.9 12 12.8 11.3 11.5 Epilepsy 62.5 6.7 8.7 9.3 9.8 8.6 Gastroesophageal Reflux Disease 62.6 7 7.8 10.4 9.4 8.7 Generalized Anxiety Disorder 30.2 11 13.2 15 12 12.8 Gout 78.2 4.7 6.5 7.6 7.9 6.7 Hearing Loss 67.1 6.4 8.5 8.8 9.5 8.3 Heart Failure 61.4 6.7 8.8 10.6 10 9.0 Heart Failure with Preserved Ejection Fraction 76 4.9 7.6 8.3 9 7.5 Hepatitis C Virus 67.4 7 9.3 7.5 10.2 8.5 Herpes Zoster 75.4 4.7 6.4 6.6 8 6.4 Hip Fracture 55.2 8.5 11.1 10.9 11.2 10.4 Hypertension 67.3 6.4 7.9 10.2 9.1 8.4 Hyperthyroidism 41 9.9 10.7 13.6 11 11.3 Hyponatremia 62.1 7.3 7.8 8.6 9.5 8.3 Hypothyroidism 35.7 10.4 11.2 14 10.8 11.6 Influenza 76.2 4.9 6.7 7.1 8.7 6.9 Insomnia 61.7 6.7 7.8 9.9 9.2 8.4 Irritable Bowel Syndrome 63.6 6.3 7.8 8.5 9.2 8.0 Low Back Pain 83.3 3.8 5.5 5.7 7.2 5.6 Lyme Disease 75.3 5.6 7 7.7 8.3 7.2 Management of Newly Diagnosed HIV Infection 69 6.5 8.6 8.5 9.6 8.3 Menopause 62.1 7.5 9.9 10.6 10.7 9.7 Migraine 68.4 6.1 7.9 9.6 9.2 8.2 Multiple Sclerosis 55.8 8.5 10.4 12.1 11.1 10.5 Nephrolithiasis 89.4 3.5 5.3 5 6.4 5.1 Obesity 78.5 5.1 7.9 6.8 9 7.2 Obstructive Sleep Apnea 57.5 8.1 9.4 11.3 10.2 9.8 Osteoarthritis 69.2 5.8 8.5 8.4 9.4 8.0 Osteoporosis 68.3 5.8 7.3 7.9 8.9 7.5 Palliative Care 73.1 5.2 7.3 8.1 8.8 7.4 Peripheral Arterial Disease 89.6 2.9 5.2 5.4 6.9 10.5 Perimenopause 50.1 8.3 10.9 11.6 11 5.1 Pharyngitis 44.7 10.3 12.4 13.5 12.5 12.2 Plantar Fasciitis 57.5 7.4 9.7 9.9 9.9 9.2 The Polycystic Ovary Syndrome 61.2 7.4 9.8 10 10.4 7.2 Polymyalgia Rheumatica 72.5 5.5 6.6 8.1 8.5 10.1 Preoperative Evaluation 64.1 8.1 11.3 9.4 11.5 7.4 Prostate Cancer 71.5 5.7 7 8.4 8.5 9.9 Pulmonary Hypertension 60.6 7.5 10.6 10.6 10.8 7.3 Restless Legs Syndrome 72 5.4 6.7 8.7 8.5 10.3 Rheumatoid Arthritis 55 8.3 10.2 11.9 10.8 8.2 Rotator Cuff Disease 67.5 6.5 7.7 9 9.5 9.3 Sarcoidosis 56.5 7.6 9 10.5 10.1 9.7 Screening for Colorectal Cancer 59.3 8.4 9.5 10.5 10.3 10.3 Sickle Cell Disease 54.2 8.7 10.2 11.1 11 7.7 Smoking Cessation 70.8 5.4 7.7 8.7 8.9 11.5 Stable Ischemic Heart Disease 51.1 9.4 12.1 12.4 11.9 7.4 Substance Use Disorders 76.4 5.2 7.6 7.6 9.3 7.8 Systemic Lupus Erythematosus 66.6 6.3 6.8 9.4 8.6 9.4 Transient Ischemic Attack 70.1 5.9 7.3 9.3 8.8 7.8 Transitions of Care 42.5 10.5 12.9 14.6 12.6 12.7 Travel Medicine 76.9 5.1 7.8 6.7 9 7.2 Tuberculosis 59.7 7.1 8.5 9.3 9.4 8.6 Type 2 Diabetes 71.4 5.5 6.9 8.1 8.8 7.3 Urinary Tract Infection 66.4 6.1 7.4 7 9 7.4 Vaginitis and Cervicitis 42.1 10.1 12.3 11.6 12.3 11.6 Venous Leg Ulcers 81.1 4.3 6.7 6.3 8.4 6.4

### Discussion

Patient reliance on online education materials to enhance their well-being and determine when to visit a physician have increased exponentially over the past decade, with more than 50% of internet users reporting that information found on the internet affected their decision in treating a medical condition [1]. While online education materials found on the patient information site on Annals.org provide patients with a valuable, evidence-based resource for maintaining their well-being, readability of the content of these materials is variable, with 87.6% of articles written above the seventh-grade level.

Health literacy is complicated, with multiple variables that play part in the overall comprehensibility of educational materials [22]. Individuals’ level of education and familiarity with medical terminology are fundamental in assessing the level of understanding they will attain in reading online education materials. Health literacy is heavily dependent on the education attained by patients, with lower health literacy rates linked to poorer health outcomes, including increased hospitalizations, poorer health status and higher mortality [23]. Studies have shown lower mammography studies and lower influenza immunizations among patients with lower health literacy [23]. On a larger scale, lower health literacy rates are associated with the tremendous economic cost to the US economy, with estimates ranging between $70 and$230 billion US dollars annually [5-6].

Comparing the readability of online education materials found on Annal.org to other major medical societies shows better overall readability of the materials with an average grade level being 8.8 ± 1.8 (SD) [6,10-13]. Using readability scales to assess comprehensibility of education materials carries inherent flaws and has its own limitations since essentially all the algorithms used, take into account word length and the number of syllables in the words to assess readability. For example, more difficult medical words such as “lipid” or “ketone”, which are short in length can be interpreted as more readable than longer words, such as “hospitalization”, which is longer, yet more understandable by the general population. Nonetheless, it remains true that the materials obtained are written beyond the recommended readability of the American population. Revisions to the materials can effectively increase readability, increasing comprehension among readers with limited health literacy capabilities. Utilizing graphics and videos can also increase the comprehension of complex health information that is difficult to clarify using textual information. Attaining a goal of near-universal comprehension can improve health outcomes and potentially minimize the financial burden associated with these outcomes.

### Conclusions

The online patient education materials found on the Annals.org website provide patients with an excellent source of information to enable them to care for themselves and loved ones. Nevertheless, the majority of the materials are written at a readability level higher than recommended by the AMA and NIH. Given the integral role that online patient education materials play in the decision-making process of patients and follow-up care with physicians, greater emphasis should be placed on the readability and comprehensibility of online educational materials.

### References

1. Online health search 2006. (2006). Accessed: November 1, 2018: http://www.pewinternet.org/Reports/2006/Online-Health-Search-2006.aspx.
2. 11% of americans don’t use the internet. who are they?. (2018). Accessed: September 25, 2018: http://www.pewresearch.org/fact-tank/2018/03/05/some-americans-dont-use-the-internet-who-are-they.
3. Record shares of americans now own smartphones, have home broadband. (2017). Accessed: September 23, 2018: http://www.pewresearch.org/fact-tank/2017/01/12/evolution-of-technology.
4. Kutner M, Greenberg E, Jin Y, Paulsen C: The Health Literacy of America’s Adults: Results from the 2003 National Assessment of Adult Literacy. U.S. Department of Education, Washington, DC; 2006. 10.1592/phco.22.5.282.33191
5. Weiss BD: Health Literacy A Manual for Clinicians. American Medical Association, Chicago, IL; 2003.
6. Agarwal N, Hansberry DR, Sabourin V, Tomei KL, Prestigiacomo CJ: A comparative analysis of the quality of patient education materials from medical specialties. JAMA Intern Med. 2013, 173:1257-1259. 10.1001/jamainternmed.2013.6060
7. Herndon JB, Chaney M, Carden D: Health literacy and emergency department outcomes: a systematic review. Ann Emerg Med. 2011, 57:334-345. 10.1016/j.annemergmed.2010.08.035
8. Institute of Medicine (US) Committee on Health Literacy: Health Literacy: A Prescription to End Confusion. Nielsen-Bohlman L, Panzer AM, Kindig DA (ed): National Academies Press (US), Washington D.C; 2004. 10.17226/10883
9. Chan CV, Kaufman DR: A framework for characterizing ehealth literacy demands and barriers. J Med Internet Res. 2011, 13:94. 10.2196/jmir.1750
10. Edmunds M, Barry R, Denniston A: Readability assessment of online ophthalmic patient information. JAMA Ophthalmol. 2013, 131:1610-1616. 10.1001/jamaophthalmol.2013.5521
11. Misra P, Agarwal N, Kasabwala K, Hansberry DR, Setzen M, Eloy JA: Readability analysis of healthcare-oriented education resources from the American Academy of Facial Plastic and Reconstructive Surgery. Laryngoscope. 2013, 123:90-96. 10.1002/lary.23574
12. Hansberry DR, John A, John E, Agarwal N, Gonzales SF, Baker SR: A critical review of the readability of online patient education resources from radiologyinfo.org. Am J Roentgenol. 2014, 202:566-575. 10.2214/AJR.13.11223
13. Hansberry DR, Patel SR, Agarwal P, Agarwal N, John ES, John AM, Reynolds JC: A quantitative readability analysis of patient education resources from Gastroenterology Society websites. Int J Colorectal Dis. 2017, 32:917-920. 10.1007/s00384-016-2730-3
14. How to write easy-to-read health materials. (2012). Accessed: October 12, 2018: http://www.nlm.nih.gov/medlineplus/etr.html.
15. Annals of internal medicine. (2018). Accessed: September 20, 2018: http://annals.org/aim/pages/patient-information.
16. Flesch R: A new readability yardstick. J Appl Psychol. 1948, 32:221-233.
17. Kincaid JP, Fishburne Jr RP, Rogers RL, Chissom BS: Derivation of New Readability Formulas (Automated Readability Index, Fog Count and Flesch Reading Ease Formula) for Navy Enlisted Personnel. Institute for Simulation and Training, University of Central Florida, Millington, TN; 1975.
18. Gunning R: The Technique of Clear Writing. McGraw-Hill, New York, NY; 1952.
19. Coleman M, Liau TL: A computer readability formula designed for machine scoring. J Appl Psychol. 1975, 60:283-284. 10.1037/h0076540
20. Mc Laughlin GH: SMOG grading-a new readability formula. J Read. 1969, 12:639-646.
21. Doak CC, Doak LG, Root JH: Teaching Patients with Low Literacy Skills. J.P. Lippincott Company, Philadelphia, PA; 1996.
22. Baker DW: The meaning and the measure of health literacy. J Gen Intern Med. 2006, 21:878-883. 10.1111/j.1525-1497.2006.00540.x
23. Berkman ND, Sheridan SL, Donahue KE, Halpern DJ, Crotty K: Low health literacy and health outcomes: an updated systematic review. Ann Intern Med. 2011, 155:97-107. 10.7326/0003-4819-155-2-201107190-00005
Original article
peer-reviewed

### Author Information

###### Ethics Statement and Conflict of Interest Disclosures

Human subjects: All authors have confirmed that this study did not involve human participants or tissue. Animal subjects: All authors have confirmed that this study did not involve animal subjects or tissue. Conflicts of interest: In compliance with the ICMJE uniform disclosure form, all authors declare the following: Payment/services info: All authors have declared that no financial support was received from any organization for the submitted work. Financial relationships: All authors have declared that they have no financial relationships at present or within the previous three years with any organizations that might have an interest in the submitted work. Other relationships: All authors have declared that there are no other relationships or activities that could appear to have influenced the submitted work.

Original article
peer-reviewed

### Figures etc.

6.4
###### RATED BY 2 READERS

Scholary Impact Quotient™ (SIQ™) is our unique post-publication peer review rating process. Learn more here.