Content Analysis and Characterization of Medical Tweets During the Early Covid-19 Pandemic

Objective The novel coronavirus disease 2019 (Covid-19) has infected millions worldwide and impacted the lives of many folds more. Many clinicians share new Covid-19-related resources, research, and ideas within the online Free Open Access to Medical Education (FOAM) community of practice. This study provides a detailed content and contributor analysis of Covid-19-related tweets among the FOAM community during the first months of the pandemic. Design, Setting, and Participants In this social media content analysis study, Twitter was searched from November 1, 2019, to March 21, 2020, for English tweets discussing Covid-19 in the FOAM community. Tweets were classified into one of 13 pre-specified content categories: original research, editorials, FOAM resource, public health, podcast or video, learned experience, refuting false information, policy discussion, emotional impact, blatantly false information, other Covid-19, and non-Covid-19. Further analysis of linked original research and FOAM resources was performed. One-thousand (1000) randomly selected contributor profiles and those deemed to have contributed false information were analyzed. Results The search yielded 8541 original tweets from 4104 contributors. The number of tweets in each content category were: 1557 other Covid-19 (18.2%), 1190 emotional impact (13.9%), 1122 FOAM resources (13.1%), 1111 policy discussion (13.0%), 928 advice (10.9%), 873 learned experience (10.2%), 424 non-Covid-19 (5.0%), 410 podcast or video (4.8%), 304 editorials (3.6%), 275 original research (3.2%), 245 public health (2.9%), 83 refuting false information (1.0%), and 19 blatantly false (0.2%). Conclusions Early in the Covid-19 pandemic, the FOAM community used Twitter to share Covid-19 learned experiences, online resources, crowd-sourced advice, and research and to discuss the emotional impact of Covid-19. Twitter also provided a forum for post-publication peer review of new research. Sharing blatantly false information within this community was infrequent. This study highlights several potential benefits from engaging with the FOAM community on Twitter.


Introduction
Millions of cases of coronavirus disease 2019 (Covid -19) have been reported globally since the first known case in December 2019 [1][2]. Covid-19's worldwide impact has been recognized through its classification as a global pandemic by the World Health Organization (WHO) [3]. Covid-19's rapid spread has spurred healthcare workers, researchers, and members of the public to search for accurate and up-to-date information online. The rate of new Covid-19-related research, however, has challenged conventional methods of scientific knowledge dissemination (e.g. peer-reviewed journals), which do not always publish on rapid timelines [4]. In response, clinicians worldwide have turned to social media to debate new research while sharing their experiences and resources [5].
Social media use among clinicians is not a new phenomenon. In the past decade, an online community of practice has developed with the goal of sharing ideas, research, and learned experiences through freely published online resources [6][7]. Termed "Free Open Access to Medical Education" (FOAM), this movement has become a valuable resource for healthcare professionals and medical learners [7][8]. In addition to relaying explicit medical knowledge, it may also be an effective medium for transmitting tacit knowledge (experiential or process-based knowledge) [9]. Compared to traditional peer-reviewed journals, FOAM has variable publication and editorial processes relying heavily on post-publication peer review [10].
Optimizing knowledge translation is important during a pandemic, as critical decisions need to be made with limited evidence, and potentially practice-changing research can be published at any time. Within the FOAM community, Twitter is the most widely used social media platform to share ideas and discuss new research on Covid-19 [5]. On Twitter, contributors generate 'tweets' of up to 280 characters in length that can be tagged with searchable hashtags (#) and can include images, website links, and documents. While the important role of Twitter during Covid-19 has been recognized by the scientific community [5], a detailed characterization of its use, strengths, and limitations, including accuracy of content, is needed. This is particularly important for a publicly accessed platform like Twitter that may be susceptible to misleading or false information.
The objective of this study was to characterize Covid-19-related Twitter use by the FOAM community and to describe its content, trends, and contributors. In addition, the potential role of Twitter in spreading misinformation was assessed. This research represents an important first step in evaluating Twitter as a platform for knowledge translation during rapidly evolving healthcare crises.

Materials And Methods
Research ethics board approval for research involving publicly available data is not required at our institutions. Our protocol was registered on the Open Science Framework (OSF) prior to the initiation of data collection (https://osf.io/3tx96/). The original data are also published on OSF. The study has been reported in keeping with the Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) statement [11]. Patients or the public were not involved in the design, conduct, reporting, or dissemination plans of our research.

Search strategy
We searched Twitter on March 21, 2020, for tweets with relevant hashtags from November 1, 2019 (the month preceding the first reported Covid-19 case in Wuhan, China) to March 21, 2020, using a commercially available hashtag collating tool, Tweet Binder (Pamplona, Spain). The period between November 2019 to March 2020 was chosen, as it represented a time period when little consolidated information on Covid-19 was available to healthcare professionals despite concern surrounding Covid-19 being high. Hashtags were selected by consensus of the authors, several of whom were clinician members of the FOAM community. Tweets were included if they contained both a hashtag commonly used by healthcare professionals to discuss FOAM topics (#FOAMed or #meded or #POCUS or #FOAMcc or #medtwitter) and a hashtag used to discuss Covid-19 (#Covid19 or #coronavirus or #Covid or #Covid-19). Alternatively, two Covid-19 FOAMspecific hashtags were also included (#Covid4MDs or #CovidFOAM). The search strategy was not casesensitive.

Tweet analysis
We extracted the following data: total number of original tweets (original text or image content), retweets (a reposted tweet without modification), reach (number of unique people who saw the tweet), impressions (number of times a tweet was liked or retweeted), the total number of contributors (accounts creating tweets), and median original tweets per original contributor.
The content of all original, English-language tweets was analyzed independently by one of five authors (MP, SB, KW, NN, RP). To assess inter-rater reliability, a duplicate extraction of 100 tweets was performed by all extractors. We assigned each tweet to one of 13 pre-determined 'content categories' created after consensus discussion between authors: 1) peer-reviewed original research study related to Covid-19, excluding editorials, commentaries, or perspective articles; 2) editorial, commentary, or perspective article published in a journal or repository relevant to Covid-19, including scientific article pre-prints; 3) FOAM resources pertaining to the care of Covid-19 patients; 4) public health agency website or university website (e.g. Centers for Disease Control and Prevention); 5) medical podcast or video relevant to Covid-19; 6) personal or learned experience caring for Covid-19 patients; 7) a statement or discussion refuting blatantly false or misleading information regarding Covid-19; 8) a discussion about policy or public health measures related to Covid-19; 9) a discussion of the personal or emotional impact of Covid-19; 10) a tweet that provided blatantly false or misleading information (defined through consensus agreement by two senior authors (RP and RU) that the tweet contained false or misleading information based on current medical consensus); 11) a tweet asking for advice or for others to share experience caring for Covid-19 patients; 12) other Covid-19 related tweets that did not fit in the other categories; 13) non-Covid-19-related tweets. The final category was included for tweets that used the aforementioned hashtags but mentioned content entirely unrelated to Covid-19. The overall best fitting category was selected if multiple classifications were possible, and consensus discussion was allowed if needed. Once categorized, we calculated the number and percentage of tweets in each content category by day and week.

Contributors, original research, and FOAM content
We determined the demographics of Covid-19 FOAM contributors by reviewing 1000 random profiles of the contributors whose tweets were captured in the search strategy, extracting: the number of followers, total tweets, country of residence, and contributor source (institution, nurse, staff physician, resident physician, medical student, respiratory therapist, pharmacist, other healthcare professional, non-healthcare professional, non-clinician researcher, healthcare-related group). The profiles were randomized and selected using the randomize function in Microsoft Excel (Microsoft Corporation, Redmond, WA). We performed a similar analysis on the contributor profiles whose tweets were flagged as blatantly false or misleading.
To evaluate the dissemination of original research via Twitter, we analyzed the journal of publication, country of the corresponding author, article type (epidemiological study, intervention study, diagnostic study, basic science, case series, or other), and the median number of days between online publication (either pre-publication or online) and tweet for each included article. To ensure a focus on new research being conducted on Covid-19 (as opposed to previous coronavirus infections), we excluded research articles published before 2020.
We also identified tweets that linked to FOAM resources and the source (website), type of resource, number of tweets, including the resource, and the median number of days between FOAM publication online and the tweet.

Data analysis
We saved the extracted data in Microsoft Excel 2013 (Microsoft Corporation) and analyzed it using R version 3.6.2 (R Project for Statistical Computing, R Core Team, Vienna, Austria). When appropriate, we assessed the distribution of our data using a Shapiro-Wilks test and calculated the mean (+/-standard deviation) for normally distributed data and median (1st and 3rd interquartile range) for data that were not normally distributed. A post-hoc Mann-Whitney U test was performed to compare the days between publication of original research or FOAM resource and tweet. A post-hoc Mann-Kendall trend test with a Bonferroni correction was used to assess for a trend in the percentage of total tweets represented by each content category per week. Inter-rater reliability was assessed using Fleiss' Kappa. Statistical significance was set at a p-value of less than or equal to 0·05.

Results
The first tweet matching the search criteria was on January 19, 2020, and from then until March 21, 2020, 74,758 original tweets and retweets from 52,917 contributors were created. Of these, 8819 (11.8%) were original tweets created by 4104 contributors, and 65,490 (88.2%) were retweets ( Table 1). We excluded 278 tweets because they were not written in English or contained broken links. Of the remaining 8541 (11.4%) original tweets, 5494 were standalone tweets, 1039 were replies, and 2008 were retweets with comments ( Figure 1). The original tweets and retweets reached 95,072,663 Twitter users and had a total of 388,701 likes or replies. Contributors to original tweets had a median number of 489 (interquartile range (IQR) 144, 1601) followers and published a median of one (IQR 1, 2) original tweet. A Shapiro-Wilks test showed the data was non-parametric (p < 0.001).

Content analysis
The number of tweets per day in each content category is displayed in Figure 2.

Dissemination of original research
There were 275 tweets that linked to 157 unique peer-reviewed original research articles. Of these, 23 were published before January 1, 2020, and were excluded from the analysis.

Discussion
Covid-19 is not the first pandemic where Twitter has played an important role in sharing information. The H1N1 pandemic and Ebola epidemic were widely discussed on Twitter, with millions of original tweets [12][13]. When compared to the H1N1 pandemic in 2009, however, the number of tweets discussing Covid-19 tweets has increased exponentially, surging in early March 2020 ( Figure 2) [13]. For context, at that time, Europe had Covid-19 cases across the continent, with several hundred deaths in Italy alone [14]. North America also had its first significant outbreak in Washington, USA [15]. Although the FOAM community has contributors worldwide, most tweets were created by North American contributors, which may account for the temporal association between Twitter use and western outbreaks of Covid-19.
Given the immediacy and reach of social media, the FOAM community may be ideally situated to share medical resources during a pandemic. We found that more than one in four tweets contained a link to a Covid-19 resource. Compared to traditional publication peer review, the publication and editorial processes of FOAM resources vary [10,16]. FOAM relies on transparent and open post-publication peer review where other contributors can discuss, critique, and sometimes even contribute to resources [7]. One illustration of effective post-publication peer review during the Covid-19 pandemic has been the Internet Book of Critical Care (IBCC) chapter on Covid-19 [17]. From March 2 to April 16, 2020, the IBCC received over 2·1 million views, with over 180 comments contributing to post-publication peer review [18]. This has led to numerous revisions of the chapter to incorporate new evidence. To consolidate and share the vast amount of information being generated during a pandemic, open post-publication peer review may help balance the timely dissemination of content whilst ensuring its accuracy and quality.
In addition to the speed and reach of FOAM publication, social media may be particularly effective in sharing tacit knowledge [9]. This stems in part from its ability to facilitate discussions and story-telling, which are key components of tacit knowledge translation [9]. During Covid-19, the geographic progression of the disease from Asia, to Europe, to North America allowed for clinicians to share their experiences managing Covid-19 patients. In our study, we found that 9.8% percent of original tweets shared learned experiences and 10·8% represented individuals asking for crowd-sourced advice. Many of these early tweets pertained to the diagnosis of Covid-19 infection, ultrasound use, airway management, personal protective equipment, and mechanical ventilation. These resources may have helped clinicians and organizations to prepare for Covid-19.
With the unprecedented amount of new Covid-19 research being published, it seems increasingly difficult to find accurate and reliable content online. This has been termed an "infodemic' by the WHO [19]. Although Twitter might contribute to this through the indiscriminate sharing of information, it can also flag important new research and give clinicians a forum to openly critique it. In our study, we found that contributors rapidly tweeted new Covid-19 research, with a median time between publication and tweet of two days. The majority were published in high-impact journals and included important early studies on Covid-19. The immediacy of discussion and rebuttal provided through Twitter also may be valuable, especially when many studies are first being published as pre-prints and have not undergone peer review. For example, when Gautret et al. published their initial pre-print advocating for the use of hydroxychloroquine and azithromycin as a treatment for Covid-19 [20], many FOAM contributors called for more rigorous studies with patient-important outcomes before widespread adoption ( Table 2). Their criticisms were substantiated with recent studies showing increased adverse events and a potential association with mortality for hydroxychloroquine [21][22].
Contributing to and participating in the FOAM community is not without risks, and the unwritten rule is caveat emptor (buyer beware). Across social media, the potential for receiving misinformation is real and significant [23]. In this study, 19 contributors contributed blatantly false or misleading information; however, this represented only 0·2% of the total number of tweets in this analysis. Whereas blatantly misleading tweets are relatively easy to identify, a significant concern is when a reader is misled through either misrepresentation of opinion as fact, sensational anecdotes, or providing content without context. Furthermore, subtly misleading or incorrect information may prove much more challenging to interpret and is a serious and ongoing concern when using open-sourced information. A study by Kouzy et al. found that one in four tweets about Covid-19 across Twitter (no FOAM hashtags) contained misinformation [24]. We suspect the rate of subtle misinformation in the FOAM community is higher than the 0·2% found in this study; however, given that the community's collective goal is to share legitimate knowledge, it is likely lower than the broader Twitter community.
FOAM has the potential to decrease the knowledge translation gap during Covid-19; however, resources may be of variable quality [25]. Readers are responsible for critically appraising online content; however, locating quality resources, to begin with, can be a challenge. The Social Media Index (SMI) provides a list of FOAM websites that are both impactful [26] and high quality [27], analogous in some ways to a journal's impact factor. When reading these resources, the Academic Life in Emergency Medicine (ALiEM) AIR tool or revised Medical Education Translational Resources: Impact and Quality (METRIQ) tool has been validated to help determine the quality of content [28][29]. Readers may use these tools as an aid when navigating unfamiliar FOAM resources to better appraise the quality of the online resource.
Despite potential pitfalls from engaging with the FOAM community, it is a vibrant community that contributes to the timely dissemination and translation of medical knowledge throughout the world. Few other media, if any, are capable of sharing information, crowdsourcing advice, and providing warnings more rapidly; again, supported by the fact that new articles and FOAM resources discussing Covid-19 were shared within several days of online publication. In situations where efficient communication is essential, such as in global pandemic, natural disasters, or political upheavals, the benefits of such a system may outweigh the possibility of misleading information. It is also important to recognize that papers published in traditional print journals are not immune to misleading information and bias, and may have their own 'Spin' or biased interpretation of results. Furthermore, Twitter allows for tailored advice to individuals through the ability to share personal experiences and ask specific questions, as was seen in our study. With these potential benefits, we feel that Twitter FOAM augments knowledge translation achieved through conventional scientific publications and will continue to do so in the years to come.

Limitations
Our study has several limitations. Although a broad hashtag search strategy was used, some FOAM hashtags were not included. As well, many Twitter contributors do not add hashtags to all of their Tweets, meaning that some tweets that would have been relevant were missed. For example, it is possible that certain tweets with relevant medical information containing only Covid-19-related hashtags could have been missed. Additionally, although the search was up to date as of March 21, 2020, the necessary time for analysis and manuscript preparation means it does not reflect current Twitter use. As well, inter-rater reliability was only fair, likely related to the fact some tweets fell into several categories. This may account for the reason why a fair proportion of tweets were classified as "other Covid-19." Finally, during the initial months of Covid-19, the largest number of cases were in Asia and Europe [1,30], yet the study was limited to English tweets, potentially selecting for a North American or Anglo-biased perspective.

Conclusions
In the age of social media, many clinicians use Twitter to share resources and ideas with the goal of improving care for their patients. Twitter is effective in disseminating information; however, it comes with challenges in ensuring content is accurate and relevant. This study provides a detailed characterization of early medical tweets during the Covid-19 pandemic and represents a first step in understanding Twitter use among the FOAM community during the Covid-19 pandemic. Further work is required to improve Twitter as a knowledge translation tool both for Covid-19 and future global crises, such that misinformation and bias is minimized and factual knowledge dissemination maximized. Covid-19 has united clinicians around the world, and perhaps more than ever, effective strategies for sharing new ideas, accurate information, and quality research are needed.

Additional Information Disclosures
Human subjects: All authors have confirmed that this study did not involve human participants or tissue. Animal subjects: All authors have confirmed that this study did not involve animal subjects or tissue.

Conflicts of interest:
In compliance with the ICMJE uniform disclosure form, all authors declare the following: Payment/services info: All authors have declared that no financial support was received from any organization for the submitted work. Financial relationships: All authors have declared that they have no financial relationships at present or within the previous three years with any organizations that might have an interest in the submitted work. Other relationships: All authors have declared that there are no other relationships or activities that could appear to have influenced the submitted work.