Predictors of Citation Rates in High-Impact Glioblastoma Clinical Trials

Clinical trials are at the top of research study designs and tend to attract high citation numbers. Glioblastoma multiforme (GBM) is a multidisciplinary disease that continues to be the subject of peak research interest. In general, the literature relating to the predictors of citation rates in clinical trials remains limited. This review aims to identify the factors that influence citation numbers in high-impact GBM clinical trials. The 100 most cited GBM trials of any phase published from 1975 to 2019 were selected and reviewed. The primary analysis correlated citation numbers of articles with various trial and publication-related predictors using the Pearson correlation coefficient. The secondary analysis compared the mean citation numbers for different subgroups using the mean difference test. The median (range) citation number for the selected 100 trials was 349 (135-16,384). The primary analysis showed a significant correlation between citation numbers of articles and the study population (P = 0.024), trial phase (I-III) (P = 0.0427), and the impact factor (IF) of the journal (P < 0.0001). The secondary analysis demonstrated significantly higher mean citation numbers in all trials with the following features: study population ≥115 (P = 0.0208), phase III (P = 0.0372), treatment protocol including radiotherapy (P = 0.0189), temozolomide (TMZ) therapy (P = 0.0343), IF of the journal ≥14.9 (P = 0.02), and general medical journals (P = 0.28). We conclude that the most significant predictors of citation rates in high-impact GBM trials were the study population, trial phase, and journal’s IF. The treatment protocol was a positive predictor when it included the currently widely accepted treatment modalities (radiotherapy and TZM). Randomization, age of publication, as well as the numbers of arms, authors, centers, countries, and references were not significant predictors. Increasing awareness of the factors that could affect citations may help researchers undertaking clinical trials to enhance the academic impact of their work.


Introduction And Background
Citation-based metrics are used for calculating the impact factor (IF) of journals and for evaluating the academic productivity of researchers. The number of citations an article receives, also referred to as the citation rate, is arguably the most important measure of a study's impact and clinical weight [1]. An analysis of the various article, journal, and author-related factors that may affect citation rates was reported in two publications [2,3]. These factors were also examined by other studies that focused on identifying the predictors of citations in published research relating to several specialties, including spine [4], neurosurgery [1], radiology [5], psychology [6], plastic surgery [7], cardiovascular [8], urology [9], and orthopedic surgery [10].
Randomized controlled trials (RCTs) are recognized as the pinnacle of clinical study designs and evidencebased medicine [11]. They are frequently published in high-impact journals and receive considerable visibility [11]. They are also likely to influence the opinions of clinicians, patients, and policymakers [11]. The association between study designs (RCTs and meta-analyses, in particular) and high citation numbers has been well documented in the literature [2,[8][9][10]. However, clinical trials are not always randomized and vary in characteristics, completion, and publication rates [12]. Furthermore, studies analyzing citation patterns of clinical trials remain limited in the literature [11,[13][14][15].
Glioblastoma multiforme (GBM) is a malignant primary central nervous tumor that represents an enigma to clinicians because of its aggressive and heterogeneous nature [16]. It is primarily a topic of oncology but includes the disciplines of neurosurgery, neurology, radiotherapy, basic science, and general medicine [16]. A recent bibliometric evaluation of high-impact GBM research did not address citation rates [16]. The objective of this review is to determine the different trial and publication-related predictors of citations in high-impact GBM clinical trials.

Methodology
PubMed and Google Scholar databases were searched in March 2021 for all GBM-related trials available in the literature. The inclusion criteria included highly cited clinical trials at any phase published from 1975 to 2019. We also searched the websites of the following journals: the New England Journal of Medicine, Lancet, Journal of American Medical Association (JAMA), Journal of Clinical Oncology, Neuro-oncology, and Journal of Neurosurgery. The main keywords for the literature search were "Glioblastoma," "GBM," "Glioblastoma Multiforme," "Grade IV Glioma," "Trials," and "Randomized Controlled Trials." Articles were assessed for suitability using the abstract, and the full text was reviewed in case of ambiguity. Using article citation numbers provided by Google Scholar, the 100 most cited GBM trials were identified. Trials that reported extended findings from an earlier trial were included if they received high citation numbers. To minimize bias, two authors conducted independent searches and prepared separate lists of the most cited articles. The two lists were compared, and any discrepancies were resolved by consensus. In view of the regular changes in citation numbers, the search findings on a single day (April 01, 2021) were documented and used for analysis. In addition, journal IFs were obtained from the journals' websites for 2019 as these were the latest available at the time of the analysis. The selected trials were analyzed, and the information relating to the characteristics of the trials and publications was collected. The data were used to generate descriptive statistics relating to the 100 high-impact GBM clinical trials.
The primary analysis correlated the total citation numbers for the various studies with the following trial and publication-related predictors: study population; randomization; the number of arms; phase; GBM status; treatment modality used in any of the trial arms [chemotherapy (including temozolomide (TZM), nitrosourea, bevacizumab (BVZ), others); radiotherapy (including electrotherapy and proton/neutron irradiation); surgery; local treatment (chemotherapy, immunotherapy, hyperthermia) and immunotherapy]; trial duration in months; duration from publication in years; publishing journal's IF and field (oncology, general medicine, neurosurgery); and the number of authors, centers, countries, and references listed on the publication. The correlation analysis was done by calculating the Pearson correlation coefficient (R) using Social Sciences Statistics [17], and significance was determined at a P-value of ≤0.05.
For further evaluation of the impact of the chosen predictors, a secondary analysis was conducted by calculating and comparing the mean citation numbers [±standard deviation (SD)] between different subgroups. The median was taken as a cut-off point in the numerical parameters, with the following comparisons: study population [<115 vs. ≥115]; randomization (yes vs. no); arms (1 vs. 2-4); phase (I, I-II, II, II-III vs. III); GBM status (newly diagnosed vs. recurrent); treatment modality [chemotherapy vs. all others, TZM vs. all others, nitrosourea vs. all others, BVZ vs. all others, radiotherapy vs. all others, surgery and local treatment vs. all others, and immunotherapy vs. all others]; study duration in months (<30 vs. ≥30); duration from publication in years (<13 vs. ≥13); journal's IF (<14.9 vs. ≥14.9); journal's field (general medicine vs. others, oncology vs. others); the number of authors (<14 vs. ≥14); the number of centers (<10 vs. ≥10); the number of countries (1 vs. >1); and the number of references (<30 vs. ≥30). The statistical analysis was carried out by calculating the mean difference (MD) using MedCalc [18], and significance was determined at a P-value of ≤0.05.

Results
The median (range) and mean (±SD) total citation numbers for the 100 most cited GBM trials were 340 (135-16,284) and 825 (±1,828), respectively. An analysis of the trials is shown in Appendices. The median (range) and findings relating to the various trial and publication parameters are summarized in Tables 1, 2. The 100 trials were published in the following journals: Journal of Clinical Oncology, 22%; Lancet Oncology and Lancet, 12%; Neuro-Oncology, 10%; New England Journal of Medicine, 8%; Journal of Neurosurgery, 8%; International Journal of Radiation Oncology Biology Physics, 5%; JAMA and JAMA Oncology, 3%; Nature and Nature Medicine, 3%; British Journal of Cancer, 3%; Clinical Cancer Research, 3%; and miscellaneous, 23%. The mean IF for oncological, general medical, and neurosurgical journals were 20.1, 40.5, and 4, respectively. Of the selected 100 trials, the treatment protocols included chemotherapy using one or more agents in 73% (TZM: 31%, nitrosourea: 17%, BVZ: 12%, erlotinib and gefitinib: 7%, irinotecan: 5%, and others: 30%), radiotherapy in 38% (including tumor treatment fields: 2%, photodynamic therapy: 2%, accelerated proton/photon irradiation: 1%, and neutron capture therapy: 1%), surgery and local treatment in 13%, and immunotherapy in 9%. Tables 1, 2 summarize the primary analysis correlation results between citation numbers and the various predictors. A significant correlation was observed between citation numbers and study population (R = 0.226; P = 0.024), journal's IF (R = 0.4085; P < 0.0001), and trial phase (R = 0.2031; P = 0.0427). No significant correlation was found between citation numbers and randomization, the number of trial arms, GBM status, treatment protocols, trial duration, duration from publication, and the number of authors, centers, countries, and references.    Table 3 summarizes the results of the mean difference comparative secondary analysis between the various subgroups. The value of the SD was greater than the mean in most reported findings in Table 3, indicating that the data had skewed distribution due to the wide range of variation in citation numbers among the selected articles. A significantly higher mean citation number was observed for trials with a study population ≥115 compared to <115 (1,248 vs. 404; P = 0.0208), trials that reported phase III results compared to others (1,223 vs. 460; P = 0.0372), trials in which the treatment protocol included radiotherapy compared to others (1,423 vs. 519; P = 0.0189), trials in which the treatment protocol included TZM compared to others (1,414 vs. 573; P = 0.0343), trials that were published in journals with IF ≥14.9 compared to <14.9 (1,251 vs. 402; P = 0.02), trials that were published in high-impact general medicine journals compared to others (1,521 vs. 594; P = 0.28). No significant difference was found in the mean citation numbers between the two subgroups relating to randomization, the number of trial arms, GBM status, treatment protocols that included chemotherapy in general, nitrosourea, BVZ, surgery, local treatment, and immunotherapy, as well as trial duration, the period from publication, and the number of authors, centers, countries, and references.

Discussion
GBM has been the focus of substantial clinical and scientific research aimed at discovering a treatment modality that can significantly improve survival [13]. A recently reported analysis of the 100 most cited GBM publications included 27 clinical studies, 19 of which were trials that were included in this study (Appendices Table 4) [1]. A review of the 44 neurosurgical RCTs in high-impact journals that were published did not contain any GBM trials [11]. None of these publications assessed citation patterns.
The mean citation number for the 100 most cited GBM trials was 825, which was higher than the mean citation of 198 for the 100 most cited meningioma articles [19]. However, it is slightly lower than the reported median citation number of 935 for the 100 most cited GBM articles [16]. This finding is not surprising as the mentioned review covered a bigger pool of GBM studies that included 52 basic science articles [16]. The latter are recognized to be associated with high citation numbers [2,10,16]. Variation in citation rates according to study topic or subject is well recognized in the literature relating to neurosurgery [1], spine [4], plastic surgery [7], and urology [9]. It is generally accepted that disciplines differ in their citation practices and that certain topics or subject areas may be cited more than others [2]. Moreover, the number of citations is influenced by the size of the literature in the field [2].
In this analysis, a significant association with the study population was observed in both primary and secondary analysis, implying that the study population was a firm predictor of citation rates in GBM clinical trials. Similar findings relating to the study population were reported by other studies [2,4,6,9,20]. A significant correlation with the trial phase was also seen in both primary and secondary analysis, indicating that being a phase III trial was a solid predictor of citation rates. However, citation rates were not affected by randomization which is surprising as the correlation between RCT-type studies and bigger citation numbers is well reported in the literature [1,[8][9][10]. This finding could be unique to the field of GBM research or could be related to the relatively limited number of articles selected in this review. Citation rates were not affected by the number of arms, status of GBM, and duration of the trial. The lack of impact of certain features of study designs on citation numbers was also reported by others [2,20].
In this study, the primary analysis did not reveal a correlation between treatment protocols and citation rates. However, the secondary analysis demonstrated significantly higher citation rates in trials in which the treatment protocol included radiotherapy and TZM. This probably reflects the current widely accepted standard treatment for newly diagnosed GBM, which includes surgery followed by concurrent radiotherapy with TZM and further adjuvant TZM [21,22]. No significant association was observed between citation rates and treatment protocols that included chemotherapy in general. This probably relates to the wide-ranging chemotherapeutic agents used in the studies and their mixed efficacy. Furthermore, chemotherapy in the general group included older studies that were conducted before the use of standard TZM in the first-line setting. The lack of association between citation numbers and treatment protocols including BVZ, nitrosourea, surgery and local treatment, and immunotherapy may be influenced by the limited number of trials that focused on the treatment modality. However, it could reflect their undetermined role in the management of GBM [21,22]. Citation rates were also not affected by the duration from the time of publication (age of the study). This is not unusual as the study covered a long period (43 years). It is recognized that the number of citations increases in the first year after publication to reach a peak and then they are less cited as time passes [2]. The latter could be because the article's information becomes outdated with time [2].
In this article, significant association with journal's IF was observed in both primary and secondary analyses showing that journal's IF was a strong predictor of citation rates in GBM clinical trials. Similar findings relating to the journal's IF were reported by other studies [1,2,15]. Furthermore, the secondary analysis demonstrated significantly higher citation rates in trials that were published in general medical journals. This was expected as the group of general medical journals in this study had a much higher mean IF than the oncological and neurosurgical groups (40.5 vs. 20.1 and 4, respectively). The association between journals and higher citations rates was documented in the literature in association with spine [4], plastic surgery [7], and transplantology [23].
In this review, no significant link was found between citation rates of GBM clinical trials and the numbers of authors, centers, countries, and references. A similar finding was reported by other studies [2,20]. However, in the literature, several publications have identified the number of authors as a significant predictor of citation [1,7,15]. Significant relationships have also been reported between the international and national collaboration of authors, the number of organizations, the number of countries producing the paper, and the frequency of citations [1,2]. A positive relationship with the number of references was reported by some studies [20]. Furthermore, some studies suggested that a proportion of variance in the number of citations an article receives can be explained by seemingly superficial factors unrelated to the content of the article such as the title, the number of authors, the number of references, the number of sentences in the abstract, the presence of a colon in the title, and the number of pages [24,25].
The countries where the GBM clinical trials originated were not examined in this study. It has been reported that the country of origin can be a positive predictor of citation counts in research relating to spine [4], radiology [5], and urology [9]. Furthermore, a recent publication [3] investigated the impact of 66 factors on citations using samples of articles from 18 leading Chinese library and information science journals. They found 46 factors were significantly associated with citations. They also observed the most significant factors to be the number of downloads, the number of citations in the first five years, the author being an independent researcher, and the percentage of monographs in the references. Several other potential predictors were not addressed in this study that were examined by other studies. These include increasing visibility through open access [5], selection for press release [26], funding [14,20], disclosure of conflict of interest [7], statistically significant results [20], and the trial being referenced in ClinicalTrials.gov [27].

Limitations
There are several limitations to this study. The study relied on the precision of online search engines PubMed and Google Scholar. The selection of the 100 trials was based on their total citations at a certain point which was likely to change relatively quickly. This could have influenced the inclusion or exclusion of a few of the lower-impact trials. The wide duration from publication may have affected the citations of older trials. There may have been potential errors in the subgrouping of the treatment protocols. Moreover, variation in author affiliation may have affected the number of centers. Collaborators were not counted in the number of authors, centers, and countries. In addition, the impact of self-citation on citation numbers was not examined.

Conclusions
Clinical trials are the pinnacle of research study designs and tend to attract high citation numbers. GBM is a multidisciplinary disease that continues to be a subject of peak research interest. The literature relating to the predictors of citation rates in clinical trials remains limited. The most consistent predictors of citation rates in GBM clinical trials were study population, trial phase, and journal IF. The treatment protocol was a positive predictor when it included the currently widely accepted treatment modalities (radiotherapy and TZM). Randomization, age of publication, as well as the numbers of arms, authors, centers, countries, and references were not significant predictors. Increasing awareness of the factors that could affect citations may help researchers undertaking clinical trials to enhance the academic impact of their work. Further research on the predictors of citations in trials related to other pathological entities is encouraged.

Conflicts of interest:
In compliance with the ICMJE uniform disclosure form, all authors declare the following: Payment/services info: All authors have declared that no financial support was received from any organization for the submitted work. Financial relationships: All authors have declared that they have no financial relationships at present or within the previous three years with any organizations that might have an interest in the submitted work. Other relationships: All authors have declared that there are no other relationships or activities that could appear to have influenced the submitted work.