Fragility Index of Recently Published Meta-Analyses in Pediatric Urology: A Striking Observation

Background and objective Fragility Index (FI) of meta-analyses determines their stability in terms of the level of confidence and strength behind the results depicted by them. The present study was conducted to estimate the FI of recently published meta-analyses in the Journal of Pediatric Urology (JPUrol). Method Twenty recently published articles on meta-analyses were screened to identify the eligible ones. The baseline data of each meta-analysis including the details of the author, number of included studies, total sample size, the total number of events, the status of the overall outcome (significant or non-significant), type of effect measure, type of method used for pooling the estimates, and type of effects model were recorded. FI was calculated by doing each single status modification. The 95% CI of the treatment effect was re-calculated until the statistical significance of the meta-analysis was reversed. Results A total of seven articles incorporating 22 meta-analyses were included. Seven (32%) of them had a statistically significant outcome prior to FI estimation. The risk ratio (17/22; 77%) was the most commonly used effect measure. The random-effects model (15/22; 68%) and the Mantel-Haenszel method (20/22; 91%) of pooling the estimates were utilized in the majority of meta-analyses. The median (Q1-Q3; range) FI of statistically significant, non-significant, and total meta-analyses were 5 (3-19.5; 2-39), 5 (3.5-6; 1-17), and 5 (3-13; 1-39) respectively. FI of ≤5 was noticed in four out of seven (57%), 9/15 (60%), and 13/22 (59%) of these meta-analyses respectively. Conclusion Based on our findings, the majority of the recently published meta-analyses in the field of pediatric urology are fragile and depend upon the event status of ≤5 participants.


Introduction
The first meta-analysis in medical literature was published by Karl Pearson in 1904 [1]. Since then, the number of published meta-analyses has been increasing every year. In fact, more than two lakh metaanalyses have been published on PubMed between 1966-2021. Due to this rapid increase in their number, it has become difficult to properly gauge the quality and reliability of the information disseminated by these meta-analyses [2]. The Fragility Index (FI) is one of the recently developed tools to scrutinize the strengths and robustness of clinical trials based on the highest level of evidence. FI of a meta-analysis is defined as the minimum number of changes in the event status (event to non-event and vice versa) in one or more included studies that are required to reverse the significance status (significant to non-significant and vice versa) of the meta-analyses. FI was first devised by Walsh et al. for randomized controlled trials (RCTs) [3]. Later, Atal et al. demonstrated its application for meta-analyses of RCTs [4]. A recent study by Schröder et al. [5] has shown its wider applicability and has supported its use for meta-analyses of observational studies also. Although the application of FI for meta-analyses of non-RCTs is a point of debate, the technical challenges of conducting good-quality RCTs in surgical specialties are even greater. In the present study, an estimation of the FI for recently published meta-analyses in the Journal of Pediatric Urology (JPUrol) was performed to document their fragility. We hypothesize that the majority of the available meta-analyses in the field of pediatric urology are fragile and their outcome depends on the event status of very few participants only.

Study selection
The online archives of the JPUrol (www.jpurol.com) were searched on June 9, 2021, and 20 recently published articles with the term 'meta-analysis' in their title were screened. For each eligible article, all the meta-analyses reported in that paper were included. The 'meta-analyses reporting a significant outcome' were not considered as an explicit inclusion criterion, and those reporting a non-significant result were also included. Consistent with the previous studies [5], the inclusion of meta-analysis of studies other than RCTs was also considered. The articles examining meta-analyses of non-comparative studies were excluded. Also, those in which a pooled treatment effect was not reported were excluded.

Estimation of the Fragility Index
The FI was calculated as done in previous studies [4,5]. After performing each event status modification, the 95% CI of the treatment effect (risk ratio or odds ratio or risk difference) was re-calculated until the statistical significance of the meta-analysis was reversed. We also confirmed our calculations of FI for each meta-analysis by comparing them with the results achieved from an online platform: http://clinicalepidemio.fr/fragility_ma/. Subsequently, the median (IQR; range) FI of statistically significant, non-significant, and all (significant + non-significant) meta-analyses were calculated and recorded.

Data extraction and analysis
The baseline data of each meta-analysis including the name of the author, year of publication, number of included studies, total sample size, the total number of events, the status of the overall outcome (significant or non-significant) prior to FI estimation, the type of effect measure used (risk ratio or odds ratio or risk difference), the type of method used for pooling the estimates (Mantel-Haenszel or inverse variance or Peto), and the type of effects model (random or fixed) were recorded in an extraction table using MS Excel (Version 15.24). The data analysis was performed using RevMan 5.4 (Cochrane Collaboration, London, UK) and Stata 12.0 (StataCorp LLC, College Station, TX).
The present study involves the collection of already available data in the archives of a journal. This primary data was used to generate secondary data. Since no direct patient contact was established during the course of this study, approval from the Institutional Review Board was not needed.

Discussion
In the evidence-based practice of medicine, meta-analyses and systematic reviews are considered to provide the highest level of evidence [26]. However, if not conducted properly, several issues can arise within them. This ultimately leads to poor quality and reliability of the information disseminated by these meta-analyses.
To overcome this, compliance with the Preferred Reporting Items for Systematic Reviews and Meta-Analysis (PRISMA) guidelines [27] must be ensured. In addition, a critical appraisal of these systematic reviews and meta-analyses can be performed by using the "A MeaSurement Tool to Assess Systematic Reviews 2" (AMSTAR 2) tool [28].
Recently, the FI has been developed as an additional tool to confirm the stability of meta-analyses in terms of the confidence and strength behind the results depicted by them [4]. A meta-analysis is considered to be fragile if the status of an outcome (significant or non-significant) can be reversed by changing the event status of a few participants. In the present study, it was observed that the FI of 59% of the meta-analyses was ≤5. It means that the outcome of almost 60% of the meta-analyses depends on the event status of five or fewer participants. Upon grouping the meta-analyses on the basis of their outcome status, similar findings of low FI were observed.
Consistent results were observed by Schröder et al. [5], depicting a median (Q1-Q3; range) FI of 5 (2-11; 1-50) among the meta-analyses related to pediatric surgery. The authors have very aptly discussed the rarity of the pediatric surgical ailments as the possible reason behind the low FI among these meta-analyses. However, we would like to highlight that 14 (64%) out of the 22 meta-analyses included in the present study were regarding hypospadias, which is one of the most common congenital malformations. Also, the median (Q1-Q3; range) FI of these 14 meta-analyses was 5 (3.25-12; 1-22), with 8/14 (57%) of them having an FI of 5 or less. Therefore, the lack of stability of these meta-analyses despite a more prevalent target population underscores the importance of reporting FI among the common pediatric surgical/urological ailments, e.g., hypospadias.
On the other hand, Atal et al. [4] have demonstrated an absolute FI of 5 or less in one-third of the metaanalyses included in their study. As compared to our study, a median FI of 9 highlights better stability among the meta-analyses included by them. The possible explanation for an almost double value of median FI in their study lies in the well-defined selection criteria adopted by them. Firstly, they included only Cochrane reviews. Secondly, they excluded all meta-analyses focusing on observational studies. Finally, the number of included meta-analyses was so high (>900) that it was unlikely to be influenced by the rarity of a particular disease.
The values of FI can be influenced by a number of factors. Of substantial importance are the sample size and p-value [5]. The greater the sample size, the higher the FI (more stable). In contrast, there is an inverse relationship between FI and p-value. It is often said that FI is nothing but a surrogate representation of the p-value only [5,29]. Other factors influencing the FI include the heterogeneity of treatments across the two treatment arms, the methods used for combining study results, blinding, several types of biases or errors, etc. [4].
To the best of our knowledge, ours is the first study to focus on the estimation of FI among the metaanalyses related to pediatric urology. Unlike other studies [4], we have calculated the FI of all individual meta-analyses within an article. It has been observed that the FI not only varies among the meta-analyses of different studies but also shows a considerable variation among the individual meta-analyses of the same study [19,21,23]. Thus, for a comprehensive evaluation, the FI of all eligible meta-analyses within a study must be estimated.
The results of this study must be interpreted within the context of a few limitations. First, we have screened 20 recently published meta-analyses from one journal only. Although the reason for exploring the JPUrol was due to the fact that this journal focuses exclusively on pediatric urology, it might be possible that the FI of previously published meta-analyses from the same journal or those from other journals have a higher FI. Second, FI was initially designed for RCTs. It was later adopted for meta-analyses of RCTs as well. In the present study, we have estimated the FI of meta-analyses including studies other than RCTs as well. It is believed that the FI of meta-analyses of observational studies must also be calculated due to the obvious challenges of conducting high-quality RCTs in surgical specialties [30]. Third, the estimation of FI involves changing an event to non-event or vice versa only. The effects model (random or fixed), type of effect measure used (risk ratio or odds ratio or risk difference), and the type of method used for pooling the estimates (Mantel-Haenszel or inverse variance or Peto) are kept constant. It will be interesting to note how changes in these variables affect the FI in future studies. Finally, the FI is just a tool to provide additional statistical information about the meta-analyses. It can actually give us a good insight into the p-value. However, it does not provide any descriptive information about the results of the meta-analyses.

Conclusions
Based on our findings, almost 60% of the recently published meta-analyses in the field of pediatric urology have high fragility and their outcomes depend upon the event status of ≤5 participants. Similar to the PRISMA guidelines, a mandatory reporting of the FI can be done to provide additional information about the p-values depicted within the meta-analyses. In fact, at the time of consideration for publication, the FI can serve as an adjunctive tool for critical appraisal of meta-analyses related to common pediatric urologic ailments.