CAST-HSROC: A Web Application for Calculating the Summary Points of Diagnostic Test Accuracy From the Hierarchical Summary Receiver Operating Characteristic Model

Background: Researchers have been advised to report the point estimate of either sensitivity or specificity and its 95% credible interval (CrI) for a fixed specificity or sensitivity value in the summary of findings (SoF) table for diagnostic test accuracy (DTA) when they use the hierarchical summary receiver operating characteristic (HSROC) model. However, there is no other tool that easily calculates the statistics. Results: We developed the calculator for the summary points from the HSROC model (CAST-HSROC), a web application for calculating the statistics easily. The existing graphical user interface software such as Review Manager and MetaDTA cannot calculate the statistics. Users should check whether convergence is reached before interpreting the results. Conclusions: CAST-HSROC can easily calculate the point estimate of either sensitivity or specificity and its 95% CrI for a fixed specificity or sensitivity value on the HSROC model. The application can help to create an SoF table for DTA when systematic reviewers or guideline developers estimate the certainty of evidence on the HSROC model.


Introduction
The number of systematic reviews (SR) has been increasingly used to assess diagnostic test accuracy (DTA) [1]. Two models are routinely used for the meta-analysis of DTA studies, namely, the bivariate and hierarchical summary receiver operating characteristic (HSROC) models [2]. The bivariate model produces summary estimates of sensitivity and specificity, whereas the HSROC model produces a summary receiver operating characteristic (ROC) curve. Their use depends on whether the diagnostic thresholds for the target condition used in the primary studies are similar. If the thresholds vary across the primary studies, the pooled estimates of sensitivity and specificity based on the bivariate model will be uninterpretable. In such a case (the HSROC model), those estimations are unrestricted by the threshold and should be appropriate. Recently, the Grading of Recommendations Assessment, Development and Evaluation (GRADE) Working Group has published new guidelines for estimating the certainty of evidence to help researchers in performing SR for DTA [3,4]. However, these guidelines were based on summary estimates of sensitivity and specificity and did not describe how the certainty of evidence can be estimated in GRADE for DTA when using the HSROC model. In contrast, the Cochrane Handbook suggests that researchers should report the point estimate of either sensitivity or specificity and its 95% credible interval (CrI) for a fixed specificity or sensitivity value in the summary of findings (SoF) table for DTA when they use the HSROC model, which is a statistical model based on latent-scale logistic regression. It considers the variabilities both within and between studies (for example, different cut-off values used in different primary studies) [2]. However, a difficult calculation based on the natural logarithm of the diagnostic odds ratio (log DOR) is required to obtain these values [2]. The calculation was performed using a complex equation. In the equation, Λ is the estimated average location parameter, and β is a scale parameter. is more restricted than access to statistical software in many settings [5]. Our objective was to develop a freely available web-based software that permits users to input their own data and generate the point estimate of either sensitivity or specificity and its 95% CrI for a fixed specificity or sensitivity value of the HSROC model in a DTA study.

Materials And Methods
We developed the calculator for the summary points from the HSROC model (CAST-HSROC), a web calculator, to easily calculate the point estimate of either sensitivity or specificity and its 95% CrI for a fixed specificity or sensitivity value in the HSROC model [6]. We released the web application in March 2020. Thanks to this software, we no longer need to directly substitute values into complex mathematical formulas for calculations ( Figure 1).

FIGURE 1: Equation of the HSROC model
Λ is the estimated average location parameter, β is a scale parameter. HSROC: hierarchical summary receiver operating characteristic [2].

Software
We used the software R and its packages Shiny and RStan to develop the application [7][8][9]. Shiny is a package that permits users to develop web applications without executing programming using web development languages [8]. RStan is a package with executing programming by Stan [9]. Stan is a probabilistic programming language that performs Bayesian statistical inference via Markov Chain Monte Carlo [9]. CAST-HSROC works on the Shiny application server, which any user can use with a web browser, without any statistical software. The web application is available at https://youkiti.shinyapps.io/CAST-HSROC/ [6].
The inference for the Stan model is HSROC.
Setting of Markov chain Monte Carlo methods (MCMC) is as follows: four chains, each with iter=1000; warmup=500; thin=1; post-warmup draws per chain=500, total post-warmup draws=2000. Here, "chain" denotes the number of Markov chains; "iter" is the number of iterations for each chain; "warmup" is the number of warmup iterations per chain, and "thin" is the period for saving samples [9].

Data import
Users should upload their datasets before calculation. They can update it after clicking the tab 'Upload Data'.
The file should be in the formats that use delimiter-separated values (DSV), that is, to store twodimensional arrays of data by separating the values in each row with specific delimiter characters. The supported delimiters are comma (,), semicolon (;), tab ( ), and space ( ). Please ensure you select the corresponding file delimiter in the left panel. We recommend uploading a comma-separated values (CSV) file.
The dataset should have five columns. Column 1 should be named as 'study_name', referring to the study ID, which can be numeric or characters. Each study contains fourfold (2 x 2) table information. Column 2 should be named as 'TP', and it includes the number of true positive patients (diseased patients with positive test results). Column 3 should be named as 'FN', and it includes the number of false-negative patients (diseased patients with negative test results). Column 4 should be named as 'FP', and it includes the number of falsepositive patients (patients who are not diseased but have positive test results). Column 5 should be named as 'TN', and it includes the number of true-negative patients (patients who are not diseased and have negative test results).
If users upload the dataset successfully, they can visualize their data by clicking the tab 'Data Confirmation'.
CAST-HSROC includes the default example dataset "example data_set.csv" to help users understand its function. Users can also download it. We present the example dataset in Table 1

Input parameter
Users need to input the parameter, sensitivity or specificity, after clicking the tab 'Results'. The sensitivity or specificity parameter can be selected based on the clinical experience or previous studies. For example, it can be the median value of the sensitivity or specificity reported by the studies included in the metaanalysis.

Functions
In reality, CAST-HSROC directly fits the HSROC model developed by Rutter and Gatsonis [10,11]. For prior distributions, we used non-informative uniform distributions with appropriate upper or lower limits. Appropriate limits that we used are "Uniform(0,+inf)" for standard deviations (SDs) and "Uniform(0,1)" for probabilities. CAST-HSROC calculates the estimated sensitivity or specificity and its 95% Crl after input from users. The application also shows a probability density plot of sensitivity or specificity. The distribution of the estimated variable is colored. Users can download the probability density plot as a PNG file. Figure 2 is an example of the probability density plot of sensitivity if specificity is 0.8. "study_name" is the study identification number.
The application also visualizes the Markov chain trace plot and shows MCMC method details after clicking the tab 'details about MCMC Method'. Users can download the Markov chain trace plot as a PNG file and download MCMC method details as a CSV file. Figure 3 is an example of a Markov chain trace plot. Table 2 presents examples of MCMC output values. "Other_snsp[max]" in Figure 3 and Table 2 is a variable and stands for estimated probability. We have attached the source codes for CAST-HSROC in Appendices 1 and 2, and the sample dataset is included in Table 3.

FIGURE 3: Example of Markov chain trace plot
Other_snsp[max]" is a variable, which is an estimated probability.

Applying calculated data to the SoF table
Users can input estimated probability and its 95% Crl to the SoF table. We present an example of an SoF table for DTA when utilizing the calculator (Figure 4). CAST-HSROC helps to easily calculate the point estimate of either sensitivity or specificity and its 95% CrI for a fixed specificity or sensitivity value in the HSROC model of DTA study, whereas existing GUI software such as Review Manager and MetaDTA cannot calculate the statistics (Table 4). Moreover, CAST-HSROC is a GUI software, in which researchers can input data more easily than a CUI software such as SAS and R (

Discussion
CAST-HSROC can help to create an SoF table for DTA when systematic reviewers or guideline developers estimate the certainty of evidence on the HSROC model. Moreover, the point estimate of either sensitivity or specificity and its 95% CrI for a fixed specificity or sensitivity value might be helpful for researchers aiming to calculate the values of sensitivity or specificity. SAS and R can also implement HSROC models and CAST-HSROC. These software products have CUIs, whereas CAST-HSROC has a GUI [9,12].