Abstract
Background: Researchers have been advised to report the point estimate of either sensitivity or specificity and its 95% credible interval (CrI) for a fixed specificity or sensitivity value in the summary of findings (SoF) table for diagnostic test accuracy (DTA) when they use the hierarchical summary receiver operating characteristic (HSROC) model. However, there is no other tool that easily calculates the statistics.
Results: We developed the calculator for the summary points from the HSROC model (CAST-HSROC), a web application for calculating the statistics easily. The existing graphical user interface software such as Review Manager and MetaDTA cannot calculate the statistics. Users should check whether convergence is reached before interpreting the results.
Conclusions: CAST-HSROC can easily calculate the point estimate of either sensitivity or specificity and its 95% CrI for a fixed specificity or sensitivity value on the HSROC model. The application can help to create an SoF table for DTA when systematic reviewers or guideline developers estimate the certainty of evidence on the HSROC model.
Introduction
The number of systematic reviews (SR) has been increasingly used to assess diagnostic test accuracy (DTA) [1]. Two models are routinely used for the meta-analysis of DTA studies, namely, the bivariate and hierarchical summary receiver operating characteristic (HSROC) models [2]. The bivariate model produces summary estimates of sensitivity and specificity, whereas the HSROC model produces a summary receiver operating characteristic (ROC) curve. Their use depends on whether the diagnostic thresholds for the target condition used in the primary studies are similar. If the thresholds vary across the primary studies, the pooled estimates of sensitivity and specificity based on the bivariate model will be uninterpretable. In such a case (the HSROC model), those estimations are unrestricted by the threshold and should be appropriate. Recently, the Grading of Recommendations Assessment, Development and Evaluation (GRADE) Working Group has published new guidelines for estimating the certainty of evidence to help researchers in performing SR for DTA [3,4]. However, these guidelines were based on summary estimates of sensitivity and specificity and did not describe how the certainty of evidence can be estimated in GRADE for DTA when using the HSROC model. In contrast, the Cochrane Handbook suggests that researchers should report the point estimate of either sensitivity or specificity and its 95% credible interval (CrI) for a fixed specificity or sensitivity value in the summary of findings (SoF) table for DTA when they use the HSROC model, which is a statistical model based on latent-scale logistic regression. It considers the variabilities both within and between studies (for example, different cut-off values used in different primary studies) [2]. However, a difficult calculation based on the natural logarithm of the diagnostic odds ratio (log DOR) is required to obtain these values [2]. The calculation was performed using a complex equation. In the equation, Λ is the estimated average location parameter, and β is a scale parameter.
We need a new graphical user interface (GUI) software to easily calculate the point estimate of either sensitivity or specificity and its 95% CrI for a fixed specificity or sensitivity value because the existing software, namely Review Manager, MetaDTA, SAS, and R, cannot calculate the statistics in the GUI. Researchers in low-income countries and citizen scientists need the software for HSROC models with GUI environments because they might lack research funding and cannot consult statisticians who can implement HSROC models in a character user interface (CUI) environment. Moreover, access to a statistician is more restricted than access to statistical software in many settings [5]. Our objective was to develop a freely available web-based software that permits users to input their own data and generate the point estimate of either sensitivity or specificity and its 95% CrI for a fixed specificity or sensitivity value of the HSROC model in a DTA study.
Materials & Methods
We developed the calculator for the summary points from the HSROC model (CAST-HSROC), a web calculator, to easily calculate the point estimate of either sensitivity or specificity and its 95% CrI for a fixed specificity or sensitivity value in the HSROC model [6]. We released the web application in March 2020. Thanks to this software, we no longer need to directly substitute values into complex mathematical formulas for calculations (Figure 1).
Software
We used the software R and its packages Shiny and RStan to develop the application [7-9]. Shiny is a package that permits users to develop web applications without executing programming using web development languages [8]. RStan is a package with executing programming by Stan [9]. Stan is a probabilistic programming language that performs Bayesian statistical inference via Markov Chain Monte Carlo [9]. CAST-HSROC works on the Shiny application server, which any user can use with a web browser, without any statistical software. The web application is available at https://youkiti.shinyapps.io/CAST-HSROC/ [6].
The inference for the Stan model is HSROC.
Setting of Markov chain Monte Carlo methods (MCMC) is as follows: four chains, each with iter=1000; warmup=500; thin=1; post-warmup draws per chain=500, total post-warmup draws=2000. Here, “chain” denotes the number of Markov chains; “iter” is the number of iterations for each chain; “warmup” is the number of warmup iterations per chain, and “thin” is the period for saving samples [9].
Data import
Users should upload their datasets before calculation. They can update it after clicking the tab 'Upload Data'.
The file should be in the formats that use delimiter-separated values (DSV), that is, to store two-dimensional arrays of data by separating the values in each row with specific delimiter characters. The supported delimiters are comma (,), semicolon (;), tab ( ), and space ( ). Please ensure you select the corresponding file delimiter in the left panel. We recommend uploading a comma-separated values (CSV) file.
The dataset should have five columns. Column 1 should be named as 'study_name', referring to the study ID, which can be numeric or characters. Each study contains fourfold (2 x 2) table information. Column 2 should be named as 'TP', and it includes the number of true positive patients (diseased patients with positive test results). Column 3 should be named as 'FN', and it includes the number of false-negative patients (diseased patients with negative test results). Column 4 should be named as 'FP', and it includes the number of false-positive patients (patients who are not diseased but have positive test results). Column 5 should be named as 'TN', and it includes the number of true-negative patients (patients who are not diseased and have negative test results).
If users upload the dataset successfully, they can visualize their data by clicking the tab 'Data Confirmation'.
CAST-HSROC includes the default example dataset “example data_set.csv” to help users understand its function. Users can also download it. We present the example dataset in Table 1.
Input parameter
Users need to input the parameter, sensitivity or specificity, after clicking the tab 'Results'. The sensitivity or specificity parameter can be selected based on the clinical experience or previous studies. For example, it can be the median value of the sensitivity or specificity reported by the studies included in the meta-analysis.
Functions
In reality, CAST-HSROC directly fits the HSROC model developed by Rutter and Gatsonis [10,11]. For prior distributions, we used non-informative uniform distributions with appropriate upper or lower limits. Appropriate limits that we used are “Uniform(0,+inf)” for standard deviations (SDs) and “Uniform(0,1)” for probabilities. CAST-HSROC calculates the estimated sensitivity or specificity and its 95% Crl after input from users. The application also shows a probability density plot of sensitivity or specificity. The distribution of the estimated variable is colored. Users can download the probability density plot as a PNG file. Figure 2 is an example of the probability density plot of sensitivity if specificity is 0.8.
The application also visualizes the Markov chain trace plot and shows MCMC method details after clicking the tab 'details about MCMC Method'. Users can download the Markov chain trace plot as a PNG file and download MCMC method details as a CSV file. Figure 3 is an example of a Markov chain trace plot. Table 2 presents examples of MCMC output values. “Other_snsp[max]” in Figure 3 and Table 2 is a variable and stands for estimated probability. We have attached the source codes for CAST-HSROC in Appendices 1 and 2, and the sample dataset is included in Table 3.
Applying calculated data to the SoF table
Users can input estimated probability and its 95% Crl to the SoF table. We present an example of an SoF table for DTA when utilizing the calculator (Figure 4).
Results
Users should check whether convergence is reached before interpreting the results. There are two ways of achieving this: (1) checking whether the Rhat of other_snsp[max] is 1.1 or less in the MCMC method details, and (2) checking the Markov chain trace plot for other_snsp[max].
CAST-HSROC helps to easily calculate the point estimate of either sensitivity or specificity and its 95% CrI for a fixed specificity or sensitivity value in the HSROC model of DTA study, whereas existing GUI software such as Review Manager and MetaDTA cannot calculate the statistics (Table 4). Moreover, CAST-HSROC is a GUI software, in which researchers can input data more easily than a CUI software such as SAS and R (Table 4).
Discussion
CAST-HSROC can help to create an SoF table for DTA when systematic reviewers or guideline developers estimate the certainty of evidence on the HSROC model. Moreover, the point estimate of either sensitivity or specificity and its 95% CrI for a fixed specificity or sensitivity value might be helpful for researchers aiming to calculate the values of sensitivity or specificity. SAS and R can also implement HSROC models and CAST-HSROC. These software products have CUIs, whereas CAST-HSROC has a GUI [9,12].
The software has some limitations. First, researchers who did not receive adequate statistical education might misunderstand the results of the HSROC model owing to cursory comprehension of the model or the result of qualitative synthesis in DTA. Therefore, we recommend that CAST-HSROC users read and understand the Cochrane Handbook and the GRADE working group’s guidelines for estimating the certainty of the evidence for DTA [2-4]. Assumptions that must be met at a minimum are that the user can use estimated statistics only when the MCMC converges. We also recommend the involvement of statisticians, if possible. Second, the setting of MCMC is fixed [6]. Third, the point estimate of either sensitivity or specificity and its 95% CrI for a fixed specificity or sensitivity value derived from this software may be complementary to the HSROC model. The HSROC curve itself is the main output of the HSROC model. MetaDTA can obtain the curve, although CAST-HSROC does not create the curve [13].
Conclusions
CAST-HSROC can easily calculate the point estimate of either sensitivity or specificity and its 95% CrI for a fixed specificity or sensitivity value on the HSROC model when systematic reviewers or guideline developers estimate the certainty of evidence on the HSROC model. The point estimate of either sensitivity or specificity and its 95% CrI for a fixed specificity or sensitivity value, derived from CAST-HSROC, may be complementary to the HSROC model. CAST-HSROC is useful because existing GUI software products, such as Review Manager and MetaDTA, cannot calculate the point estimate of either sensitivity or specificity and its 95% CrI for a fixed specificity or sensitivity value on the HSROC model.
References
- McInnes MD, Moher D, Thombs BD, et al.: Preferred reporting items for a systematic review and meta-analysis of diagnostic test accuracy studies: the PRISMA-DTA statement. JAMA. 2018, 319:388-396. 10.1001/jama.2017.19163
- Cochrane handbook for systematic reviews of diagnostic test accuracy version 1.0. (2010). Accessed: October 13, 2020: https://methods.cochrane.org/sdt/handbook-dta-reviews.
- Schunemann HJ, Mustafa RA, Brozek J, et al.: GRADE guidelines: 21 part 2. Test accuracy: inconsistency, Imprecision, publication bias and other domains for rating the certainty of evidence for test accuracy and presenting it in evidence profiles and summary of findings tables. J Clin Epidemiol. 2020, 122:142-152. 10.1016/j.jclinepi.2019.12.021
- Schunemann HJ, Mustafa RA, Brozek J, et al.: GRADE guidelines: 21 part 1. Study design, risk of bias, and indirectness in rating the certainty across a body of evidence for test accuracy. J Clin Epidemiol. 2020, 122:129-141. 10.1016/j.jclinepi.2019.12.020
- Welsing PM: Statistical modelling: essentially, all models are wrong, but some are useful. Review series on statistical modelling. Rheumatology (Oxford). 2015, 54:1133-1134. 10.1093/rheumatology/kev116
- CAST-HSROC: calculator for the summary points from the HSROC model. (2020). Accessed: October 13, 2020: https://youkiti.shinyapps.io/CAST-HSROC/.
- The R Project for statistical computing. (2020). Accessed: October 13, 2020: https://www.R-project.org/.
- shiny: Web application framework for R. (2019). Accessed: October 13, 2020: https://CRAN.R-project.org/package=shiny.
- RStan: the R interface to Stan. R package version 2.19.3. (2020). Accessed: October 13, 2020: http://mc-stan.org/.
- Rutter CM, Gatsonis CA: Regression methods for meta-analysis of diagnostic test data. Acad Radiol. 1995, 2:48-56.
- Rutter CM, Gatsonis CA: A hierarchical regression approach to meta-analysis of diagnostic test accuracy evaluations. Stat Med. 2001, 20:2865-2884. 10.1002/sim.942
- METADAS: a SAS macro for meta‐analysis of diagnostic accuracy studies. (2008). Accessed: February 1, 2021: https://methods.cochrane.org/sites/methods.cochrane.org.sdt/files/public/uploads/MetaDAS%20Quick%20Reference%20v1.3%2....
- Freeman SC, Kerby CR, Patel A, Cooper NJ, Quinn T, Sutton AJ: Development of an interactive web-based tool to conduct and interrogate meta-analysis of diagnostic test accuracy studies: MetaDTA. BMC Med Res Methodol. 2019, 19:81. 10.1186/s12874-019-0724-x
Appendices
Appendix 1
Title of Data
Archive of source code of the current version of CAST-HSROC
Description of Data
The code used to construct the application
library(shiny)
library(shinythemes)
library(rstan)
library(tidyverse)
library(ggplot2)
library(HDInterval)
lmcss <- "
#plot-container1 {
position: absolute; left: 50%; top: 40%; z-index: -1;
}
#loading-spinner1 {
position: absolute; left: 50%; top: 50%; z-index: -1;
margin-top: -33px; /* half of the spinner's height */
margin-left: -33px; /* half of the spinner's width */
}
#plot.recalculating {
z-index: -2;
}
#loadmessage1 {
position: absolute; top: 50%; left: 10%; width: 80%; padding: 5px 0px 5px
0px;
text-align: center; font-size:130%; font-style:italic; color: #708090;
background-color:white; z-index: -1;
}
"
df = read.csv("sampledata_wide.csv")
ui <- fluidPage(
fluidRow(
headerPanel(h2("CAST-HSROC: CAlculator for the Summary poinT from HSROC
model"),
windowTitle = "CAST-HSROC: Calculator for the Summary point
from HSROC Model")
),
fluidRow(
navbarPage(theme = shinytheme("flatly"), "Menu",
tabPanel("Introduction", style = "position:absolute; marginleft:
80px; margin-right: 100px",
h3("Estimating the Certainty of Evidence for HSROC Model
in the Meta-analysis of Diagnostic Test Accuracy",
align = "center"),
br(),
p("The Grading of Recommendations Assessment, Development
and Evaluation (GRADE) Working Group
have published a new guideline regarding estimating the
certainty of evidence for the meta-analysis
of diagnostic test accuracy (DTA) [1]. However, no
concensus has been achieved on how to estimate the certainty of evidence
for ",
span("hierarchical summary receiver operating
characteristic (HSROC) model.", style = "color:lightseagreen"), style =
"font-size:19px"),
p("HSROC model is a statistical model based on latent
scale logistic regression. It considers variability
both within and between studies (for example, different
thresholds used in primary studies) [2].
HSROC cannot estimate a summary point of sensitivity and
specificity. Instead, we can present",
strong("the estimate of specificity and its 95% credible
interval (CrI), if a fixed value of sensitity is given, or vice versa,",
style = "color:steelblue"),
" to demonstrate the changes in sensitivity and
specificity along the curve.", style = "font-size:19px"),
p("We think this result should be included in the Summary
of Finding (SoF) table for a meta-analysis of DTA using HSROC model.
However, in order to obtain these values, difficult
calculation based on log diagnostic odds ratio (log DOR) is needed [2].
Therefore, we develop this calculator for users to
easily estimate it. We also present ",
strong("an example of the SoF table for meta-analyses of
DTA", style = "color:steelblue"),
" in below: ", style = "font-size:19px"),
div(img(src = "sof_table.png", height = "200%", width =
"80%"), align = "center"),
hr(),
p("[1] Schunemann HJ, Mustafa RA, Brozek J, Steingart KR,
Leeflang M, Murad MH, et al.
GRADE guidelines: 21 part 2. Inconsistency, Imprecision,
publication bias and other domains for rating
the certainty of evidence for test accuracy and
presenting it in evidence profiles and summary of findings
tables. J Clin Epidemiol 2020.", style = "font-size:
15px; color: grey"),
p("[2] Deeks JJ, Bossuyt PM, Gatsonis C (editors).
Cochrane Handbook for Systematic Reviews of Diagnostic
Test Accuracy Version 1.0. Available from: http://
srdta.cochrane.org/.; 2010 [accessed March 1 2020].",
style = "font-size: 15px; color: grey")
),
tabPanel("Upload Data",
sidebarLayout(
sidebarPanel(width = 3,
fileInput("file", label = "Please select the file",
multiple = FALSE,
accept = c("text/csv","text/comma-separatedvalues,
text/plain",".csv")
),
tags$hr(style = "border-color: lightgray;"),
strong("File Options", style = "font-size: 16px"),
br(),
checkboxInput("header", label = "First row as column
headings", value = TRUE),
br(),
radioButtons("sep", label = "File Delimiter/
Separator",
choices = c(Comma = ",", Semicolon = ";",
Tab = "\t", Space = " "), selected = ","),
br(),
radioButtons("display", label = "Display the full
dataset or the first 10 rows",
choices = c(All = "all", Head = "head"),
selected = "all"),
tags$hr(style="border-color: lightgray;"),
strong("Download the example dataset", style = "fontsize:
16px"),
br(),
br(),
downloadButton("dlexample", label = "Download the
example")
),
mainPanel(
tabsetPanel(
tabPanel("Data Format Requirement",
br(),
h4("Please upload your file first."),
br(),
p("(1) The file should be in the formats that use
delimiter-separated values (DSV), meaning that to store
two-dimensional arrays of data by separating the
values in each row with specific delimiter characters.
The supported delimiters are: comma (,),
semicolon (;), tab (\t), and space ( ). Please make sure to
select the corresponsing file delimiter in the
left panel. ",
strong("We recommend to upload a comma-separated
values (CSV) file!", style = "color:steelblue"),
style = "font-size:17px"),
br(),
p("(2) The dataset should have 5 columns.", style
= "font-size:17px"),
p(strong("Column 1"), "should be named as",
strong(" 'study_name' ", style = "color:steelblue"),
", referring to the study ID, which can be can
be numerics or characters. Each study contains fourfold (2x2) table
information.", style = "font-size:17px"),
p(strong("Column 2"), "should be named as",
strong(" 'TP' ", style = "color:steelblue"),
", the number of true positive patients (disease
+ positive result).", style = "font-size:17px"),
p(strong("Column 3"), "should be named as",
strong(" 'FN' ", style = "color:steelblue"),
", the number of false negative patients
(disease + negative result).", style = "font-size:17px"),
p(strong("Column 4"), "should be named as",
strong(" 'FP' ", style = "color:steelblue"),
", the number of false positive patients (no
disease + positive result).", style = "font-size:17px"),
p(strong("Column 5"), "should be named as",
strong(" 'TN' ", style = "color:steelblue"),
", the number of true negative patients (no
disease + negative result).", style = "font-size:17px"),
br(),
p("(3) If you upload the dataset successfully, you
can visualize your data by clicking the tab",
strong(" 'Data Confirmation' ", style =
"color:steelblue"), style = "font-size:17px"),
br(),
p("(4) The example dataset looks like this in
below. You can also download it. ", style = "font-size:17px"),
div(tableOutput("showexample"), style = "marginleft:
150px"),
br(),
br()
),
tabPanel("Data Confirmation",
br(),
tableOutput("dataconfirm")
)
)
)
)
),
tabPanel("Results",
sidebarLayout(
sidebarPanel(width = 3,
strong("Input the parameter", style = "font-size:
18px"),
br(),
br(),
radioButtons("parametername", label = "Choose the
class of input parameter",
choices = c("Sensitivity",
"Specificity"), selected = "Sensitivity"),
textInput("parameter", label = "Input the parameter
value", placeholder = "Enter your parameter value"),
tags$hr(style="border-color: lightgray;"),
strong("Note: ", style = "color:gray"),
p("*The ", span("sensitivity or specificity
parameter", style = "color:steelblue;font-style:italic"),
" can be selected based on the clinical experience
or previous studies. For example, it can be the
average value of the studies included in the metaanalysis.",
style = "color:gray"),
withMathJax(),
p("**The HSROC model takes the form [1]: ",
span("$$logit(sensitivity_i) = (\\theta_i + 0.5\
\alpha_i)exp(-0.5\\beta)$$
$$logit(specificity_i) = 1 - (\\theta_i - 0.5\
\alpha_i)exp(0.5\\beta)$$", style = "font-size:13px"),
"\\(i\\) refers to the \\(i\\)th study. If we set
sensitivity or specificity to 0.5, the left side of the above equation
becomes
0. Since exp is constantly positive, \\(\\beta\\)
can be any value, making it an indeterminate equation.
Therefore, if you input a sensitivity or specificity
value as 0.5, you may get an error message.", style = "color:gray"),
p("***We provide the details about the MCMC methods.
You should check if convergence is reached before
interpreting the result. There are two ways: (1) to
check if the Rhat of other_snsp[max] is 1.1 or less;
(2) to check Markov Chain Trace Plot for
other_snsp[max].", style = "color:gray"),
p("****The model used in this calculator was built in
RStan. The code can be found at:",
span("https://github.com/y-luo06/HSROC_shiny.",
style = "color:lightseagreen;font-style:italic"),
style = "color:gray"),
tags$hr(style="border-color: lightgray;"),
p("[1] Macaskill P, Gatsonis C, Deeks J, Harbord R,
Takwoingi Y. Cochrane handbook for systematic reviews
of diagnostic test accuracy. The Cochrane
Collaboration 2010.", style = "font-size: 14px; color: grey")
),
mainPanel(
tabsetPanel(
tabPanel("Results", style = "margin-left:20px;",
fluidRow(
h4("Estimation"),
br(),
uiOutput("estimation"),
br(),
br(),
tags$hr(style="border-color: lightgray;"),
),
fluidRow(
h4("Probability Density Plot"),
br(),
#Add a loading message
tags$head(tags$style(HTML(lmcss))),
conditionalPanel(condition = "$
('html').hasClass('shiny-busy')",
tags$div(id = "plotcontainer1",
tags$img(src = "spinner.gif", id = "loading-spinner1")),
tags$div("Please wait a moment
for the analysis to finish.",id="loadmessage1")),
div(plotOutput("densityplot", width = "600px",
height = "500px"), align = "center"),
br(),
p("You can download this probability density
plot here.", style = "font-size:17px"),
downloadButton("dldensityplot", label =
"Download the plot"),
br(),
br()
)
),
tabPanel("Details about MCMC Method", style =
"margin-left:20px;",
fluidRow(
h4("Markov Chain Trace Plot"),
br(),
#Add a loading message
tags$head(tags$style(HTML(lmcss))),
conditionalPanel(condition = "$
('html').hasClass('shiny-busy')",
tags$div(id = "plotcontainer1",
tags$img(src = "spinner.gif", id = "loading-spinner1")),
tags$div("Please wait a moment
for the analysis to finish.",id="loadmessage1")),
div(plotOutput("traceplot", width = "80%",
height = "500px"), align = "center"),
br(),
p("You can download this trace plot here.",
style = "font-size:17px"),
downloadButton("dltraceplot", label = "Download
the plot"),
br(),
hr()
),
fluidRow(
h4("MCMC Method Details"),
verbatimTextOutput("mcmcfit"),
br(),
p("You can download it in the form of csv
here.", style = "font-size:17px"),
downloadButton("dlfit", label = "Download the
file"),
br()
)
)
)
)
)
)
)
),
fluidRow(column(2,div(style = "height:1000px;backgroundcolor:
rgba(0,0,0,0);"))),
fluidRow(
fillRow(div(style = "height:110px;backgroundcolor:
rgba(229,232,235,1);")),
br(),
column(1),
column(1,
br(),
img(src = "ccby.png", height = 30, width = 80, align =
"center")),
column(10,
strong("Please cite:", style = "color:rgba(21,45,70,1)"),
p("Banno M, Tsujimoto Y, Luo Y, Miyakoshi C, Kataoka Y.
Estimating the certainty of evidence in Grading of
Recommendations Assessment, Development and Evaluation for
test accuracy.", em(" (In submission)"), style = "font-size: 14px;
color:rgba(21,45,70,1)")
)
)
)
server <- function(input, output, session) {
input_file <- reactive({
read.csv(input$file$datapath,
header = input$header,
sep = input$sep)
})
file_reshape <- reactive({
df.tmp <- read.csv(input$file$datapath,
header = input$header,
sep = input$sep)
df.tmp$study <- seq.int(1, length(unique(df.tmp$study_name)))
names(df.tmp)[names(df.tmp) == "TP"] <- "np.1"
names(df.tmp)[names(df.tmp) == "FN"] <- "nn.1"
names(df.tmp)[names(df.tmp) == "FP"] <- "np.0"
names(df.tmp)[names(df.tmp) == "TN"] <- "nn.0"
reshape(data = df.tmp, idvar = "study_name", direction = "long",
varying = c(2:5), timevar = "status", sep = ".")
})
output$dlexample <- downloadHandler(
filename = "example_dataset.csv",
content = function(file){
file.copy("sampledata_wide.csv", file)
}
)
output$showexample <- renderTable(bordered = TRUE, striped = TRUE, {
return(head(df, 6))
})
output$dataconfirm <- renderTable(bordered = TRUE, striped = TRUE, {
req(input$file)
if (input$display == "head") {
return(head(input_file(), 10))
} else {
return(input_file())
}
})
fit <- reactive({
parameter <- as.numeric(input$parameter)
req(input$file, input$parameter, parameter > 0 & parameter <1 &
parameter != 0.5)
datalist <- list(
N = nrow(file_reshape()),
I = length(unique(file_reshape()$study)),
study = file_reshape()$study,
status = file_reshape()$status,
np = file_reshape()$np,
nn = file_reshape()$nn,
one_snsp = parameter
)
model <- stan_model("HSROC.stan")
set.seed(1234)
fit <- sampling(model,
data = datalist,
iter = 1000,
warmup = 500,
chains = 4,
thin = 1,
cores = 4)
})
other_snsp <- reactive({
other_snsp <- rstan::extract(fit())$other_snsp
list =
colnames(other_snsp) <- c(as.character(input_file()$study_name), "all")
other_snsp <- as.data.frame(other_snsp) %>%
tidyr::gather(key = "study_name",value = "prob")
})
other_snsp_global <- reactive({
filter(other_snsp(), study_name == "all")
})
densityplot <- reactive({
ggplot() +
geom_density(data = other_snsp_global(), aes(x=prob), fill = "red",
alpha = 0.2) +
geom_density(data = other_snsp(), aes(x=prob, color = study_name))
})
traceplot <- reactive({
stan_trace(fit(), pars = "other_snsp")
})
output$estimation <- renderUI({
parameter <- as.numeric(input$parameter)
validate(
need(input$file, "Please make sure to upload a dataset with
required format."),
need(parameter >= 0 & parameter <= 1, "Please make sure to input a
correct parameter value between 0 to 1.")
)
if (parameter == 0.5) {
withMathJax(
helpText("Please input a parameter value other than 0.5.",
"This is because HSROC model takes the form [1]:
$$logit(sensitivity_i) = (\\theta_i + 0.5\
\alpha_i)exp(-0.5\\beta)$$
$$logit(specificity_i) = 1 - (\\theta_i - 0.5\
\alpha_i)exp(0.5\\beta)$$
\\(i\\) refers to the \\(i\\)th study. If we set
sensitivity or specificity to 0.5, the left side of the above equation
becomes
0. Since exp is constantly positive, \\(\\beta\\) can
be any value, making it an indeterminate equation.
Therefore, please input a value other than 0.5."))
} else {
if (input$parametername == "Sensitivity") {
paste("The estimated specificity is ",
round(mean(other_snsp_global()$prob), digits = 3),
", with a 95% credible interval from ",
format(round(hdi(other_snsp_global()$prob)[[1]], digits = 3),
nsmall = 3), " to ",
format(round(hdi(other_snsp_global()$prob)[[2]], digits = 3),
nsmall = 3), ".")
} else {
paste("The estimated sensitivity is ",
round(mean(other_snsp_global()$prob), digits = 3),
", with a 95% credible interval from ",
format(round(hdi(other_snsp_global()$prob)[[1]], digits = 3),
nsmall = 3), " to ",
format(round(hdi(other_snsp_global()$prob)[[2]], digits = 3),
nsmall = 3), ".")
}
}
})
output$densityplot <- renderPlot({
req(input$file, input$parameter)
densityplot()
})
output$dldensityplot <- downloadHandler(
filename = "probability_density_plot.png",
content = function(file){
ggsave(file, plot = densityplot(), width = 6, height = 5, device =
"png")
}
)
output$traceplot <- renderPlot({
req(input$file, input$parameter)
traceplot()
})
output$dltraceplot <- downloadHandler(
filename = "trace_plot.png",
content = function(file){
ggsave(file, plot = traceplot(), width = 8, height = 5, device =
"png")
}
)
output$mcmcfit <- renderPrint({
print(fit())
})
output$dlfit <- downloadHandler(
filename = "fit_summary.csv",
content = function(file){
write.table(data.frame(summary(fit())$summary), file, sep=',',
row.names = TRUE, col.names = NA)
}
)
}
shinyApp(ui = ui, server = server)
Appendix 2
Title of Data
Archive of source code for the current version of CAST-HSROC
Description of Data
The model used to calculate the value of interest based on the HSROC model
data{
int N; //number of row of datasheet
int I; //number of studies included
int study[N]; //study ID
int status[N]; //dummy variable for patient status (1 for pts w/ dis,0
for pts w/o dis)
int np[N]; //number of patients with positive test result in each status
int nn[N]; //number of patients with negative test result in each status
real<lower=0,upper=1> one_snsp; //fix one of sn/sp
}
parameters{
vector[I] theta; //theta for each study
vector[I] alpha; //alpha for each study
real beta; //common scale parameter
real theta_g; //global mean of theta
real alpha_g; //global mean of alpha
real<lower=0> s_theta; //square of between study variance for theta
real<lower=0> s_alpha; //square of between study variance for alpha
}
transformed parameters{
vector<lower=0,upper=1>[I] pi_1; //prob of having positive result among
pts w/ dis
vector<lower=0,upper=1>[I] pi_0; //prob of having positive result among
pts w/o dis
vector<lower=0,upper=1>[I] sn; //sensitivity of each study(=pi_1)
vector<lower=0,upper=1>[I] sp; //specificitiy of each study(=-pi_0)
vector<lower=0,upper=1>[I+1] other_snsp;
pi_1 = inv_logit((theta + 0.5*alpha)*exp(-0.5*beta));
pi_0 = inv_logit((theta - 0.5*alpha)*exp(0.5*beta));
sn = pi_1;
sp = 1-pi_0;
for(i in 1:I){
other_snsp[i] = inv_logit((0.25*(alpha[i]^2) - theta[i]^2)/
log(one_snsp/(1-one_snsp)));
}
other_snsp[I+1] = inv_logit((0.25*(alpha_g^2) - theta_g^2)/log(one_snsp/
(1-one_snsp)));
}
model{
for(n in 1:N){
if(status[n]==1){
np[n] ~ binomial(np[n]+nn[n],pi_1[study[n]]); //Level I model for
true positive
}else{
np[n] ~ binomial(np[n]+nn[n],pi_0[study[n]]); // Level I modek for
false positive
}
}
theta ~ normal(theta_g,s_theta); //Level II
alpha ~ normal(alpha_g,s_alpha); //Level II
}
CAST-HSROC: A Web Application for Calculating the Summary Points of Diagnostic Test Accuracy From the Hierarchical Summary Receiver Operating Characteristic Model
Ethics Statement and Conflict of Interest Disclosures
Human subjects: All authors have confirmed that this study did not involve human participants or tissue. Animal subjects: All authors have confirmed that this study did not involve animal subjects or tissue. Conflicts of interest: In compliance with the ICMJE uniform disclosure form, all authors declare the following: Payment/services info: English proofreading fee of this work was supported by the Department of Respiratory Medicine, Hyogo Prefectural Amagasaki General Medical Center and Systematic Review Workshop Peer Support Group (SRWS-PSG). Financial relationships: All authors have declared that they have no financial relationships at present or within the previous three years with any organizations that might have an interest in the submitted work. Other relationships: All authors have declared that there are no other relationships or activities that could appear to have influenced the submitted work.
Acknowledgements
This work was supported by the Department of Respiratory Medicine, Hyogo Prefectural Amagasaki General Medical Center and Systematic Review Workshop Peer Support Group (SRWS-PSG). The funders played no role in the design of the study and collection, analysis, and interpretation of data, and in writing the manuscript. We would like to thank the Grading of Recommendations Assessment, Development and Evaluation (GRADE) Working Group for publishing new guidelines for estimating the certainty of the evidence for diagnostic test accuracy (DTA). We would also like to thank Editage (www.editage.jp) for their English language editing services.
Article Information
DOI
10.7759/cureus.13257
Cite this article as:
Banno M, Tsujimoto Y, Luo Y, et al. (February 10, 2021) CAST-HSROC: A Web Application for Calculating the Summary Points of Diagnostic Test Accuracy From the Hierarchical Summary Receiver Operating Characteristic Model. Cureus 13(2): e13257. doi:10.7759/cureus.13257
Publication history
Peer review began: October 18, 2020
Peer review concluded: February 04, 2021
Published: February 10, 2021
Copyright
© Copyright 2021
Banno et al. This is an open access article distributed under the terms of the Creative Commons Attribution License CC-BY 4.0., which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
License
This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
CAST-HSROC: A Web Application for Calculating the Summary Points of Diagnostic Test Accuracy From the Hierarchical Summary Receiver Operating Characteristic Model
Figures etc.
RATED BY 1 READER
CONTRIBUTE RATINGScholarly Impact Quotient™ (SIQ™) is our unique post-publication peer review rating process. Learn more here.