"Never doubt that a small group of thoughtful, committed citizens can change the world. Indeed, it is the only thing that ever has."

Margaret Mead
Technical report

Threshold Limit Graphical Approach to Understanding Outcome Predictive Metrics: Data from the Osteoarthritis Initiative


Scatter plots, bar charts, linear regressions, analysis of variance, and other graphics and tests are frequently used to document associations between an independent variable and an outcome. However, these methods are also frequently limited when understanding how to use an independent variable in subsequent research or patient management. A novel graphical approach to visualizing data—the threshold limit graph—was therefore developed.

Publically available data from the Osteoarthritis Initiative was used to illustrate the graphical approach to understanding the association between the change in joint space width (ΔJSW, independent variable) over four years, and knee symptoms at four years (using the Knee Injury and Osteoarthritis Outcome Score [KOOS], dependent variable).

Using data for 4,202 knees, the traditional scatter plot and linear regression approach showed a significant but weak linear relationship between the symptom subscore of the KOOS and ΔJSW. However, the threshold level of ΔJSW that affects symptoms was not clear from the data. The same dataset was then plotted using the threshold limit graphical approach, which revealed a non-linear relationship between the variables. In contrast to the scatter plot, plotting the average KOOS symptom subscore for subgroups of the data, with each subgroup defined using sequentially increasing or decreasing ΔJSW thresholds revealed that symptoms got worse with joint space loss, but only when there was a significant amount of ΔJSW. A threshold limit analysis was repeated using small, randomly selected subsets of the data (N = ~100) to demonstrate the utility of the technique for identifying trends in smaller datasets.

The threshold limit graph is a simple, graphical approach that may prove helpful in understanding how an independent variable might be used to predict outcomes. This approach provides an additional option for visualizing and quantifying associations between variables.


Regression equations, scatter plots, bar graphs, and associated statistics are frequently used to illustrate that a variable has a significant effect on clinical outcomes. However, based on these tools, it is often difficult to understand how an apparently predictive variable might be used to manage an individual patient. This technical report describes an approach to graphically displaying data that can help users better understand the effect of a variable on clinical outcomes. The approach is based on identifying any significant changes in outcomes that occur, as subgroups of the subjects are analyzed based on a threshold level of the independent variable. To the authors’ knowledge, there is no previously published literature using this methodology.

Publically available data from the Osteoarthritis Initiative (OAI) were analyzed for this report. The OAI is a nationwide research program sponsored by the National Institutes of Health (NIH) to study the treatment and prevention of knee osteoarthritis. The purpose of the analysis was to determine the effect of the change in joint space width (ΔJSW) over four years on the clinical outcome at four years, as determined by the Knee Injury and Osteoarthritis Outcome Score symptom subscore (KOOS) [1-2].

Technical Report

The data used in the preparation of this report was obtained from the OAI database after completing the required data use agreement. Data is available for public access at http://www.oai.ucsf.edu/. Specific datasets used were from JointSx06SAS.zip 26-Oct-2011, kXRQJSWDuryea00SAS.zip 11-Mar-2016, and kXRQJSWDuryea06SAS.zip 11-Mar-2016. These datasets include the reference, progression, and incident cohorts from the OAI study [3]. Data for a total of 4,202 knees from 2,506 subjects were analyzed. This data represents every subject from the OAI study that had available radiographic knee joint space width (JSW) measurements at baseline and four years, and KOOS symptom subscores at four years. JSW was measured at the location of minimum joint space [4-5], and ΔJSW was calculated as the difference in JSW between baseline and four years. A negative ΔJSW value represents joint space loss, and a positive value represents joint space gain over time. KOOS symptom subscores were reported as a normalized outcome in the OAI data such that a score of 100 indicated no symptoms and a score of 0 indicated extreme symptoms. All variables were reported separately for left and right knees.

The data was analyzed in Stata (version 14, Stata Corp, College Station, TX) using a custom script to calculate the average KOOS symptom subscore for subgroups defined by a threshold level of ΔJSW. For example, the average KOOS symptom subscore was calculated including all subjects with a ΔJSW < -1 mm. Average KOOS symptom subscores were then plotted as a function of the threshold limit used to define the subgroup (threshold limit graph). Those data were used to visualize the effect of ΔJSW on KOOS symptom subscore graphically. Bootstrap sampling was conducted in Stata to estimate the overall variability in the KOOS symptom subscore data for all 4202 knees. The 95% confidence interval from the bootstrap function was used to define the range of KOOS symptom subscores that can be explained by the overall variability in the data. Thus, a subgroup that has an average KOOS symptom subscore outside of the 95% confidence interval is considered to differ significantly from the overall population.

Using a traditional scatter plot approach, including all available data, no clear trend is evident between ΔJSW and KOOS symptom subscores (Figure 1A). From the scatter plot, it would appear that factors other than the ΔJSW account for variability in the KOOS symptom subscore. Nevertheless, a linear regression analysis indicates a significant (P < 0.0001) but weak (R2 = 0.01) relationship between clinical outcomes and ΔJSW. With a binned bar graph plot of the same data, the trend in lower KOOS symptom subscores with greater joint space loss becomes apparent (Figure 1B). However, it is unclear whether this trend is linear across the entire range of ΔJSW in the OAI data. This could be important in application of personalized medicine toward predicting prognosis based on a specific ΔJSW for a particular patient.

Figure 2 displays an alternative way to visualize the same data using the novel threshold limit graph. With this plotting technique, two complimentary series of data points depict clear trends in KOOS symptom subscores when ΔJSW goes above or below certain threshold limits. In the light blue series, with filled marker arrows pointing left (<), each marker represents the average KOOS symptom subscore of a subgroup that only includes those knees with ΔJSW less than the marker ΔJSW value (or the threshold limit). For example, the marker at -2 mm ΔJSW represents a subgroup of the population where only knees with ΔJSW < -2 mm were included, and ΔJSW ≥ -2 mm were excluded. Moving from right to left on the curve, the light blue series incrementally excludes greater amounts of data. In contrast, the dark blue series, with open marker arrows pointing right (>), only includes knees where ΔJSW was greater than the threshold limit at each marker. Thus, data is incrementally excluded moving from left to right on the curve. The horizontal green line shows the average KOOS symptom subscore at four years for the entire population, with the shaded green region depicting the 95% confident interval around the mean.

In the threshold limit graph, when the markers on either the light or dark blue series move above or below the 95% confidence interval for the entire population, the subgroup represented by the marker is significantly different from the entire population. Focusing on the light blue series, when only those subjects with ΔJSW < -0.5 mm are included, the KOOS symptom subscore is significantly below the average for the entire population. Using data for the markers on the light blue line that include only subjects with ΔJSW < -0.5 mm, a linear regression line supports that the KOOS symptom subscore decreases by three for every 1 mm loss in joint space (P < 0.0001, R2 = 0.95; data not shown). This graphical approach suggests there is an association between ΔJSW over four years and patient outcomes at four years, but only for patients that lost significant joint space. Interestingly, the dark blue series shows a trend of improved outcome score for patients with an increase in joint space over a 1.5-mm threshold; this association was not investigated further.

It is important to appreciate that as an increasingly larger proportion of data is excluded, the sample size can get very small. Figure 3 contains the same data as Figure 2, with the addition of data showing how the sample size is affected by excluding data. It can be seen from Figure 3 that a relatively small proportion of the subjects lost > 2 mm or gained > 1 mm JSW. These graphs help to illustrate that ΔJSW may be predictive of outcomes, but only for those subjects with substantial changes in joint space. This is an observation that cannot be readily appreciated from a scatter plot or simple linear regression analysis of data for the entire population.

The threshold limit graph can also be helpful for the analysis of small datasets. To illustrate this, data for the 4,202 knees were randomly divided into 42 subsamples using a random number generator, and then each of those subsamples (~2.5% of the data) was plotted using the threshold limit technique. Figure 4 provides data for the first nine subsamples, which show similar trends to those observed in the entire population. Note that the 95% confidence interval for the mean for the subsamples (Figure 4) is wider than that of the entire population (Figure 2), which is a consequence of the smaller sample size. As with all graphical and statistical approaches to visualizing and analyzing data, a small sample size may or may not provide similar results to those that would be found using a large sample size. Nevertheless, a threshold limit graph of data from a small study could potentially help in the design of a larger study and offers an alternative and additional method to identify trends in the data.


Multiple approaches are available to understand how specific variables may affect an outcome [6]. A graphical approach such as the threshold limit graph may help to isolate the effect of individual variables by systematically including/excluding samples based on threshold values and comparing outcomes to the population mean. This information may be helpful in designing subsequent studies and in understanding the relative importance of a variable in patient management. Threshold limit graphs allow the user to estimate the effects on subsequent study outcomes when setting thresholds for variables for patient inclusion and/or exclusion criteria.

There has long been an interest in identifying any possible association between change in the knee joint space and knee outcomes [7-8]. A previous study showed that subjectively graded joint space narrowing was a risk factor for frequent pain in subjects that had frequent knee pain in one knee but not in the contralateral knee [9]. However, other investigators have not been able to document an association between joint space narrowing and symptoms [10-11]. A re-analysis of the data using the threshold limit graphical approach described in this paper may provide additional insights into the association between change in the knee joint space and outcomes in the OAI study, other knee studies, and eventually in clinical practice.

Although this graphical approach to understanding data requires an independent variable that is reported on a continuous scale, the dependent variable may be continuous or binary. When using a continuous variable, the average value can be plotted on the y-axis for specific subsets of the data. When using a binary dependent variable, the proportion of subjects with a positive outcome can be plotted on the Y-axis for each of a range of subgroups defined by threshold levels of the independent variable.

As with most statistical testing, a large sample size is desirable when using threshold-limit graphs. Application of the threshold-limit graph to a small sample of data may fail to show relationship between independent and dependent variables that would be seen with a larger sample size.


There are both orthopedic and non-orthopedic problems where multiple factors may affect an outcome. By focusing on how a dependent variable is affected when data are included or excluded based on a threshold level of the independent variable, the threshold-limit approach to graphically displaying data may help to better understand associations between independent variables and outcomes.

Based on our analysis of the OAI data, there is a significant correlation between ΔJSW from baseline to four years and KOOS symptom subscores at four years for patients with specific ΔJSW characteristics. Patients with a loss of joint space of 0.5 mm or more had significantly lower KOOS symptom subscores than the average population. In contrast, patients with an increase in joint space of 1.5 mm or more had significantly higher KOOS symptom subscores than the average population.


  1. Roos EM, Toksvig-Larsen S: Knee injury and Osteoarthritis Outcome Score (KOOS)-validation and comparison to the WOMAC in total knee replacement. Health Qual Life Outcomes. 2003, 1:17. 10.1186/1477-7525-1-17
  2. Knee injury and Osteoarthritis Outcome Score. 2017, Accessed: June 7, 2017: http://www.koos.nu..
  3. Eckstein F, Kwoh CK, Link TM, et al.: Imaging research results from the Osteoarthritis Initiative (OAI): a review and lessons learned 10 years after start of enrolment. Ann Rheum Dis. 2014, 73:1289–1300. 10.1136/annrheumdis-2014-205310
  4. Neumann G, Hunter D, Nevitt M, et al.: Location specific radiographic joint space width for osteoarthritis progression. Osteoarthritis Cartilage. 2009, 17:761–765. 10.1016/j.joca.2008.11.001
  5. Duryea J, Li J, Peterfy C, et al.: Trainable rule‐based algorithm for the measurement of joint space width in digital radiographic images of the knee. Med Phys. 2000, 27:580–591. 10.1118/1.598897
  6. Abu Hanna A, Lucas PJ: Prognostic models in medicine-AI and statistical approaches. Methods Inf Med. 2001, 40:1–5.
  7. Oak S, Ghodadra A, Winalski C, et al.: Radiographic joint space width is correlated with 4-year clinical outcomes in patients with knee osteoarthritis: data from the osteoarthritis initiative. Osteoarthritis Cartilage. 2013, 21:1185–1190. 10.1016/j.joca.2013.06.024
  8. Bedson J, Croft PR: The discordance between clinical and radiographic knee osteoarthritis: a systematic search and summary of the literature. BMC Musculoskelet Disord. 2008, 9:116. 10.1186/1471-2474-9-116
  9. Neogi T, Felson D, Niu J, et al.: Association between radiographic features of knee osteoarthritis and pain: results from two cohort studies. BMJ. 2009, 339. 10.1136/bmj.b2844
  10. Eckstein F, Maschek S, Wirth W, et al.: One year change of knee cartilage morphology in the first release of participants from the Osteoarthritis Initiative progression subcohort: association with sex, body mass index, symptoms and radiographic osteoarthritis status. Ann Rheum Dis. 2009, 68:674–679. 10.1136/ard.2008.089904
  11. Klein JS, Jose J, Baraga MG, et al.: Baseline cartilage thickness and meniscus extrusion predict longitudinal cartilage loss by quantitative magnetic resonance imaging: data from the Osteoarthritis Initiative. J Comput Assist Tomogr. 2016, 40:979–984. 10.1097/RCT.0000000000000464
Technical report

Threshold Limit Graphical Approach to Understanding Outcome Predictive Metrics: Data from the Osteoarthritis Initiative

Author Information

John A. Hipp Corresponding Author

Research, Medical Metrics

Elaine F. Chan

Technical Services, Medical Metrics

Ethics Statement and Conflict of Interest Disclosures

Human subjects: All authors have confirmed that this study did not involve human participants or tissue. Animal subjects: All authors have confirmed that this study did not involve animal subjects or tissue. Conflicts of interest: In compliance with the ICMJE uniform disclosure form, all authors declare the following: Payment/services info: All authors have declared that no financial support was received from any organization for the submitted work. Financial relationships: All authors have declared that they have no financial relationships at present or within the previous three years with any organizations that might have an interest in the submitted work. Other relationships: All authors have declared that there are no other relationships or activities that could appear to have influenced the submitted work.


The OAI is a public-private partnership composed of five contracts (N01-AR-2-2258; N01-AR-2-2259; N01-AR-2-2260; N01-AR-2-2261; and N01-AR-2-2262) funded by the NIH, a branch of the Department of Health and Human Services, and conducted by the OAI Study Investigators. Private funding partners include Merck Research Laboratories; Novartis Pharmaceuticals Corporation, GlaxoSmithKline; and Pfizer, Inc. Private sector funding for the OAI is managed by the Foundation for the National Institutes of Health. This manuscript was prepared using an OAI public-use data set and does not necessarily reflect the opinions or views of the OAI investigators, the NIH, or the private funding partners.

Technical report

Threshold Limit Graphical Approach to Understanding Outcome Predictive Metrics: Data from the Osteoarthritis Initiative

Figures etc.

PDF Print Share