A Retrospective Cohort Study Evaluating the Use of the Modified Early Warning Score to Improve Outcome Prediction in Neurosurgical Patients

Introduction The modified early warning score (mEWS) has been used to identify decompensating patients in critical care settings, potentially leading to better outcomes and safer, more cost-effective patient care. We examined whether the admission or maximum mEWS of neurosurgical patients was associated with outcomes and total patient costs across neurosurgical procedures. Methods This retrospective cohort study included all patients hospitalized at a quaternary care hospital for neurosurgery procedures during 2019. mEWS were automatically generated during a patient’s hospitalization from data available in the electronic medical record. Primary and secondary outcome measures were the first mEWS at admission, maximum mEWS during hospitalization, length of stay (LOS), discharge disposition, mortality, cost of hospitalization, and patient biomarkers (i.e., white blood cell count, erythrocyte sedimentation rate, C-reactive protein, and procalcitonin). Results In 1,408 patients evaluated, a mean first mEWS of 0.5 ± 0.9 (median: 0) and maximum mEWS of 2.6 ± 1.4 (median: 2) were observed. The maximum mEWS was achieved on average one day (median = 0 days) after admission and correlated with other biomarkers (p < 0.0001). Scores correlated with continuous outcomes (i.e., LOS and cost) distinctly based on disease types. Multivariate analysis showed that the maximum mEWS was associated with longer stay (OR = 1.8; 95% CI = 1.6-1.96, p = 0.0001), worse disposition (OR = 0.82, 95% CI = 0.71-0.95, p = 0.0001), higher mortality (OR = 1.7; 95% CI = 1.3-2.1, p = 0.0001), and greater cost (OR = 1.2, 95% CI = 1.1-1.3, p = 0.001). Machine learning algorithms suggested that logistic regression, naïve Bayes, and neural networks were most predictive of outcomes. Conclusion mEWS was associated with outcomes in neurosurgical patients and may be clinically useful. The composite score could be integrated with other clinical factors and was associated with LOS, discharge disposition, mortality, and patient cost. mEWS also could be used early during a patient's admission to stratify risk. Increase in mEWS scores correlated with the outcome to a different degree in distinct patient/disease types. These results show the potential of the mEWS to predict outcomes in neurosurgical patients and suggest that it could be incorporated into clinical decision-making and/or monitoring of neurosurgical patients during admission. However, further studies and refinement of mEWS are needed to better integrate it into patient care.


Introduction
Early warning scores are vital sign-based scoring systems that have been used in hospital settings to help identify patients whose condition is deteriorating and automatically alert staff [1][2][3][4][5][6]. Early identification of decompensating patients may allow for more prompt interventions, potentially leading to better outcomes and safer, more cost-effective patient care [1][2][3][4][5][6]. Described by Stenhouse et al. in 1999 as a modification of the early warning system by Morgan et al., the modified early warning score (mEWS) was developed for the real-time detection of sepsis decompensation in hospitalized adults from operative or emergency room settings [2,[7][8][9][10][11]. A score of >4 was recommended to flag patients for additional review. Escobar et al. describe the conceptual framework for early detection of patient decline using biometric data rather than standard clinical judgment to alter overall patient care trajectories with the earlier intervention [2]. Multiple studies since the development of mEWS and other scoring systems have shown efficacy in triaging septic and nonseptic patients in an emergency department, inpatient, and critical care settings [2,[7][8][9][10][11].
Although an increasing mEWS has been predictive of general surgery postoperative complications [12], to our knowledge, no study has evaluated postoperative complications in a neurosurgical population. One study of mEWS included neurosurgical patients as part of a larger cohort of sepsis patients and showed that automatically generated alerts could reduce poor patient outcomes and costs by prompting earlier care [13]. However, there has been no specific subanalysis of mEWS in neurosurgical patients, which are distinct from other patient populations. The purpose of this study was to evaluate whether the use of the mEWS in monitoring and treatment of neurosurgical patients can be predictive of patient outcomes. We assessed the predictive value of the score for discharge disposition, length of stay (LOS), mortality, and total patient costs across various neurosurgical procedures.

Patients
Institutional Review Board approval with a waiver of informed consent was acquired for retrospective analysis of a prospectively generated database. All hospitalized neurosurgical patients treated at a single tertiary-care center in 2019 were included in the analysis. As previously described [13] software was built into the electronic health record (EHR) that would automatically calculate a vital sign-based mEWS any time vital signs were entered by the nursing staff ( Table 1). Development and implementation of the EHR software involved a lead-in time involving staff training. If a complete set of vital signs was not entered, the EHR would redirect the staff back to complete the collection of vital signs. The mEWS thresholds were set based on internal mortality data.  Patient costs were obtained using the value-driven outcome (VDO) tool, which has been described previously [14,15]. Costs measured by the VDO tool are direct care costs attributable to patient care activities at the level of individual patient encounters; thus, they are not patient charges, cost-to-charge ratios, or reimbursements, which are considered to be less accurate [15]. Although true costs cannot be reported because of the sensitivity of such data, costs were summed for the group, and the proportion of an individual patient's cost is reported as a percentage to allow for comparison between patient groups.
Patient outcomes included discharge disposition, LOS, mortality, and cost. Routine disposition was defined as a home or home-health disposition. Longer LOS and higher cost were defined as greater than the 50th percentile (median) for the entire group.
Data means with standard deviations are reported for continuous data. Numbers and percentages of totals are reported for discrete data. Continuous data were analyzed by one-way analysis of variance with Tukey post-hoc correction whereas discrete data were analyzed by chi-squared testing. Pearson correlation was performed for the mEWS and biomarkers, namely, C-reactive protein (CRP), erythrocyte sedimentation rate (ESR), white blood cell (WBC) count, and procalcitonin level. Statistical analysis was performed by SPSS (V23.0, IBM Corp., Armonk, NY). Predictive logistic regression nomograms and various machine learning algorithms were used to analyze patient data to predict outcomes using Orange software (University of Ljubljana, Ljubljana, Slovenia) [16]. The predictive normogram used a logistic regression classifier to generate score weights. Default settings were used for machine learning analysis: neural networks used a multi-layer perceptron supervised algorithm via scikit-learn version 0.24.1 (sklearn.neural_network.MLPClassifier) with default settings, and support vector machines used the LIBSVM algorithm (www.csie.ntu.edu.tw/~cjlin/libsvm/). All other methods (e.g., naïve Bayes, tree, and random forest) used Orange software defaults (orangedatamining.com).
Other common biomarkers of infection and potential sepsis were recorded, including first WBC count            Figure 2C).

Machine learning algorithms
Predictive nomograms were generated using the mEWS for patient outcomes, allowing relative comparison of different factors, including routine disposition, longer LOS, increased mortality, and increased cost ( Figure 3). New patients can have points sequentially added for each corresponding variable (e.g., maximum mEWS, CCI, age, first mEWS, and sex), and the sum of those points would correlate with a probability of outcome ranging from 0 to 100%. The maximum mEWS played the largest role in predicting outcomes as assessed by the greater contribution to the overall risk score and greater incremental space between specific values and scores. The machine learning algorithm prediction of outcome was compared by receiver operating characteristic (ROC) curves ( Figure 4). Logistic regression, naïve Bayes, and neural networks were the most predictive across all outcomes. Tree algorithm with forward pruning, support vector machines, and random forest provided less robust predictive value.

Study findings
Although it may seem intuitive that elevated mEWS can predict worse outcomes in a neurosurgical population, this has not been quantified or evaluated before now. This study suggests that mEWS in the field of neurosurgery can be predictive of outcomes across a variety of neurosurgical procedures. Among all clinical factors, the maximum mEWS was the best predictor of outcome for discharge disposition, LOS, and total patient cost, and the first recorded mEWS was the best predictor of mortality. Moreover, the mEWS was instructive of outcomes across various diseases and after controlling for patient age and comorbidity. During the implementation of a mEWS-based quality improvement project at our institution, there were concerns among subspecialties that such a score did not apply to their specific patient populations; however, these results indicate that the mEWS can be used for this purpose in neurosurgery. The maximum mEWS was achieved on average within zero to one day of hospitalization, indicating its potential utility in improving patient outcomes if early interventions and monitoring are implemented as a result of the score. Unlike other studies showing a discrete cutoff for mEWS, our results suggested a continuous increase in morbidity with scores. Further study is needed to identify a cutoff to flag specific patient groups for interventions or closer care.

Predictive value of the mEWS
Over the past two decades, the use of the mEWS has been implemented by hundreds of hospitals [17,18]. Since its inception, the system has been used for a variety of purposes because of its efficacy as a predictive tool. It has been used to determine the need for a patient in the emergency department to be admitted to the hospital, to predict inpatient mortality rates, and to guide the postoperative care of surgical patients [12,17,19,20]. The use of the mEWS has also been associated with lower rates of intensive care unit (ICU) admissions, lower overall hospital costs, and a shorter LOS without an increase in mortality rates [13,21].
However, evidence to support the use of the mEWS in neurosurgery is new.
Multiple single-center studies have looked at different facets of the predictive value of the mEWS. Cei et al. [22] prospectively measured morbidity and mortality rates in 1,107 consecutive elderly patients using the mEWS and found that an elevated admission mEWS was significantly associated with higher rates of mortality and morbidity. Levin et al. [23] retrospectively evaluated 8,322 emergency department patients that had a combined ICU admission rate of 17% and an overall mortality rate of 2%. The authors reported that dynamic vital signs (evaluated with the mEWS) that were abnormal and failed to normalize were associated with increased rates of mortality, ICU admission, LOS, and sepsis diagnosis. The failure to normalize the mEWS was better associated with the outcome than any single score evaluated, namely, the triage mEWS, maximum mEWS, and last mEWS. Several studies have aimed to use the mEWS in this manner to identify high-risk patients and intervene early [13,24]. Our results support these findings suggesting that mEWS can predict poor outcomes in the general and subspecialty neurosurgery population.
One study also compared the CRP/albumin ratio to the mEWS and showed that the mEWS might be more sensitive in predicting mortality [25]. Our results support a correlation between inflammatory biomarkers and the mEWS but highlight the easier implementation of the mEWS into clinical decision-making. Kangas et al. [26] found that combining the systemic inflammatory response syndrome (SIRS) criteria with the mEWS resulted in correctly capturing three times more sepsis diagnoses while triggering 46% fewer falsepositive alerts. Often biomarkers are used for evaluation of sepsis prior to culture results or in settings without clear symptoms. The use of mEWS may be a way to further improve the accuracy of sepsis in these neurosurgical patients.

Avenues for future development
This study suggests that further research into the combination of predictive tools, including those originally used for sepsis, is warranted to evaluate their effectiveness in different patient populations in an effort to improve safety and outcomes for patients and to further refine the predictive value of these tools. One finding of this study was the variability in utility for mEWS in specific neurosurgical patient populations, where a rise in mEWS distinctly impacted LOS and cost by patient type. Incorporation of the neurological examination into such a predictive score, for example, might be beneficial for neurosurgical patients because of the importance of the examination in surgical decision-making and its strong correlation with outcomes [27,28]. The addition of other biomarkers may also be a direction to improve predictive accuracy. The addition of imaging parameters and other neurosurgical patient classification could further refine patient prediction. Machine learning algorithms, briefly addressed in this study, may also be powerful in integrating large numbers of mEWS values across different time points. Such an algorithm could undergo real-time updates with any new data points during a patient's hospitalization to predict any number of metrics.

Limitations of the mEWS
Limitations to the mEWS have been described in multiple studies. Cooksley et al. [29] performed a retrospective analysis of 840 patients at a specialist oncology hospital to assess the effectiveness of the mEWS to predict outcomes in oncology patients. ROC curves showed that the mEWS had a poor predictive value for admission to the ICU (0.55) and 30-day mortality (0.6). These data indicate that the mEWS may be limited in its ability to identify critical physiological parameters that contribute to outcomes in some cohorts, warranting ongoing evaluation and study among different patient populations. We felt our early results suggest mEWS could be helpful for a cohort of neurosurgical patients but needed further refinement. In addition, our results require further validation in other patient settings and institutions.
A limitation in our study involves the aggregation of multiple patient types into specific treatment categories, thus limiting understanding of specific patient procedures or diagnoses on the outcome. Further breakdown of patient procedures is required and planned for future studies. Implementing findings from the mEWS has been shown to improve survival after sepsis despite the variable presentation and sources of sepsis [13]. It is unclear whether and how prospectively implementing a mEWS protocol in neurosurgical patients would help improve outcomes in this population of patients beyond sepsis alone. This would be a question opportune for further study. As mentioned, incorporating neurological examinations and additional biomarkers in a neurosurgical mEWS may be a way of further improving the prediction of prognosis. Lastly, the specific components of the mEWS were integrated into a final score, and future work will aim to evaluate the impact of individual score components in relation to neurological examination findings or other biomarkers to assess for the most effective variables. Capturing these additional data in future studies could potentially help refine a neurosurgical mEWS.

Conclusions
Our results suggest that the first recorded mEWS and the maximum recorded mEWS are highly predictive of disposition, LOS, mortality, and overall direct patient cost in a cohort of neurosurgical patients. The results suggested that real-time and dynamic physiologic alert systems could help distinguish patient risk early in a hospital course. These data add to the current literature involving the use of the mEWS as both an indicator of patient deterioration and a tool for predicting patient outcomes, and this study is the first evaluation of its use in a neurosurgical population. Understanding the ability of the mEWS to both improve outcomes as well as predict them across the entire spectrum of medical specialties is vital to proper patient care and management of hospitalized patients. Potential next steps would be further refinement of a predictive system, implementation of a real-time mEWS alert system among a neurosurgical population, and validation of a predictive algorithm in different patient settings or hospitals.

Additional Information Disclosures
Human subjects: Consent was obtained or waived by all participants in this study. University of Utah Institutional Review Board issued approval 96120. Institutional Review Board approval with a waiver of informed consent was acquired for retrospective analysis of a prospectively generated database. Animal subjects: All authors have confirmed that this study did not involve animal subjects or tissue. Conflicts of interest: In compliance with the ICMJE uniform disclosure form, all authors declare the following: Payment/services info: All authors have declared that no financial support was received from any organization for the submitted work. Financial relationships: All authors have declared that they have no financial relationships at present or within the previous three years with any organizations that might have an interest in the submitted work. Other relationships: All authors have declared that there are no other relationships or activities that could appear to have influenced the submitted work.