A Novel Patient-oriented Tool for Evaluating Quality Measurements

Quality measurements (QMs) have emerged as quantitative tools for measuring “quality”, an elusive term that has been historically difficult to define and quantify. However, current literature has demonstrated that these measurements are flawed. The purpose of this study was to identify the strengths and weaknesses of quality measurements and provide a novel scorecard for evaluating quality measurements. In this retrospective analysis, 246 quality measurements that are integrated into the most significant payer-provider contracts within our institution were analyzed. Each measurement was dissected based on type of measurement, evidence, precision, data exchange, alignment, and how patient-oriented. Our research showed a significant lack of quality measurement alignment across payer-provider contracts. As such, we developed and proposed a Quality Measurement Evaluation Tool (QMET) that scores a quality measurement’s ability to 1) reflect population health and 2) promote patient-oriented goals. Our research demonstrated the majority of quality measurements scored in the inadequate range (i.e., QMET score <6) and only few in the optimal range (i.e., QMET score 10-12). QMET provides a standardized and comprehensive method for appraising quality measurements, promoting continued use of QMs that accurately reflect population health and promote patient-oriented measurements. Future research into the application and reliability of QMET is needed.


Introduction
For decades, health care organizations have struggled to measure "quality" -an elusive term that has been historically difficult to define and quantify. More recently, quality measurements (QMs) have emerged as quantitative tools for measuring quality and have been widely adopted by providers, hospitals, and health care systems across the country [1]. However, previous research demonstrates that these measurements are flawed, often struggling to adequately measure quality due to shortcomings in several key areas, such as data collection and clinical evidence [1][2][3][4][5]. Our study analyzes these key areas within our own organization and provides examples of such shortcomings. Furthermore, our study proposes a novel patient-oriented scorecard for reproducible analysis of quality measurements. The scale emphasizes the need for a movement towards patient-oriented quality measurements, a concept that is underrepresented and undervalued [5]. The ultimate goal of our study is to encourage adoption of patient-oriented quality measurements, which aim to understand and evaluate the definition of "quality" from the patient's perspective.

Materials And Methods
This was a retrospective analysis of 246 quality measurements within a single metropolitanarea institution in Pennsylvania. Since 1993, our institution's non-for-profit subsidiary includes 700+ physicians within 45 diverse medical specialties throughout 160 offices in Allentown and Bethlehem, Pennsylvania. There are over 50 payer-provider contracts within this institution, and quality measurements are integrated into these contracts [6]. To further understand the connection between quality metrics and patient-oriented care, this research analyzes the 10 largest contracts within this network. Each measurement within these contracts was dissected based on type of measurement, evidence, precision, data exchange, and alignment. More detailed definitions of these categories are shown in Table 1.

Measurement Type
Whether a measurement is based on a series of actions that lead to a potential health benefit (process) or a measurement based on quantitative results (outcome).

Precision
A measurement with appropriate numerators/denominators, with proper exclusions and inclusions.

Exchange
Data shared from payer to provider or vice versa (unidirectional), or equal exchange of data between payer and provider (bidirectional).

Evidence
Measurement with clear evidence to support its validity (e.g., USPSTF Grade A or B; ACC/AHA highmoderate level evidence).

Alignment
Prevalence of measurement among payers (i.e., percentage of payers using given measurement).

Patient-
Oriented Measurement is of direct value to the patient (e.g., preventing vision loss or stroke).

Evidence
The adoption of quality measurements is based upon clinical evidence and support from national criteria, such as the United States Preventive Service Task Force (USPSTF). However, our research indicates that while a large percentage of QMs are evidence-driven, a substantial portion of QMs is driven by payer data and/or expertise. For example, cervical cancer screening is strongly evidence-based as it measures the number of female patients aged 21 to 65 years who received cervical cytology in the past three years, a criteria endorsed by USPSTF and supported in clinical literature [10][11]. Conversely, quality measurements that monitor comprehensive control of diabetes do so by measuring the percentage of diabetic patients that received low-density lipoprotein (LDL) testing in the last year. There is limited clinical evidence to support the efficacy of these measurements, and national agencies do not support the claim that these measurements improve quality. As such, the failure of quality measurements to be both evidence-based and improve quality once again brings up the question of patients' values. Patient-oriented quality measurements should have clinical evidence to support their efficacy in improving patient care and outcomes; otherwise, they are of no benefit to the patient.

Precision
QMs are quantified by a ratio: a numerator consisting of the number of patients appropriately meeting the measurement divided by a denominator of every patient that qualifies for the measurement. For QMs to reflect population health, it is essential that the way they are quantified is also precise. An example of a precise QM is prescribing controller medications for asthma. Asthma severity is clearly defined in medical practice, and providers prescribe controller medications according to severity -there is little variation in asthma management across providers or payers [12]. However, our research indicates that certain measurements are imprecise, such as cervical cancer screening. When measuring cervical cancer screening, multiple payers define the denominator as "women 24-64 years as of the last measurement year"; however, it does not exclude women who have a history of a hysterectomy, invalidating the denominator and reducing the measurement's precision [12].
Furthermore, transfer and leakage are two other causes of poor precision. Transfer is the movement of patients between payers. Our research indicates that a patient with a prior hysterectomy, when transferred into this network, is inappropriately included in the denominator due to non-transferrable payer codes. The work-around for this problem is to manually input the appropriate code; however, this extra step is rarely carried out, leaving the patient in the denominator and skewing the data. Secondly, leakage causes further inaccuracies. For instance, the quality measurement on diabetic eye exam screenings has been especially difficult for this network. This measurement requires patients with diabetes from age 18 to 75 years old to receive an eye screening for retinal disease [9]. Unfortunately, patients often travel out-of-network and are missed by our network, resulting in improper reimbursements and inaccurate population data [13].
Our research demonstrates the shortcomings in properly defining quality measurements. This challenges the success of quality measurements in reflecting quality and population health. The lack of a standard procedure for creating quality measurements -or a method for revising their flaws -results in an inability of providers to tailor medical services. This inability hinders the movement toward the goal of patient-oriented quality measurements.

Data exchange
Our research also pinpoints several other areas in which quality measurements struggle to accurately measure the quality of health care systems. For example, electronic medical record (EMR) systems -in their current form -have barriers to inputting data and properly documenting patient information. For example, this institution's EMR system can only query data for diabetic foot exams if it is placed in a particular field and will not be queried if it is written in narrative form, which deflates the rate of diabetic foot exams.
Furthermore, this network utilized three types of data extraction: insurance claims, clinical reports, and self-reporting. These different forms of data distribution result in the unidirectional (i.e., from payer to provider or vice versa) or bidirectional (i.e., equal exchange of data between payer and provider) sharing of data. For example, the QM measuring body mass index (BMI) is bidirectional as the provider shares its clinical data with the payer and the payer shares its claims data with the provider. The QM measuring diabetic foot exams is unidirectional as only the provider shares clinical data collected on foot exams to the payer. In the future, every provider-payer relationship should adapt to shift the culture from unidirectional data exchange to bidirectional data exchange in order to best assess population health. Without this shift, the healthcare system will continue to fail to reflect population health simply due to a lack of communication between providers and payers.

Alignment
Arguably the greatest issue facing quality measurements is the lack of alignment. This research finds significant variation in type of measurement, evidence, precision, and data exchange. Of the 246 QMs analyzed in this study, there were only three quality measurements that were utilized by more than 75% of payers. 'Breast Cancer Screening' is an example of a quality measurement with strong alignment as it was standardized across 77% of the payers within this study. On the other hand, 'Fall Risk Assessment for Older Adults' is a quality measurement that is included in only 31% of contracts. This misalignment of quality measurements across payers is unacceptable when considering the goal of patient-oriented quality care. Significant improvement is needed in order to provide patients with quality care that is uniform across payers, such that patients can maneuver between insurance plans without concern for drastic changes in quality.

Limitations
This analysis was limited by the nature that it was retrospective. The analysis was also limited to one health network, and thus might not apply to all networks in various locations. The analysis was also only completed on select payers within the network, therefore, may not apply to all payers.

Discussion
Our solution to the issue of alignment is a novel scoring system for objectively grading quality measurements. Quality Measurement Evaluation Tool (QMET), shown in Table 3, is a 12-point scoring system that evaluates quality measurements based on the areas described in our study. Further examples of each category are listed in Table 4. The scoring criteria is as such that less than six points is an inadequate measurement (i.e., should be revised or reconsidered), six to seven points is a mediocre measurement (i.e., should be re-evaluated for efficacy), eight to nine points is an adequate measurement (i.e., should be used for internal quality improvement), and ten to twelve points is an optimal measurement (i.e., should be used for universal quality improvement).

Factor Score Subtotal
Measurement Type   Colorectal cancer screening scores 8/12 due to strong performances in the areas of alignment, precision, evidence, and how patient-oriented. HCAHPS receives a perfect score and serves as an example of an optimal quality measurement; however, our study finds few quality measurements that meet these criteria, demonstrating the significant need for further development.
QMET focuses on two important elements of quality measurements: population health and patient-oriented goals. The first four factors listed in the scale -measurement type, precision, evidence, and data exchange -emphasize the importance of utilizing quality measurements that fulfill the requirement necessary to reflect population health. Fulfilling these requirements allows quality measurements to accurately measure and analyze population health. It also provides healthcare organizations with data to help drive organizational decisions that will best improve the health of their patients. QMET helps achieve the original goal of quality measurements, which is to advance the quality of our healthcare system. Patient-oriented quality measurements position the patient at the center of how we define quality. They are respectful and responsive to the individual needs, values, and preferences of each patient. The success of each measurement is linked to how well it meets the patient's own definition of quality [14]. This is a dramatic shift away from our previous understanding of quality, which focused predominantly on population health and provider performance.

Conclusions
Our research analyzes the failure of quality measurements to accurately represent population health and promote patient-oriented goals. These failures are derived from shortcomings in the following domains: measurement type, precision, evidence, data exchange, and alignment. We propose Quality Measurement Evaluation Tool (QMET) as a method for evaluating and critiquing quality measurements. QMET provides a reproducible and objective tool for choosing which quality measurements need reconsideration, revision, or are ready for implementation.

Additional Information Disclosures
Human subjects: All authors have confirmed that this study did not involve human participants or tissue. Animal subjects: All authors have confirmed that this study did not involve animal subjects or tissue. Conflicts of interest: In compliance with the ICMJE uniform disclosure form, all authors declare the following: Payment/services info: All authors have declared that no financial support was received from any organization for the submitted work. Financial relationships: All authors have declared that they have no financial relationships at present or within the previous three years with any organizations that might have an interest in the submitted work. Other relationships: All authors have declared that there are no other relationships or activities that could appear to have influenced the submitted work.