Regressions to Monitor Health Care Quality : A User ’ s Guide and a New Index

While there is a rich literature regarding hospital quality in general, research examining trends in hospital quality using regression analysis of trends are relatively rare. In the presence of the increasing digitalization of hospital data, the wide accessibility of regression software, and the need for real time feedback on ongoing quality of medical care outcomes, this paper presents simple-to-implement techniques for analyzing the quality of health care outcomes. We present graphical illustrations of the techniques discussed, and we also include a SAS appendix with the code to estimate all the models discussed, including a proposed new regression index: the Exchange index. We employ data from Arizona’s acute care hospitals to illustrate uses of our new index.

characteristics" [3].Again, they do not emphasize a time trend correction approach.
Many studies examine hospital quality across hospitals in one time period, concluding that variation between hospitals is correlated to hospital characteristics, but quality in one metric may not imply quality in another metric (see Jha, Li, Orav, and Epstein, [4]; as well as Des Harnais, McMahon Jr, and Wroblewski [5]).Pauly, Brailer, Kroch, and Even-Shoshan rate hospitals based on quality measures (without emphasis on time trends) to predict hospital costs for patients and insurers [6].Dimick, Welch, and Birkmeyer examine the frequency of the surgeries that generate the quality measures used by the Agency for Healthcare Research and Quality [7].They find that most of the surgeries are not performed frequently enough to generate an accurate quality measure.
But the opportunity for real-time feedback grows nearly exponentially with standardized regression software, increasingly powerful computers and the digitization of medical records.As such use of digital records becomes widespread, and reporting across treatment and insuring entities becomes more standardized, real-time, healthcare quality 'dashboards'-offering almost instantaneous feedback on relevant hospital metrics-are possible for the first time.Such dashboards, or feedback loops, allow health treatment facilities and insurers (hereafter, these will be referred to as 'hospitals') to examine whether their current treatment outcomes differ significantly from their usual experience, but also whether their outcomes differ from other hospitals, using standard regression techniques.
In this article, we illustrate these techniques graphically, and present a new index for quality control across hospitals which we call the 'exchange index'.Intuitively, the exchange index estimates the average percentile shift in a given hospital's outcomes if it behaved just like the other hospitals in its comparison group.The distributional aspects for this class of index was developed in Butler and McDonald, and illustrated for changes in birth outcomes in Butler, Johnson, and Wilson [8,9].This index has several favorable properties: 1) it uses all the information on both the comparison group and the specific hospital being examined, 2) it scales readily to as many explanatory factors as desired, 3) its value is bounded between plus one (better outcomes in its comparison class than any other hospital) and minus one (worse than other hospitals) with zero indicating no difference on average, and 4) these index values are independent of any linear transformation of the units of measurement (pounds or kilograms, makes no difference).Moreover, as an exchange index value can be readily calculated for each observation (each month-hospital datum), exchange index values as subsequent dependent values can be used to estimate the determinants of quality, much in the same way the propensity scores are used to generate equivalent matches or examine sample selection.
The development of this index and its application to hospital quality measures is a distinguishing feature of this article.Given the increase in hospital-quality data, understanding the quality of each hospital will become increasingly important for patients, payers, and medical care providers.If used appropriately, the increase in data will lead to an overall increase in medical care quality across the nation.In the following sections, we provide an outline for appropriate analyses of hospital data.
In an appendix, we include SAS programs to generate the example outcomes used in this primer.

II. REGRESSION MODELS: A ROBUST FRAMEWORK FOR ANALYZING TRENDS IN OUTCOMES
Consider analyzing the monthly trends from several hospitals, where we wish to compare each hospital to the set of the other hospitals that report the same metric.Note that our methods outlined here can apply to only one, or any subset, of the hospital pool, so that information generated from the regression dashboard is perfectly flexible.Presentations can be made graphically, as we do here, or numerically, with the appropriate tests for statistical significance.

A. Trend Analysis for Each Hospital
Though we illustrate our approach in the context of a simple regression with month of outcomes as the sole explanatory factor, the indexes and methods discussed in the next four sections of this proposal are readily generalized to simultaneously include several determinants of hospital quality (type of hospital, hospital size, nursing/patient ratios, etc).This generalization is discussed in the last section of the article.
Graph 1: The 'No Surprises' Case without trend.
We begin by relating hospital performance to calendar time.In particular, assume that Catheter-Associated Urinary Tract Infections (CAUTI) are to be monitored given monthly observations for CAUTI for a given hospital.The regression establishes the slope/intercept line, and if the outcomes are solely the result of trend and idiosyncratic variation (due to random factors likely beyond the control of the hospital), then the regression graph will appear as in Graph 1.
There are three types of data displayed in Graph 1, as well as the subsequent graphs below.The blue dots are actual data points: combinations of CAUTI rates and months observed for 12 consequent months for the hospital under consideration.For example, in the first month the average CAUTI rate per day was 4.8; in the second month, 5.1; and so forth for the year-length period under consideration.The red line running through the middle of the blue dots is the regression graph estimating the trend of CAUTI rates over months.CAUTI rates are given on the vertical axis; months, on the horizontal axis.The flat regression line in Graph 1 indicates virtually no trend in CAUTI, which remains roughly at 5% throughout the sample period.The blue dots are the actual CAUTI rates for any given month, and the differences between the red, regression graph and the actual outcomes given by the blue dots are the unexplained variation in outcomes, called the residuals.Here, the arbitrary pattern of residuals scattered in an apparent random fashion above and below the regression line indicate that the unexplained variation is indeed idiosyncratic, and hence, uninformative about the hospitals quality control efforts.If Graph 1 represented actual data, we would conclude that there is no discernible trend in CAUTI for this hospital, and no variation from the trend that is unexpected.
The third data exhibited in the first graph are the gray lines symmetrically bracketing the regression line, from above and below, and generally enclosing the blue dots.These are the 95% prediction intervals which should capture the blue dots 95% of the time.That is, 19 out of 20 times, we should expect the blue dots to lie within these limits if CAUTI rates follow a linear trend (including, of course, no trend at all).As we illustrate below, data lying outside of these confidence intervals are notable, unexpected results that need to be investigated, either to improve (if the CAUTI rate is above the upper confidence interval lines) or to emulate (if the CAUTI rate is below the lower confidence interval line).

B. Spotting Anomalies in Trend: Serial Correlation and Changes in the Trend Line
Even without a trend in CAUTI rates, it may be the pattern of residuals may be informative about episodic hospital practices (changing policies, changing personal, or systematic seasonal variation).An example is provided by the second regression graph, Graph 2: Graph 2 exhibits 'serial correlation' of the residuals, which suggest time-related and correlated deviations in quality outcomes.Serial correlation is present when the sign of the residuals tend to change non-randomly: for example, for months 6 through 11, the residuals are all positive indicating worse than expected (according to the trend line) results.(Note also better-than-trend results for a three month period from month 3 to 5).In Graph 1, no such systematic positive or negative runs were found in the outcomes.
Finding serial correlation in the residuals is not necessarily an indication of a quality control problem (the question of a changing trend is different, and we deal with that next).The serial correlation may be due to factors outside the hospital's control, related perhaps to seasonal variation in health problems for certain segments of the population, such as an influx of older patients in the winter (or summer, depending upon location) due to seasonal migration.On the other hand, detecting serial correlation in health outcomes suggests an evaluation of policy and personal that may be correlated with these nonsystematic outcomes.For example, several new orderlies may have been hired in month 7, and the higher rates may be reflecting poor sterile techniques.If this were the case, the results suggest perhaps better training for new orderlies.Hence, serial correlation is a warning gauge but not necessarily a problem that the hospital can address.At a minimum, additional data comparison of the serial correlation with hospital procedural changes, if present, would be advisable.To distinguish Graph 1 results from Graph 2 the administrator employs standard tests for serial correlation in regressions, paying attention to unexplained, but systematic, variation of the respective outcome.

C. Change in Trend
At first glance, Graph 3 appears to be a serial correlation problem, for example, with negative residuals from month 3 through month 9.There is one significant difference, however, between these residuals in Graph 3 and those in Graph 2.Where the residuals in Graph 2 have the shape of 'smooth' wave, the changes in Graph 3, particularly from month 6 onward, are systematically increasing.Such constantly increasing (or constantly decreasing) residuals are an indication of a shift in the trend line, perhaps associated with a change in hospital procedure or unmeasured patient risk.
Spline functions (also called piece-wise linear regressions) are used to detect, and measure shifts in the trend line.Spline function analysis may be particularly useful in the analysis of the effect of changing technology, modes of patient treatment, and changing environmental risks and incentives.With enough data, multiple changes in the trend line can be examined.However, as quality control needs to generate interpretable results, over shorter periods, looking for one change in the trend line is generally sufficient.Allowing for more than one change per period, without significant indications of such multiple changes, would introduce more complexity into the process of detection at the expense of clarity of interpretation.
Though the appendix contains SAS code that allows the data to indicate when the break occurs in the trend line, in our particular graphical example here, it is obvious that it likely occurs at month 6.Allowing for such a change in this data, we find the following, statistically significant change in the trend of CAUTI rates as indicated in Graph 4: Changes in the slope are often not so obvious, or an apparent change may be spurious.Note that once we allow a shift in the trend of CAUTI rates, the 95% confidence interval around the line in Graph 4 is much smaller than the 95% confidence interval in Graph 3: that is, once we get the right model for the trend in CUATI rates, detecting outliers becomes much easier as the confidence interval narrows.

D. Evaluating Current Outcomes Relative to the Baseline Trend: Outlier Analysis
Having outlined the regression graph approach for finding the right model for trends in CAUTI rates, and explained confidence intervals for such trends, perhaps the most intuitively important statistic for quality monitoring is this month's CAUTI rate relative to what is expected based on past experience.In particular, the question we address becomes: Is this month's CAUTI rate from our hospital unusually high (a bad outcome), or unusually low (a good outcome), given CAUTI rate trends for our hospital?This monthly updating of quality monitoring is relatively simple using regression graphs: we simply test for whether the most recent CAUTI rate lies outside of the 95% confidence of the trend line generated by prior data.If it lies above or below or 95% interval, then it is unusually large (or small) given historical trends.A graphical example where this is the case (indicating very bad outcomes in the most recent, 13 th , month) is given as follows for month 13, and observing the prior 12 months of data as indicated in Graph 5.
The confidence interval for this regression graph was generated for the data included months 1 through 12, and then month 13's result was superimposed on the graph.Clearly the 6.5 CAUTI rate in month 13 is well above 5.7, the approximate upper bound for the 95% confidence interval.6.5, then, is an unusually large value given historical trends and would need to be investigated as a highly unusual outcome.
Notice that we also can examine if past outcomes where outliers using this analysis.In particular, we can form a regression baseline from 2012 data, and ask if the mean value during 2011 where usually low or high relative to that baseline.If, for example, the 2011 mean was much higher than what the 2012 baseline would predict, then the hospital is doing much better in 2012 than it did in 2011.

III. COMPARING HOSPITALS: THE "EXCHANGE INDEX"
The graphical approaches to monitoring in section II above provide internal monitoring of quality in the sense of comparing a hospital's current outcomes with its own historical trends.Additionally, we compare a hospital's outcomes to those of other hospitals.Regression graphs readily accommodate such comparisons through the 'exchange index' (our name for this statistic).The exchange index, constructed from regression graph comparisons, avoids the possible misleading impression that a hospital may get from only a point in time comparison.That is, comparing my hospital's mean with the average mean will be misleading if its mean happens to be higher this month for some idiosyncratic reason.The exchange index avoids this problem by comparing regression graphs with each other, essentially using all the available information.
The exchange index compares two regression graphs by computing the following number: if our regression graph were switched to the other hospital's regression graph, how much of a change in the distribution of outcomes would there be? (Technically, the exchange index is the average percentile shift in distribution, multiplied by 2, in order to make the upper bound plus 1 and the lower bound minus 1).The index is 1 if our hospital's CAUTI rates are always better than the comparison's hospital rates (so there is no overlap in CAUTI rates for the comparison period, in the context of the graphs above).The index is 0 if, on average, the CAUTI rates are about the same for the comparison period.And the index is -1 if our hospital's CAUTI rates are always worse than the comparison's hospital's rates.The index is always bounded between plus one and minus one, and is not affected by any linear transformation of the outcome variables of interest.For example, weight associated outcomes would yield the same exchange index value whether measured in pounds or kilograms.Exchange index values would be the same whether rates were measured as decimals or percents, or absolute number of cases (if the at-risk patient pool is held constant).A numerical example, with accompanying graph, is given in Butler, Johnson, and Wilson for a very different context [9].
A typical use of the exchange index would be to report a given hospital's CAUTI exchange index value relative to all other hospital's in their comparison group.This uses all available trend data for comparison, and the exchange (as indicated by Butler and McDonald [8]) is a type of Gini coefficient with the usual desirable welfare properties of that class of index functions.

IV. MORE THAN JUST TIME, THESE REGRESSION QUALITY INDEXES ALLOW MULTIPLE DETERMINANTS
Since all the indexes above are based on the graphical residuals or on the predicted outcomes under the exchange index, our indices readily accommodate any number of additional control variables.For example, besides month of report, we can also include type of hospital, patient capacity, staff to patient ratios, location indicators, etc, as controls when generating quality indexes if the participating hospitals are willing to supply the information.This flexibility helps to ensure the results are robust to any number of other measured control variables.
As an application, consider hospital-care related health conditions for short-term acute hospitals from Arizona.Arizona is especially useful as it provides evidence of the influence of metropolitan information networks, including inter-physician information networks (physicians admitting patients to multiple hospitals within the same dense, geographic region), and formal and informal training/monitoring externalities in metropolitan areas, as well as competition for better care induced by choice in a compact geographical service area.Maricopa County excluding the sparsely settled Indian reservations contains over half of Arizona's population, but is less than one twentieth of the state's total land mass.The hospitals within the Maricopa county are much closer together, have many more overlapping physicians admitting patients, and so are subject to more competitive pressures to minimize hospital-induced bad health outcomes.The hospital characteristics come from the American Hospital directory website [10]; the health care outcomes data from a Medicare website [12].
On the other hand, hospitals outside of Maricopa county are not only smaller on average, but more likely to have stronger community ties and social capital, including a greater likelihood that one or more the staff knows the patient or a member of his/her family.So the net effect of Maricopa county location is uncertain on deleterious hospital related outcomes: market pressures and inter-physician networks on the one hand, versus closer personal knowledge of the patient and a sense of 'hometown' care on the other hand.
To examine the hypothesis that competing metropolitan area hospitals provide better care than less densely populated rural hospitals, we ran multiple regressions of four hospital-related health outcomes: blood infections, falls and injuries, urinary tract infections, and bed pressure sores, and regressed these on indicators of hospital technology: staffed beds, number of discharges within the last year, and total patient days for the last year.Since the regression analysis under-laying the exchange index calculation compares the metro regression function for each type of hospital-induced health outcome with the rural regression function, at the same values of the independent variables (in this case, at the metro hospital values of the independent variables), the exchange index provides an apples-to-apples comparison of differences in outcomes.We are viewing the treatment outcome as metro location, with the control outcome as non-metro (i.e., not in Maricopa county), so that a positive index value is a good thing for metro counties (ceterus paribus, they are doing a better job), a zero index indicates no difference, and a negative index value indicates worse outcomes.Again, the index is constructed so that it is bounded between plus and minus one.
Table 1 indicates a mixed result for metro location: blood infections are lower outside of Maricopa county (exchange index of -.475), but falls and injuries (exchange index of .515),and urinary tract infections (exchange index of .490)are lower in Maricopa county hospitals.When holding the hospital characteristics constant, there is little difference in bed pressure sores (the exchange index is -.145, close to zero but indicates a slight advantage to non-metro hospitals).The unadjusted mean differences in health outcomes have the same qualitative indication, but the regression-adjusted exchange index indicates a different quantitative magnitude.The exchange index provides an apples-to-apples comparison that unadjusted means cannot.In particular, the exchange index for urinary tract infections indicates a more significant metro difference than the unadjusted mean difference in the rate per 1000 of .0532(=.2695-.2163).
A big advantage of the exchange index, besides using all the information embodied in both the metro and non-metro regression functions, is that a percentile shift for each hospital (that is, an exchange value for each hospital) can be employed in subsequent analysis to explain differences in outcomes within and between the metro/non-metro partition used here, relating hospital specific values to such things as human resource management structure at the hospital, training programs, staff-tenure at the hospital and in the profession, etc.
We illustrate this second useful aspect of our index by focusing on differences in Falls and Injuries, and differences in Urinary Tract Infections, just for the sample of metro hospitals using the hospital specific exchange index values.As we partition the sample for the Table 2 analysis to just the metro area, all areaspecific factors (metro information networks, competition between hospitals in Maricopa county, multiple-hospital admitting privileges by physicians, etc.) are all held constant.We examine scale in two ways: fixed hospital size (staffed beds) and patient throughput given size (patient days).Recall that a positive index variable for each hospital represents a better outcome (fewer falls and infections).Table 2 indicates hospital size (beds) are associated with fewer falls, holding days constant, and patient throughput (days) is associated with a lower urinary tract infection rate holding hospital size constant (beds) among Maricopa county hospitals.These associations are statistically significant.

V. CONCLUSION
This paper explored the use of regression analysis in explaining trends in hospital quality outcomes, including the introduction of a new index of hospital quality, the exchange index.Using data from Arizona hospitals, we indicate how this index can be use to measure and compare both large scale differences in area outcomes (Table 1), as well as between-hospital quality differences at the individual hospital level (Table 2).The exchange index is advantageous for several reasons including its ability to allow for as many factors as can be measured, including its potential for measuring outcome differences relative to hospital management structure.Careful regression analysis will prove more and more useful for hospital measurements as hospital data becomes increasingly available.The methods presented in this paper will be valuable for researchers and hospital management in understanding and examining hospital quality.
In our application of 2011 Arizona Hospital data using the exchange index, we find hospitals in the metropolitan Maricopa county area have fewer falls and injuries, and fewer catheter-associated urinary tract infections than those outside of the Phoenix area (Maricopa county).When we restrict our analysis to just metro hospitals, we find hospital scale (beds) reduces fall and injuries holding throughput (days) constant, while hospital days are associated with fewer urinary tract infections holding hospital scale (beds) constant.

ACKNOWLEDGEMENT
We appreciate the assistance of Grant Gannaway on early versions of this paper.

APPENDIX OF ANNOTATED SAS PROGRAMS FOR THE ANALYSES EMPLOYED
*delete the old stuff from memmory; proc datasets lib=work memtype=catalog; delete regression; run; data example1; *baseline case with no outliers, no serial correlation; exchange_index=(total_perc_diff/12) * 2; *exchange index is average perc_diff times 2; put exchange_index= ; *will be write the value of the Exchange index to the sas log file; end; run; proc print; run;

Graph 3 :
Changes in Trend: When the residuals keep increasing in value.

Graph 4 :
Trend Shift Measured with a Spline Function.

Graph 5 :
An Outlier in the Latest Month.

Table 1 : 2011 Hospital-Related Rates of Health Outcomes and the Exchange Index (Std dev)
Note: the underlying SAS code, data and output are available upon request.

Table 2 : 2011 Analysis of Metro Area Hospital Quality: Physical Size vs. Patient Flow (Probability Significance Level)
N=35 observations on Maricopa county hospitals in 2011.Hospital discharges where highly correlated with patient days (VIF=52), and so are excluded from these regressions due to collinearity.Again, the SAS data and programs for these results are available upon request.