JSTOR

Methods

Hospitals were recruited for the EPIC study through a mailing to the SHEA membership in June 1998.^13,14 The EPIC study consisted of the following 2 components: process assessment and indicator evaluation. Of the 61 hospitals completing the EPIC study, 28 participated in this indicator‐evaluation component (22 of them also participated in the process‐assessment component). Representatives of each participating hospital received a data‐collection manual and additional indicator evaluation training by 1‐hour teleconference calls.

The BSI indicators were identified by a member of the research team through a 1998 mailing to the leaders of 211 performance measurement systems listed in the JCAHO ORYX initiative. An ORYX‐listed performance measurement system is an entity or organization with an automated database that facilitates performance improvement in healthcare organizations by collecting, analyzing, and disseminating process and outcome measures of performance from multiple organizations.¹⁵

Fourteen measurement systems were found to use BSI indicators, and their leaders were invited to participate in the study. Because this project was unrelated to the ORYX initiative, participation by a measurement system was entirely voluntary, and confidentiality was assured. The leaders of 6 measurement systems agreed to participate and provided their BSI indicator specifications; the leaders of the other 8 systems did not respond. Appendix A describes how the BSI indicators differed on the following: association with a central line; inclusion of primary and/or secondary BSIs; data sources (clinical sepsis, laboratory‐confirmed BSI, and ICD‐9‐CM codes); scope of measurement (hospital wide or ICU specific); risk‐adjustment; and exclusion criteria.

To identify all unique data element requirements, project staff conducted a detailed comparison of the measure specifications. These elements were then divided across 2 data‐collection forms from which the different rates could be derived. One form was based on automated searches of administrative data, and the other was based on concurrent surveillance of clinical data. A third form was developed for counting ICU central‐line days according to the NNIS method. Each participating hospital was assigned a confidential identifier.

Hospitals submitted data for 2–5 of the indicators. Data from 1999 were submitted twice (discharges from January through March and from April through September) so that the influence of seasonal and organization‐related fluctuations in rates could be minimized. The study protocol was approved by institutional review boards affiliated with the CDC; the University of Tennessee Health Center, Memphis; and the JCAHO.

All data were carefully reviewed by project staff before entry into the database; hospital staff were contacted for follow up as needed to obtain missing values or to correct inappropriate values. Indicator algorithms were programmed in SAS, version 8 (SAS Institute), which was used to generate rates, rankings, and correlations according to the specifications of each indicator. Hospitals received customized reports that included a graphical display of each hospital’s ordinal rank, percentile rank, and z score, along with a flowchart of each indicator algorithm. The z score standardized the hospital‐specific rates to a common metric (the number of standard deviations from the mean for each indicator) and was not substantially affected by different numbers of hospitals. Spearman rank correlation was used to assess the correlation of rankings across indicators.

Results

Hospital characteristics. Table 1 presents the characteristics of the 28 participating hospitals. The number of beds ranged from 100 to 1351 (median, 449.5 beds; mean [±SD], beds). A total of 21 (75%) of the 28 participating institutions were teaching hospitals. Hospitals in the United States were located in the following geographic regions: East (46.4%), Midwest (17.9%), South (14.3%), and West (7.1%). Four hospitals (14.3%) were located outside the United States. Most hospitals (85%) chose to collect data in their medical‐surgical ICU or medical ICU. Twenty‐four hospitals collected data for the indicators that required clinical data, and 20 sites collected data for the indicators that required administrative data.

Table 1.  Characteristics of Hospitals Participating in the Evaluation of Processes and Indicators in Infection Control Study

Open New Window

Bloodstream infection rates by indicator. Table 2 presents the aggregated BSI rates for each indicator. The range of rates varied with the scope of the indicator population (ie, ICU specific or hospital wide). The median BSI rates for the indicators based on ICU‐specific clinical data (indicators C, F, and E) varied slightly, ranging from 2.23 to 2.91 BSIs per 1000 central‐line days. In contrast, the median rates for indicators based on hospital‐wide administrative data varied widely, ranging from 0.046 to 7.03 BSIs per 100 patients.

Table 2.  General Specifications of Bloodstream Infections (BSIs) for Each BSI Indicator

Open New Window

Bloodstream infection rates, rankings, and z scores by hospital. Table 3 presents a comparison of hospital rates and rankings across the indicators. Because the specifications for indicators C and F were identical (except for the exclusion of hospital length of stay <2 days), they yielded the same rates for all hospitals and are presented in a single column of the table. Rates and rankings varied substantially for indicators based on administrative data but were more consistent for indicators based on clinical surveillance data. Not surprisingly, substantial differences appeared when rates based on administrative data were compared with those based on ICU surveillance. For example, site 126 was at the 90th percentile for indicator B and at the 12th percentile for indicator C/F. Rates based on ICU surveillance (indicators C/F and E) were identical for 15 of 24 hospitals. When differences occurred, the rates were usually higher in system E because of the inclusion of secondary BSIs.

Table 3.  Hospital Rates, Ordinal Rankings, and Percentile Rankings Across Bloodstream Infection Indicators from Different Measurement Systems by Hospitals

Open New Window

The variation in hospital performance when rates were standardized to z scores is presented in the Figure for the 10 hospitals that provided data for all indicators. The figure contains 2 bars for each hospital. One displays rates based on indicators derived from administrative data (indicators A, B, and D), and the other displays rates based on clinical surveillance data (indicators C/F and E). For all but 1 hospital, the range of values based on administrative data was much wider than that of values based on clinical data. The rates for 8 hospitals were both greater than and less than the mean; the rates of the other 2 were consistently either all greater than or all less than the mean across all indicators.

(89 KB)

Figure.  z Scores of bloodstream infection (BSI) rates derived from administrative and surveillance‐based data. Ten hospitals collected data for all indicators. BSI rates are standardized to z scores as follows: [hospital system rate] / standard deviation of the system rate. ICU = intensive care unit; ID = identification.

Open New Window

Correlation of BSI indicators. Table 4 presents the correlation between pairs of indicators for hospitals collecting data for both indicators. Although each hospital used administrative data to identify BSIs, there was essentially no correspondence between indicators A, B, and D. The strongest correlation among the 3 was between indicators B and D ( ; ). The correlation was much stronger between the clinically derived indicators C/F and E ( ; ). Among the 24 hospitals reporting data for both indicators, the rank for hospital 153 differed by 14 places, and for hospitals 106 and 153, performance was better than the median for one indicator and worse than the median for the other indicator. There were weak correlations between indicators of different types (ie, administrative or clinical indicators). Indicators A and D were both moderately correlated with C/F and E; the strongest correlation was between C/F and D. Indicator B, however, was not correlated with either clinically based indicator.

Table 4.  Correlation Between Different Bloodstream Infection Indicators

Open New Window

Discussion

This study investigated the extent to which variation in BSI indicator specifications affected infection rates and rankings of hospital performance. We found that absolute rates, ordinal ranks, and percentile ranks are strongly influenced by differences in the definition and collection of an indicator. Examination of the correlation of hospital rankings across the indicators revealed that, in many instances, there was no correspondence between several of the indicators in the assessment of relative performance. Only when both indicators were calculated by using clinical surveillance in an individual ICU was the correspondence acceptable ( ). Although BSI rates seem to be a logical choice for use as an indicator, the selection of indicator specifications results in remarkably different conclusions about hospital performance.

As demonstrated by this study, indicator specifications are often quite complex, and there is ample opportunity for error. For example, if an infection is to fulfill the NNIS surveillance definition of a primary BSI, the main site of infection must be the bloodstream, the infection must be either laboratory‐confirmed BSI or clinical sepsis, and there must be no apparent or laboratory‐confirmed infection at another (ie, secondary) site.¹⁶ Clinical skill and experience are needed for determining that the infection was neither due to contamination from common skin bacteria nor acquired in the community rather than in the hospital. Even the process for counting line‐days in the denominator is susceptible to error if multiple lines per patient are inappropriately counted.¹⁴

Common terminology and coding conventions add to the complexity. For rates derived from administrative claims data, the terms “primary” and “secondary” refer to the diagnosis code rather than to the BSI.¹⁷ (The primary diagnosis refers to the diagnosis, taken from the list of secondary diagnoses, that is responsible for most of the care given to the patient, other than the care required for the principal diagnosis. Secondary diagnoses include all other diagnoses except for the principal diagnosis, which is the diagnosis that caused the admission to the hospital.) Infections of the bloodstream can be coded in a number of ways, such as bacteremia, fungemia, septicemia, and generalized sepsis. Complication codes combine infections associated with vascular devices and those associated with prostheses, implants, and grafts; there are no specific codes for infections associated with central lines.¹⁸

Although the use of administrative data is appealing because of the relative ease of data collection, the exclusive use of such data in calculating indicator rates for hospital comparisons remains controversial for several reasons, including limitations in the accuracy of coded information and the need for large sample sizes with low rate events.^19‐23 Administrative data have not been useful in detecting patients with central venous catheters.²⁴ However, an approach that calculated BSI rates by applying algorithms to automated clinical data was recently considered to be an accurate alternative to manual collection of data.²⁵ Like differences in data sources, differences in approaches to data collection can also influence rates. Data collected prospectively have been shown to yield different BSI rates than data collected retrospectively.²⁶ Similar results have been found when surgical site infection rates derived from administrative data are compared with those derived from clinical data.^27,28

Analysis of indicator rates can also be confounded by errors in data quality, algorithm programming, choice of analytic procedure and statistical test, and report display. Certainly, whether the data have been appropriately risk adjusted for differences in case mix, central line use, or both makes a difference.²⁹ In this study, multivariable risk adjustment for patient factors was not part of the analysis for any indicator. However, each indicator narrowed the eligible patient population in some way, so that differences in patient risk were accounted for. For example, some indicators restricted the population to patients in a specific type of ICU; another excluded patients with AIDS or cancer.

This study has several limitations. Hospitals participating in the study are not likely to be representative of the population of US hospitals. Because we did not intend to draw conclusions about the validity of the BSI rates, generalizability was not a primary concern. The indicators examined were those available at the beginning of the study and do not necessarily correspond with the indicators in use today. However, the 4 BSI measures in the National Healthcare Quality Report are confounded by many of the same issues identified in this study. Two of the 4 measures were derived by applying NNIS methodology to clinical data, and 2 were derived from administrative data from the Healthcare Cost and Utilization Project.³⁰ Appendix B provides additional information about these measures.

Another limitation of this study is that not all hospitals reported all of the BSI indicators; this lack of information introduces additional variation in ranking. Also, because we did not receive feedback from all solicited measurement systems, the specifications do not necessarily reflect all of the ways in which BSI rates were defined and measured. For indicator A, we learned during quality checks of the data that 4 hospitals may have inadvertently included secondary BSIs in their data. This inclusion could have inflated their rates, but the specific cases could not be identified.

On the basis of our findings, we conclude that different indicators purporting to measure the same events should not be compared, unless the rates are defined and operationalized identically across entities. Without standardization, variability in indicator‐rate specifications can yield widely different evaluations of hospital performance. Theoretically, if given complete information about how measures are specified and collected, persons could choose to participate in those systems from which they get the best results. In reality, hospital staff who collect data for the measurement systems are rarely provided with information about exactly how the measure is calculated.

To improve standardization of indicators, disparate stakeholders are now more often working together to standardize measure specifications.^31,32 For indicators that still lack standardization, measurement systems should provide transparent measure specifications and full disclosure. Persons involved in performance measurement initiatives should strive to thoroughly understand the specifications and issues that affect the accuracy of the results.

The question of how best to define and measure BSI rates remains unanswered. Because this study was not designed to assess the validity of the specifications, we cannot recommend one rate over another. Ideally, validity would be determined by expert consensus, traditional methods (eg, sensitivity, specificity, and positive predictive value), a demonstrated relationship to care processes, and demonstrated usefulness to those desiring the information. In actuality, establishing the validity of an indicator is challenging and costly, and consensus is rarely achieved. To date, the NNIS system, which uses specifications based on ICU surveillance data, is one of the few approaches with demonstrated validity and reliability.^26,33 The NNIS approach, however, is quite resource intensive. An effective simplification of the NNIS approach was found to have a greater reliance on microbiological data,³⁴ although one study did not confirm this finding.³⁵

Additional studies are needed for determining the accuracy and reliability of indicators and the burden associated with data collection. Policy makers should explicitly state which limitations in accuracy they are willing to accept to ensure that the burden of data collection remains acceptable to those required to perform it. Finally, studies that measure the value of performance measurement in terms of costs and benefits to all stakeholders are sorely needed.

Although the science of performance measurement is still considered to be young, recent national initiatives that tie payment with performance on indicator rates have heightened the need for studies that establish the validity and utility of indicator information.³⁶ In 1998, Emori et al.²⁶ concluded that data integrity is essential to public disclosure of infection rates and can be accomplished only when an ongoing and objective method of assessing the quality of the data is included as an integral part of the surveillance or measurement system. It remains to be seen whether proponents of the use of infection‐rate data for public reporting and reimbursement thoroughly understand and heed this important message.

Contributing Members of the EPIC Study Group

This indicator evaluation is 1 component of a larger study entitled “Evaluation of the Processes and Indicators in Infection Control” (EPIC).¹³ The EPIC study is part of an ongoing collaboration between the Society for Healthcare Epidemiology of America (SHEA) and the Joint Commission on Accreditation of Healthcare Organizations (JCAHO) and is intended to support the development, understanding, and effective use of clinical indicators.¹²

SHEA‐member epidemiologists and staff from hospitals participating in the EPIC Study Indicator Evaluation Component included Linda Matrician, RN, CIC; Michael F. Parry, MD; Diane Baranowsky, RN; Brenda Grant, RN; Richard J. Duma, MD, PhD; Jacquelyn Seibert, RN; Francis J. G. Liu, MD; Mary J. K. Kim, MSPH, CIC; Janine Chapman, RN, BSN; Robert L. Pinsky, MD; Jean Maurice, RN; Terri Bethea, BSN; Janet Moody, RN, BSN; Jeffrey Engel, MD; Sue Barnett; Peter A. Gross, MD; Cristina Cicogna, MD; Peg Janasie;Martin Topiel, MD; Carol Ward, RN, CIC; Chatrchai Watanakunakorn, MD; Mary Kundus, RN, CIC;, W. Lee Fanning, MD; Mary Dahlmann; Paul M. Newell, MD; Ann Schlimm, RN; Wesley Kozinn, MD; Joan Kies, RN; Leonard Mermel, DO, SCM; Steve Parenteau, MT (ASCP), MS; Richard Rose III, MD; Patricia Lawson; John S. Adams, MD; Margaret Chambers, RN; Pam Falk, MPH; Gregory Bond, MSN, RN, CIC; Jane Lane, RN; Edward Wong, MD; Donna Winborne, RN, BSN; Michael Lamacchia, MD; Marie Rella; Claude Tremblay, MD, FRCP; Stefan Weber, MD, MS; Kenji Kono, MD; and Carol Jarvis, CIC.