Validation of Statewide Surveillance System Data on Central Line–Associated Bloodstream Infection in Intensive Care Units in Australia
Objective. To measure the interobserver agreement, sensitivity, specificity, positive predictive value, and negative predictive value of data submitted to a statewide surveillance system for identifying central line–associated bloodstream infection (BSI).
Design. Retrospective review of hospital medical records comparing reported data with gold standard according to definitions of central line–associated BSI.
Setting. Six Victorian public hospitals with more than 100 beds.
Methods. Reporting of surveillance outcomes was undertaken by infection control practitioners at the hospital sites. Retrospective evaluation of the surveillance process was carried out by independent infection control practitioners from the Victorian Hospital Acquired Infection Surveillance System (VICNISS). A sample of records of patients reported to have a central line–associated BSI were assessed to determine whether they met the definition of central line–associated BSI. A sample of records of patients with bacteremia in the intensive care unit during the assessment period who were not reported as having central line–associated BSI were also assessed to see whether they met the definition of central line–associated BSI.
Results. Records of 108 patients were reviewed; the agreement between surveillance reports and the VICNISS assessment was 67.6% (
). Of the 46 reported central line–associated BSIs, 27 were confirmed to be central line–associated BSIs, for a positive predictive value of 59% (95% confidence interval [CI], 43%–73%). Of the 62 cases of bacteremia reviewed that were not reported as central line–associated BSIs, 45 were not associated with a central line, for a negative predictive value of 73% (95% CI, 60%–83%). Estimated sensitivity was 35%, and specificity was 87%. The positive likelihood ratio was 3.0, and the negative likelihood ratio was 0.72.
Discussion. The agreement between the reporting of central line–associated BSI and the gold standard application of definitions was unacceptably low. False‐negative results were problematic; more than half of central line–associated BSIs may be missed in Victorian public hospitals.
Received January 8, 2009; accepted June 15, 2009; electronically published October 2, 2009.
Bacteremia associated with the use of central lines (central line–associated bloodstream infection [BSI]) is a common,1 preventable cause of patient morbidity and mortality2 and of increased length of hospital stay and cost.3 Nosocomial BSI is estimated to be the eighth leading cause of death in the United States.1 A substantial proportion of deaths due to nosocomial BSI are due to central line–associated BSI. Coello et al4 found that device‐related infection accounted for 52.4% of cases of hospital‐acquired bacteremia in teaching hospitals and 43.2% in nonteaching hospitals.
Central line–associated BSI can be prevented by means of multifaceted systems interventions.5 Pronovost et al showed that a sustained reduction in central line–associated infections, from 7.7 to 1.4 cases per 1,000 catheter‐days, could be achieved if concurrent measures were instituted.6 The recommended procedures were hand washing, the use of full barrier precautions during the insertion of central venous catheters, cleansing of the patient's skin with chlorhexidine, avoidance of the femoral site, and the removal of unnecessary catheters.
The use of comparative data and feedback to individual hospitals can lead to a reduction in infection rates, and it also can be a guide to monitoring the effectiveness of other interventions7 and can alert hospitals that have unacceptable rates of central line–associated BSI. For this reason, infection prevention centers such as the National Healthcare Safety Network in the United States and the Victorian Hospital Acquired Infection Surveillance System (VICNISS) in Victoria, Australia, have determined to collect data on rates of central line–associated BSI in intensive care units (ICUs).
Infection surveillance programs conducted across multiple sites and during an extended time period must be consistent. Standard definitions and methods must be used, and data quality assessed.8 Reported rates of central line–associated BSI are being used to evaluate hospital units and determine which have high rates of infection.9 Such data will mean little if there is inconsistency in the application of definitions of central line–associated BSI among facilities.
The surveillance methods previously used for hospital‐acquired infection include a retrospective review of medical records10 and prospective methods, centered around the laboratory and wards.11 Without an objective gold standard, validation of a surveillance system is challenging. One study revealed that for primary BSI, the correlation between retrospective medical record review and surveillance reporting was particularly good.12
The VICNISS collects data on hospital‐acquired infection in Victoria, Australia. Hospitals with ICUs participate in reporting rates of central line–associated BSI. The hospitals are categorized as type 1 (hospitals with more than 100 beds, except large metropolitan hospitals), type 1A (large metropolitan hospitals), or type 2 (smaller hospitals). At type 1 or 1A hospitals, infection control practitioners (ICPs) are provided with standard definitions and asked to assess each case of bacteremia in the ICU and report those due to a central line.
The VICNISS was put in place to ensure consistency of monitoring of healthcare‐associated infection, with a view to external benchmarking. ICPs undergo centralized training on the application of VICNISS definitions, and a single individual at each hospital interprets definitions from a centralized manual to ensure maximum consistency of application of definitions. However, such a process requires validation that definitions are being used accurately and consistently across all sites.13
The objective of our study was to measure the accuracy of the surveillance data on central line–associated BSI. We used a retrospective medical record review and a laboratory review as the gold standard and determined the sensitivity, specificity, positive predictive value, and negative predictive value of surveillance data on central line–associated BSI submitted to the VICNISS by ICPs.
Methods
Study Site and Design
Three of the 6 participating large metropolitan (type 1A) hospitals and 3 of the 12 other large (type 1) hospitals were chosen randomly among the hospitals participating in this surveillance and reporting activity. At the 6 selected hospitals, a retrospective review of medical records was conducted of central line–associated BSI surveillance data collected during the period from January 1 through December 31, 2006. In the 6 selected hospitals, the surveillance data had been collected by trained ICPs who were asked to collect data on all adult (age, more than 16 years) patients who developed a primary central line–associated BSI while in the ICU.
Definition of Central Line–Associated BSI
Cases diagnosed as central line–associated BSI must meet the following VICNISS criteria, which are based on those of the US National Healthcare Safety Network system:
Criterion 1. Blood culture yields a recognized pathogen from 1 or more blood samples, and positive laboratory results are not related to an infection at another site.
Criterion 2. Patient has 1 of the following signs or symptoms: fever (temperature, greater than 38°C), chills, or hypotension; the signs and symptoms and positive laboratory results are not related to an infection at another site; and at least 1 of the following is present: (a) common skin contaminant is cultured from 2 or more blood samples drawn on separate occasions; (b) common skin contaminant is cultured from at least 1 blood sample, and the physician institutes appropriate antimicrobial therapy; (c) result of antigen test of blood sample is positive.
This definition of central line–associated BSI was based on the National Nosocomial Infections Surveillance system definition, which has now been updated.14
Cases that appeared on a list of central line–associated BSIs as reported by the ICP to the VICNISS were regarded as “surveillance positive.” Microbiology laboratories produced lists of patients with bacteremia who were in intensive care at the participating study sites from January 1 through December 31, 2006, and who were not reported as having central line–associated BSI; their cases were regarded as “surveillance negative.”
Data Collection
A sample of cases of bacteremia that were not reported as associated with a central line and a sample of reported central line–associated BSIs were scrutinized by use of the following algorithm: up to 10 results reported as positive and 10 results reported as negative were selected from each hospital with under 100 cases of bacteremia recorded during the study period, and up to 20 results reported as positive and 20 results reported as negative were selected from each hospital with more than 100 cases of bacteremia recorded during the study period. For hospitals with enough cases to necessitate the taking of a sample, a random number generator was used to determine which medical records would be reviewed. The actual number of medical records reviewed for each combination of result (central line–associated BSI or BSI not associated with a central line) and hospital was therefore the minimum of the number reported and 10.
A VICNISS‐trained ICP who was unaware of the surveillance status of each case collected data by conducting a retrospective review of each selected patient's medical records, laboratory results, and radiology and microbiology notes. A VICNISS investigator (J.B.) made an assessment on the basis of the medical history and laboratory results of the patient. Results were discussed with an infectious diseases physician and with another senior ICP (the validation team), who were unaware of the surveillance status of each case, and a consensus was reached. The blinded review classifications were compared with the cases reported to the VICNISS as central line–associated BSI, and the extent of agreement between the blinded assessments of central line–associated BSI and the cases of bacteremia that had been reported to the VICNISS was determined, as was the extent of agreement between the blinded assessments of BSI not associated with a central line and the cases of bacteremia that had not been reported to the VICNISS.
Analysis
The review classification of infection was used as the reference or gold standard. The positive predictive value and negative predictive value were calculated directly from a contingency table by using data from ICP surveillance and the review classification.
To estimate sensitivity and specificity, we first looked at the sample of patients reported to the VICNISS as having central line–associated BSIs (ie, surveillance‐positive cases). We assumed that the proportion of patients in the sample whose surveillance results were false positive (by comparison with the gold standard) was representative of the proportion of false‐positive results for all patients reported as having central line–associated BSIs. The number of true‐positive and false‐positive findings detected in the sample was therefore divided by the surveillance‐positive sampling fraction. Similarly, with the sample of patients who were not reported to the VICNISS, we determined the number of reported negatives that were false negatives (surveillance‐negative cases that were positive according to the gold standard) and assumed that this proportion was representative of all patients who were not reported to the VICNISS, by dividing by the surveillance‐negative sampling fraction. This allowed the calculation of sensitivity and specificity from the standard contingency table. This approach has been used in the past,15,16 and the method has been described and validated elsewhere.17,18
Statistical analyses were performed with Stata software, version 10.0 IC (StatCorp). This study was reviewed and approved by the institutional research review boards at the 6 participating hospitals.
Results
In 2006, the rates of central line–associated BSI among hospitals in Victoria that participated in the surveillance system were 5.6 cases per 1,000 central venous catheter–days for large hospitals and 1.2 cases per 1,000 central venous catheter–days for small hospitals. In the subgroup of studied hospitals, 398 cases of bacteremia occurred in patients with central venous catheters in ICUs during the study period (ie, 2006). Of these, 81 were reported as central line–associated BSIs, and 317 were not reported as associated with a central line. A sample set of 46 medical records of patients reported as having a central line–associated BSI and 62 medical records of patients not reported as having a central line–associated BSI were randomly chosen for review.
Table 1 shows the outcome of the review process of the 108 sampled medical records. Table 2 shows the number of true positives, true negatives, false positives, and false negatives that would be expected for all 398 cases of bacteremia, by using the review by the validation team as the gold standard. The values in Table 2 were obtained by dividing the numbers in Table 1 by the sampling ratio, as described in other studies.15,17,18
Interobserver Agreement
For the 108 sampled medical records, there was 67% interobserver agreement between the central line–associated BSI report by the ICP and the VICNISS review, with a κ score of 0.31 (
), which suggests fair agreement.
Positive Predictive Value, Negative Predictive Value, Sensitivity, and Specificity
Using the VICNISS investigator findings as the gold standard, we found that of the 46 reported central line–associated BSIs, 27 were confirmed to be central line–associated BSIs, for a positive predictive value of 59% (95% confidence interval [CI], 43%–73%). Of the 62 cases of bacteremia reviewed that were not reported by the local ICPs as central line–associated BSIs, the VICNISS investigators found that 45 were not associated with a central line, for a negative predictive value of 73% (95% CI, 60%–83%).
The sensitivity of the surveillance reports was calculated to be 35% (95% CI, 23%–48%), while the specificity was calculated at 87% (95% CI, 82%–92%). The positive likelihood ratio was 2.8, while the negative likelihood ratio was 0.74.
False‐Negative and False‐Positive Central Line–Associated BSIs
The discordant results were analyzed further. Of the 17 false negatives, 9 (53%) were found to be central line–associated BSIs according to criterion 2b, 1 (6%) according to criterion 2a, and 7 (41%) according to criterion 1. Removing the central line–associated BSIs that met criterion 2b from the pool of false negatives leads to a hypothetical sensitivity of 50%.
Of the 19 false positives, 11 cases had another source of sepsis. Two cases did not fulfill clinical criteria for sepsis, and 2 cases had fewer than 2 blood cultures with a possible skin contaminant and no administration of antibiotics. The remainder did not meet the criteria on grounds of timing (eg, blood sample obtained prior to ICU admission).
Discussion
Validated surveillance of hospital‐acquired infections has long been known to reduce the incidence of healthcare‐associated infection,19 and it also enables hospitals to monitor the effectiveness of strategies to reduce infection. Our study revealed poor accuracy and consistency in reporting of central line–associated BSIs in the sampled Victorian public hospitals. The large discrepancies between the gold standard and surveillance classifications make the system inadequate for rate comparison between hospitals and within hospitals following changes in practice.
The definitions used for central line–associated BSI in Victoria, Australia, are similar to those that were used by the US National Healthcare Safety Network. The patient mix and standards of care are also similar. Nevertheless, our results contrast with those of Emori et al, who assessed the accuracy of reporting of infection in ICUs under National Nosocomial Infections Surveillance system guidelines.12 The results in that study, which assessed central line–associated BSI surveillance compared with the gold standard, were positive predictive value of 87%, a sensitivity of 85%, and a specificity of 98.3%. A recent review that questioned the utility of surveillance for central line–associated BSIs in smaller Victorian hospitals revealed that the positive predictive value of reported central line–associated BSIs was only 24% when compared with physician review as the gold standard.20 Reasons for a different experience in Victoria, Australia, and the United States need to be examined.
This study revealed that most false negatives met the definition of central line–associated BSI according to criterion 2b (positive blood culture result, sepsis, and institution of appropriate antibiotics), which suggests underreporting of this criterion. This is consistent with a recent Victorian hospital survey in which ICPs were asked to respond to hypothetical scenarios.21 The survey showed confusion in the application of criterion 2b for central line–associated BSI. In the current study, there were substantial numbers of false negatives due to nonreporting of recognized pathogens that were assessed by the gold standard review as being attributable to a central line (criterion 1 for central line–associated BSI). In this survey, many ICPs attributed such infection as secondary to another source.21 In the current study, false‐positive rates were high, mostly as a result of interpretation of secondary bacteremia as a primary BSI.
If reporting of institutional rates of central line–associated BSI is to become mandatory, the validity of surveillance data must be assessed. Unfortunately, this study has highlighted the lack of agreement between the ICPs and VICNISS reviewers about whether a case of bacteremia is associated with a central line. In particular, the false‐negative rate was high, which could mean that true central line–associated BSI rates are underreported by the current system at the level of approximately 50% overall.
If reporting of central line‐associated bacteremia rates in hospitals is to become mandatory and public, a more objective and reliable tool for reporting is required. In line with the current recommendations from the Centers for Disease Control and Prevention National Healthcare Safety Network's revised definitions for central line–associated BSI,14 the VICNISS has removed criterion 2b from the definition of central line–associated BSI, which could lead to an improvement in surveillance sensitivity from 33% to 50% on the basis of the study results. Further education and training of ICPs on the interpretation of central line–associated BSI definitions is also required. The new case definition will need to be validated to ensure consistency and reliability in application of the definitions of central line–associated BSI.
Acknowledgments
We thank the Quality and Safety Branch, Department of Human Services, Victoria, and Melbourne Health.
Financial support. The VICNISS program is fully funded by the Department of Human Services, Victoria.
Potential conflicts of interest. All authors report no conflicts of interest relevant to this article.
References
- 1. Wenzel RP, Edmond MB. The impact of hospital‐acquired bloodstream infections. Emerg Infect Dis 2001; 7:174–177.
- 2. Rosenthal VD, Guzman S, Migone O, Safdar N. The attributable cost and length of hospital stay because of nosocomial pneumonia in intensive care units in 3 hospitals in Argentina: a prospective, matched analysis. Am J Infect Control 2005; 33:157–161.
- 3. Vandijck DM, Depaemelaere M, Labeau SO, et al. Daily cost of antimicrobial therapy in patients with intensive care unit‐acquired, laboratory‐confirmed bloodstream infection. Int J Antimicrob Agents 2008; 31:161–165.
- 4. Coello R, Charlett A, Ward V, et al. Device‐related sources of bacteraemia in English hospitals—opportunities for the prevention of hospital‐acquired bacteraemia. J Hosp Infect 2003; 53:46–57.
- 5. Berenholtz SM, Pronovost PJ, Lipsett PA, et al. Eliminating catheter‐related bloodstream infections in the intensive care unit. Crit Care Med 2004; 32:2014–2020.
- 6. Pronovost P, Needham D, Berenholtz S, et al. An intervention to decrease catheter‐related bloodstream infections in the ICU. N Engl J Med 2006; 355:2725–2732.
- 7. Emori TG, Culver DH, Horan TC, et al. National nosocomial infections surveillance system (NNIS): description of surveillance methods. Am J Infect Control 1991; 19:19–35.
- 8. Gaynes RP, Culver DH, Emori TG, et al. The National Nosocomial Infections Surveillance system: plans for the 1990s and beyond. Am J Med 1991; 91(Suppl 3B):S116–S120.
- 9. Kritchevsky SB, Braun BI, Kusek L, et al. The impact of hospital practice on central venous catheter–associated bloodstream infection rates at the patient and unit level: a multicenter study. Am J Med Qual 2008; 23:24–38.
- 10. Haley RW, Schaberg DR, McClish DK, et al. The accuracy of retrospective chart review in measuring nosocomial infection rates: results of validation studies in pilot hospitals. Am J Epidemiol 1980; 111:516–533.
- 11. Glenister H, Taylor L, Bartlett C, Cooke M, Sedgwick J, Leigh D. An assessment of selective surveillance methods for detecting hospital‐acquired infection. Am J Med 1991; 91(Suppl 3B):S121–S124.
- 12. Emori TG, Edwards JR, Culver DH, et al. Accuracy of reporting nosocomial infections in intensive‐care‐unit patients to the National Nosocomial Infections Surveillance system: a pilot study. Infect Control Hosp Epidemiol 1998; 19:308–316.
- 13. Gaynes R, Richards C, Edwards J, et al. Feeding back surveillance data to prevent hospital‐acquired infections. Emerg Infect Dis 2001; 7:295–298.
- 14. Horan TC, Andrus M, Dudeck MA. CDC/NHSN surveillance definition of health care–associated infection and criteria for specific types of infections in the acute care setting. Am J Infect Control 2008; 36:309–332.
- 15. Huotari K, Lyytikainen O, Seitsalo S. Patient outcomes after simultaneous bilateral total hip and knee joint replacements. J Hosp Infect 2007; 65:219–225.
- 16. Friedman ND, Russo PL, Bull AL, Richards MJ, Kelly H. Validation of coronary artery bypass graft surgical site infection surveillance data from a statewide surveillance system in Australia. Infect Control Hosp Epidemiol 2007; 28:812–817.
- 17. McBryde ES, Kelly H, Marshall C, Russo PL, McElwain DL, Pettitt AN. Using samples to estimate the sensitivity and specificity of a surveillance process. Infect Control Hosp Epidemiol 2008; 29:559–563.
- 18. Kelly H, Bull A, Russo P, McBryde ES. Estimating sensitivity and specificity from positive predictive value, negative predictive value and prevalence: application to surveillance systems for hospital‐acquired infections. J Hosp Infect 2008; 69:164–168.
- 19. Haley RW, Culver DH, White JW, et al. The efficacy of infection surveillance and control programs in preventing nosocomial infections in US hospitals. Am J Epidemiol 1985; 121:182–205.
- 20. Bennett NJ, Bull AL, Dunt DR, et al. Bloodstream infection surveillance in smaller hospitals. Aust Infect Control 2007; 12:45–47.
- 21. Worth LJ, Brett J, Bull AL, McBryde ES, Russo PL, Richards MJ. Impact of revising the National Nosocomial Infection Surveillance system definition for catheter‐related bloodstream infection in ICU: reproducibility of the National Healthcare Safety Network case definition in an Australian cohort of infection control professionals. Am J Infect Control 2009; 37:643–648.