Quality of Data Reported to a Smaller‐Hospital Pilot Surveillance Program
This data quality study assessed the accuracy of data collected as part of a pilot smaller‐hospital surveillance program for methicillin‐resistant Staphylococcus aureus (MRSA) infection and bloodstream infection (BSI). For reported MRSA infection, estimated values were as follows: sensitivity, 40%; specificity, 99.9%; and positive predictive value, 33.3%. For reported BSI, estimated values were as follows: sensitivity, 42.9%; specificity, 99.8%; and positive predictive value, 37.5%.
Received September 4, 2005; accepted February 23, 2006; electronically published February 26, 2007.
The purpose of the Victorian Hospital Acquired Infection Control Surveillance System (VICNISS) Coordinating Centre is to develop, implement, support, and evaluate standardized surveillance of hospital‐acquired infection (HAI) in public acute care hospitals in Victoria, Australia. There are currently 28 larger hospitals (with more than 100 beds) and 89 smaller hospitals (with 100 beds or fewer) participating. In late 2003, a novel surveillance program for the smaller hospitals was simultaneously evaluated in a pilot study for 18 weeks in 14 eligible hospitals. These pilot hospitals were considered representative of the other smaller hospitals statewide. The number of patients discharged during the pilot program ranged from 25 to 751 (mean, 243) per hospital.
Comparisons of HAI rates over time, or among health care institutions, may be misleading if data are collected inaccurately.1,2 For this reason, a data quality study was initiated 3 months after completion of the smaller‐hospital pilot surveillance program. The primary aim of this study was to assess the accuracy of data collected as part of the pilot surveillance modules on methicillin‐resistant Staphylococcus aureus (MRSA) infection and bloodstream infection (BSI). (Trained infection control personnel collected data for these modules using published MRSA infection definitions3 and primary laboratory‐confirmed BSI definitions.4) To our knowledge, this is the first published study that specifically addresses this aim in smaller hospitals.
Methods
The pilot hospitals and their patients were subcategorized into 2 sample groups. Time constraints did not allow data to be collected for the data quality study on all patients discharged during the pilot program. Sample group 1 included all acute‐care patients who were discharged during the pilot program from a hospital with 14 or fewer acute‐care beds. Sample group 2 included all acute‐care patients who were discharged during the pilot program from a hospital with 15‐100 acute‐care beds and who were directly transferred from other healthcare facilities and/or hospitalized for 48 hours or more. Patients in sample group 2 hospitals who did not meet these criteria were considered to be at low risk for acquiring MRSA infection.3
Data collection was undertaken retrospectively in 2 stages. During stage 1, the VICNISS infection control consultant reviewed electronic microbiology reports for all patients in both sample groups. Medical records were checked for those patients who had a microbiology report indicating the presence of MRSA or had a blood culture positive for a pathogen during the pilot program. Infections recorded during stage 1 that matched infections reported during the pilot study were deemed to be true infections.
During stage 2, the VICNISS epidemiologist reexamined the medical records in which there was a discrepancy between the infection reported during the pilot program and the infection recorded during stage 1. These “discrepant infections” were either excluded or confirmed to be true infections. True infections from both stage 1 and 2 were used to estimate the accuracy of the reports of infections during the pilot program by calculating the sensitivity, specificity, and positive predictive value.
Results
Twelve of the 14 pilot program hospitals agreed to participate in the data quality study. This group included 6 hospitals in each of the 2 sample groups. (The 2 pilot hospitals that did not consent to participate in the study would have been categorized into sample group 1). During stage 1, electronic microbiology reports for 2,921 patients who were discharged during the pilot program were examined.
Five true MRSA infections and 7 true BSIs were recorded as part of the data quality study. During stage 1, the VICNISS infection control consultant retrospectively matched 2 of the 5 MRSA infections reported during the pilot study, but also detected 3 MRSA infections that were not reported. Three of the 8 BSIs reported during the pilot study were matched, but 4 BSIs were detected that were not reported. These stage 1 results were consistent with those recorded by the VICNISS epidemiologist during stage 2. Reasons for excluding some of the infections reported during the pilot study are shown in Table 1. The sensitivity, specificity, and positive predictive value for the MRSA infections and BSIs reported during the pilot study are shown in Table 2.
Discussion
It is difficult to define acceptable sensitivity, specificity, and positive predictive value estimates for a smaller‐hospital surveillance program. Surveillance programs specifically developed for smaller hospitals have not been described or evaluated extensively in the literature. We believe, however, that the quality of data collected as part of the VICNISS smaller hospital surveillance program needs to be improved.
In larger hospitals, 80% sensitivity and 97% specificity estimates have been reported as the minimal required levels.5,6 Sensitivity estimates reported for different HAI surveillance programs have, however, varied widely (from 14% to 100%).7‐9 Specificity estimates tend to be high (greater than 92%),6,7,10‐14 although one study did report an estimate of 48%.15 Positive predictive value estimates, not commonly reported, have ranged from 72% to 92%.7,14 In one major study,7 estimated values for BSI reporting across 9 intensive care units were as follows: sensitivity, 85%; specificity, 98.3%; and positive predictive value, 87%. In another study,5 in which the infection control personnel used varied definitions, the score for identifying BSIs was 78%.
The sensitivity estimates indicated that the pilot hospitals' infection control personnel failed to report more than half of the MRSA infections and BSIs. The specificity estimates suggested that reports for patients who did not acquire these infections (ie, the vast majority of patients) were correct. The positive predictive value estimates for reported infections indicated that if the pilot hospital infection control personnel reported an infection, it was unlikely to be a true infection.
To promote accuracy in data collection (and hence, in infection rate analysis), it is recommended in the literature7,16‐18 that the following practices be implemented as part of a surveillance program: the use of standardized, nonambiguous definitions (including numerator and denominator); the use of sensitive case finding methods; and the consistent application of these criteria by trained and experienced personnel. To understand the reasons for our unexpected results, these practices should be reviewed in relation to the VICNISS smaller hospital surveillance program.
Some limitations of the data quality study need to be highlighted. First, for sample group 2 there was a potential for selection bias. The length of time in hospital would most likely affect the number of true MRSA infections that might be counted.3 Second, the results are only considered applicable for the pilot surveillance program in the 12 hospitals that participated in the data quality study. Assessment of the surveillance program during a different time period, or in different hospitals, may have produced different results. Lastly, the VICNISS employees retrospectively reviewed microbiology databases and medical records to collect data for the data quality study. Although this is considered a valid approach,10 the study results depended on the completeness and accuracy of these data sources.
The data quality study emphasized the importance of assessing the accuracy of data collected in novel surveillance programs. An objective assessment of our smaller hospital pilot surveillance program highlighted the fact that both sensitivity and positive predictive value estimates need to be improved. Changes to the program will be required after a detailed review.
References
- 1. Centers for Disease Control and Prevention National Nosocomial Infection Surveillance System. Nosocomial infection rates for inter‐hospital comparison: limitations and possible solutions. Infect Control Hosp Epidemiol 1991; 12:609‐621.
- 2. Mulholland SG, Creed J, Dierauf LA, Brumn JN, Blackmore WS. Analysis and significance of nosocomial infection rates. Ann Surg 1974; 180:827‐830.
- 3. Colignon P, Looke D, Ferguson J, McLaws M‐L, Olsen D. Surveillance definitions for multi‐resistant organisms (MROs). Australian Infect Control J 2002; 7.
- 4. Gaynes RP, Horan TC. Surveillance of nosocomial infections. In: Mayhall G, ed. Hospital Epidemiology and Infection Control. Baltimore: Williams & Wilkins, 1996:1285‐1317.
- 5. Larson E, Horan T, Cooper B, Kotilainen HR, Landry S, Terry B. Study of the definition of nosocomial infections. Am J Infect Control 1991; 19:259‐267.
- 6. Ehrenkranz NJ, Schulz LM, Richter EI. Recorded criteria as a gold standard for sensitivity and specificity estimates of surveillance of nosocomial infection: a novel method to measure job performance. Infect Control Hosp Epidemiol 1995; 16:697‐702.
- 7. Emori TG, Edwards JR, Culver DH, et al. Accuracy of reporting nosocomial infections in intensive care unit patients to the National Nosocomial Infections Surveillance system: a pilot study. Infect Control Hosp Epidemiol 1998; 19:308‐316.
- 8. Perl TM. Surveillance, reporting and the use of computers. In: Wenzel RP, ed. Prevention and Control of Nosocomial Infections. 3rd ed. Baltimore: Williams & Wilkins, 1997:25‐161.
- 9. Gastmeier P, Brauer H, Hauer T, Schumacher M, Daschner F, Ruden H. How many nosocomial infections are missed if identification is restricted to patients with either microbiology reports or antibiotic administration? Infect Control Hosp Epidemiol 1999; 20:124‐127.
- 10. Haley RW, Schaberg DR, McClish DK, et al. The accuracy of retrospective chart review in measuring nosocomial infection rates: results of validation studies in pilot hospitals. Am J Epidemiol 1980; 111:516‐533.
- 11. Glenister HM, Taylor LJ, Bartlett CLR, Cooke EM, Sedgwick JA, Mackintosh CA. An evaluation of surveillance methods for detecting infections in hospital inpatients. J Hosp Infect 1993; 23:229‐242.
- 12. Gastmeier P, Kampf G, Hauer T, et al. Experience with two validation methods in a prevalence survey on nosocomial infections. Infect Control Hosp Epidemiol 1998; 19:668‐673.
- 13. Cardo DM, Falk PS, Mayhall CG. Validation of surgical wound surveillance. Infect Control Hosp Epidemiol 1993; 14:211‐215.
- 14. Broderick A, Mori M, Nettleman MD , Streed SA, Wenzel RP. Nosocomial infections: validation of surveillance and computer modelling to identify patients at risk. Am J Epidemiol 1990; 131:734‐742.
- 15. Laxson LB, Blaser MJ, Parkhurst SM. Surveillance for the detection of nosocomial infections and the potential for nosocomial outbreaks. 1. Microbiology culture surveillance is an effective method of detecting nosocomial infection. Am J Infect Control 1984; 12:318‐324.
- 16. The Quality Indicator Study Group. An approach to the evaluation of quality indicators of the outcome of care in hospitalized patients, with a focus on nosocomial infection indicators. Am J Infect Control 1995; 23:215‐222.
- 17. Larson E, Oram LF, Hedrick E. Nosocomial infection rates as an indicator of quality. Med Care 1988; 26:676‐685.
- 18. Freeman J, McGowan JE Jr. Methodological issues in hospital epidemiology. I. Rates, case finding and interpretations. Rev Infect Dis 1981; 3:658‐667.