JSTOR

Methods

Two physicians (I.Z. and C.K.) were trained in diagnosing nosocomial infections according to the Centers for Disease Control and Prevention definitions by means of case examples and by reviewing selected patient medical records, as described elsewhere.⁹ Sample size calculations showed that approximately 300 cases of LRTIs or bloodstream infections and 1,200 medical records of patients without nosocomial infections were needed for the study to achieve the desired precision (defined as half the width of the 95% confidence interval [CI]). These requirements were met by reviewing 15 medical records of patients with reported LRTIs or bloodstream infections and 60 medical records of patients without reports of these infections in 20 KISS ICUs.

The inclusion criterion for the ICUs was participation in KISS for at least half a year before the beginning of the study period in August 2000. All ICUs with a shorter duration of participation were excluded. Therefore, 20 ICUs participating in KISS were randomly chosen from a list of all 77 KISS ICUs that started KISS surveillance in February 1999 or earlier. This list was sorted by patient‐days per year to represent the size and workload of the ICU. For each ICU selected, a sample of medical records was reviewed by the trained physicians; the sample included 15 medical records from patients with bloodstream infections or LRTIs reported successively to KISS and 60 medical records from consecutively admitted patients without reports of these infections.

Diagnoses were made by the trained physicians. In cases of discrepancy with the diagnosis of the surveillance personnel, a supervising epidemiologist from the National Center of Surveillance of Nosocomial Infections was consulted.

After determining the positive predictive value (ie, the number of confirmed nosocomial infections/number of reported nosocomial infections) and the negative predictive value (number of confirmed patients without nosocomial infection reported/number of patients without reported nosocomial infection) of the diagnoses made by surveillance personnel, sensitivity and specificity were calculated according to the following Bayes formulas: sensitivity = (the positive predictive value × the reported incidence) / {(the positive predictive value × the reported incidence) + [(1 − the negative predictive value) × (1 − the reported incidence)]}; specificity = [the negative predictive value × (1 − the reported incidence)] / {[the negative predictive value × (1 − the reported incidence)] + [(1 − the positive predictive value) × the reported incidence)]}. Statistical analyses were performed with SAS statistical software, version 6.12 (SAS Institute). After validation for each ICU, the medical records were discussed between the study physicians and the surveillance personnel, and possible reasons for misdiagnoses were recorded.

Results

Nine surgical ICUs, 8 medical‐surgical ICUs, 2 neurosurgical ICUs, and 1 medical ICU were randomly selected for analysis. The median number of ICU beds was 12, the median number of patients admitted per year was 678, and the median number of patient‐days per year was 3,509. The number of patient‐days did not differ significantly from the median for all KISS ICUs during the same period (ie, through June 2000), which was 3,040 patient‐days. In the retrospective medical record review, a total of 1,481 cases were analyzed, including 286 reported nosocomial infections (51 bloodstream infections and 235 LRTIs) and 1,195 medical records with no reported bloodstream infections or LRTIs. A total of 242 of the reported nosocomial infections were confirmed, and 19 previously undiagnosed nosocomial infections were identified by the trained physicians (Table 1). The pooled mean of the positive predictive value of the surveillance personnel in KISS was 85% (95% CI, 79%‐89%; median, 90%), and the negative predictive value was 98.4% (95% CI, 97.5%‐99.0%; median, 98%). The overall sensitivity for the surveillance personnel of the 20 ICUs was 66% (95% CI, 54%‐77%; median, 81%), and the overall specificity was 99.4% (95% CI, 99.2%‐99.6%; median, 99.6%).

Table 1.  Data on Nosocomial Infections (NIs) From the Validation Study Data and Device‐Associated Infection Rates from the Krankenhaus Infektions Surveillance System (KISS) in German Intensive Care Units

Open New Window

Discussion

Surveillance systems that provide reference data depend on the quality of the data reported by the single contributor. An important reason for reporting incomplete data is that participants may fear the consequences of detecting high infection rates. Thus, voluntary and anonymous participation is an essential factor for achieving valid information. However, the accuracy of reporting should be assessed periodically to promote data reliability and to find out whether there are systematic problems concerning the application of the surveillance method. Hence, 3 years after the implementation of KISS, the validity of the ICU component’s data was evaluated.

Prospective evaluation was conducted in several studies,^10‐12 but this approach, as well as the prevalence survey evaluation,^13,14 only allows assessment of a relatively small amount of data. Furthermore, in the prospective evaluation, ongoing surveillance might be influenced by the known presence of an observer. As done by several other investigators,^15,16 we chose a method of retrospective medical record review to assess the accuracy of nosocomial infections surveillance data. Although the retrospective approach has certain limitations, such as loss of undocumented information, this method seems to be appropriate for the evaluation of the identification of nosocomial infections^17,18; in one investigation, retrospective medical record review was found to be even more sensitive than the prospective approach.¹⁹ In our investigation, we found that discrepancies in diagnosing usually appeared because of uncertainty of the surveillance personnel in diagnosing nosocomial infections (eg, reporting of 2 cases of pneumonia in the same patient because of changes of the microorganism found) or because of surveillance gaps (eg, during personnel vacation) and not because of different approaches regarding the interpretation of the patient medical records because of missing information.

The results for the particular types of nosocomial infection found in our study are similar to those of the NNIS evaluation study¹⁵ (Table 2). However, in the NNIS study, sensitivity and specificity were calculated for the medical record sample selected for the study, whereas in our investigation we estimated sensitivity and specificity for the whole system, taking into account the incidence of nosocomial infections by applying the Bayes formulas; therefore, the comparability of the 2 investigations is limited. Furthermore, validation studies for surveillance of nosocomial infections in ICUs were performed in Belgium and France, with results comparable to our findings.²⁰ The ratio of the true incidence to the reported incidence of the different infection types can be regarded as a corrective factor for estimating the true infection rates for pneumonia, bronchitis, and bloodstream infections in German intensive care units (Table 1).

Table 2.  Comparison of the Results of the National Nosocomial Infections Surveillance Systems (NNIS) Validation Study With Results of an Evaluation of the Krankenhaus Infektions Surveillance System (KISS)

Open New Window

KISS started in 1997 with 25 ICUs participating in the ICU component. In December 2004, a total of 323 ICUs were registered. Ongoing validation is important for the credibility and acceptance of the surveillance system, and the KISS participants are clearly aware of this fact. The observers were well received and supported by the surveillance personnel in all selected units. Finding the “bad apples” that do not provide correct surveillance data is crucial. However, validation studies are time‐consuming and costly. Therefore, for continuous validation other methods have to be found to ensure data quality. In KISS, plausibility checks are implemented at the data entry level. Additionally, surveillance personnel in units with low infection or device utilization rates or other conspicuous data are contacted periodically and asked to explain their rates. If they fail to give a plausible explanation, the data of the respective unit are eliminated from the reference database. Nevertheless, validation by visiting the participating units is indispensable for detecting systematic problems with regard to surveillance and for drawing conclusions for improving the training for participants.

Conclusion

One of the most important aspects of surveillance systems is to provide quality management data for its participants. KISS data are appropriate for the internal quality management of the participating ICUs. However, if the data are used for assessing the situation in ICUs in Germany (eg, for cost analyses), a corrective factor is needed. This study reveals the importance of identifying invalid data from the reference database (eg, by regular validation queries searching the electronic database), and it shows the necessity of continuous training and maintaining contact with surveillance personnel.