Improving Risk-Adjusted Measures of Surgical Site Infection for the National Healthcare Safety Network
Background. The National Healthcare Safety Network (NHSN) has provided simple risk adjustment of surgical site infection (SSI) rates to participating hospitals to facilitate quality improvement activities; improved risk models were developed and evaluated.
Methods. Data reported to the NHSN for all operative procedures performed from January 1, 2006, through December 31, 2008, were analyzed. Only SSIs related to the primary incision site were included. A common set of patient- and hospital-specific variables were evaluated as potential SSI risk factors by univariate analysis. Some ific variables were available for inclusion. Stepwise logistic regression was used to develop the specific risk models by procedure category. Bootstrap resampling was used to validate the models, and the c-index was used to compare the predictive power of new procedure-specific risk models with that of the models with the NHSN risk index as the only variable (NHSN risk index model).
Results. From January 1, 2006, through December 31, 2008, 847 hospitals in 43 states reported a total of 849,659 procedures and 16,147 primary incisional SSIs (risk, 1.90%) among 39 operative procedure categories. Overall, the median c-index of the new procedure-specific risk was greater (0.67 [range, 0.59–0.85]) than the median c-index of the NHSN risk index models (0.60 [range, 0.51–0.77]); for 33 of 39 procedures, the new procedure-specific models yielded a higher c-index than did the NHSN risk index models.
Conclusions. A set of new risk models developed using existing data elements collected through the NHSN improves predictive performance, compared with the traditional NHSN risk index stratification.
Surgical site infection (SSI) is one of the most common healthcare-associated infections (HAIs) and is a major cause of increased length of hospital stay and mortality.1-3 SSI surveillance is integral to hospital infection control and quality improvement programs, with feedback of SSI rates being an important component of SSI reduction strategies.4,5 However, hospitals with surgeons who treat patients with multiple nonmodifiable risk factors would expect higher SSI rates. Therefore, risk adjustment that accounts for differences in patient case mix is critical to allow for more meaningful comparisons between surgeons or between hospitals, especially when using SSI summary data as a quality improvement performance metric.6,7
Controversies exist regarding several aspects of such risk adjustment. One is the inclusion of intraoperative or postoperative variables in any risk adjustment strategy, because these variables may reflect surgical technique more than patient case mix, and adjustment for surgical technique may inappropriately allow for adjusting rates down among surgeons with poor technique. Another is the inclusion of SSIs detected through SSI surveillance after discharge from the hospital, which is a setting with great variation in case-finding intensity. In addition, including more procedure-specific variables to generate improved procedure-specific models adds to the data collection burden.
These controversies are relevant to the National Healthcare Safety Network (NHSN), a secure Web-based system used by the Centers for Disease Control and Prevention (CDC) and its healthcare and public health partners for surveillance of HAIs, other adverse events in health care, and adherence to prevention practices in hospitals and other reporting facilities. Traditionally, SSI rates calculated by the CDC and other NHSN data users from data reported to the NHSN have been risk stratified using a risk index of 3 equally weighted factors: the American Society of Anesthesiologists (ASA) score, wound classification, and procedure duration.8,9 However, for some procedures, these variables are not associated with SSI risk, are not equally important in the risk they confer, and are candidates for replacement by other, more important risk factor variables that should be taken into account. Second, beginning in 2012, hospitals participating in the Center for Medicare and Medicaid Services (CMS) Inpatient Prospective Payment System (IPPS) will be required to report SSI data through NHSN, and these data will be included in the Inpatient Quality Reporting data that are publicly reported by CMS at the Hospital Compare Web site.10 Publicly reported SSI data should account for variability in patient case mix, adjust for all possible risk factors, and be based on consistent case detection systems.7,11 Procedure-specific, multivariate risk models that incorporate additional weighted patient factors could calculate more credible, standardized, and reliable risk-adjusted SSI metrics than stratified SSI rates that are limited to the traditional NHSN risk index.12-15
The objectives of our evaluation were to develop procedure-specific risk models for each of the procedure categories reported to the NHSN, incorporating existing NHSN data elements, and to compare their predictive performance with procedure category–specific models composed of only the variable of the traditional NHSN risk index. A secondary objective was to utilize similar methodology to develop models for proposed public reporting metrics (ie, using only deep incisional and organ/space SSIs detected during hospitalization or rehospitalization at the same hospital).
The resulting procedure-specific risk models can be used as a reference of how risk adjustment is currently performed in the NHSN application, and this article will essentially replace the historical annual report containing risk stratification tables.16
Study Population, Endpoints, and Statistical Approach
As of September 2010, more than 1,900 hospitals reported SSI data to the NHSN. Reporting has been predominately voluntary and confidential; however, during 2008–2009, several states enacted laws mandating SSI reporting to the NHSN for specific procedures at hospitals in their jurisdiction.17 The methodology of SSI surveillance has been described elsewhere.18 In brief, infection preventionists (IPs) choose a procedure category to follow for an entire month and report data on all patients undergoing all procedures within the procedure category for each month of surveillance performed. IPs also are required to identify and report all SSIs detected during the initial hospitalization, through surveillance after hospital discharge, or upon rehospitalization at the same hospital at which the initial procedure was performed. SSIs are classified using standard definitions as superficial incisional, deep incisional (involving the fascia or muscle), or organ/space. SSIs reported to the NHSN are limited to those detected within 30 days after the initial procedure (superficial incisional) or up to 1 year for deep incisional and organ/space if the procedure included an implant (eg, sternal wires or prosthesis).18
SSI data were analyzed for all reported procedures performed from January 1, 2006, through December 31, 2008, including data for all 40 NHSN procedure categories. For this analysis, the NHSN procedure code CBGB (with both sternal and harvest site incisions) and CBGC (with sternal site incision only) were grouped into a single procedure category, CABG, for a total of 39 procedures. In addition, only primary incisional SSIs were analyzed, because no patient- or procedure-specific variables were collected for secondary incision sites; therefore, any SSIs related to secondary incision sites for the NHSN codes CBGB, FUSN, and RFUSN were excluded. All SSIs (superficial incisional, deep incisional, and organ/space) detected through all methods of surveillance (hospitalization, rehospitalization, and surveillance after hospital discharge) for both inpatient and outpatient surgical procedures were included.
Procedures containing outlier values were excluded according to rules described in Appendix A. As a result, a total of 6,432 (0.75%) procedures were excluded from the analysis; the final number of procedures eligible for further analysis was 849,659.
First, patient and hospital characteristic data were evaluated. Second, NHSN risk index models were created for all 39 NHSN procedures. Third, new procedure-specific predictive risk models were created for the same set of procedures through an interactive process that included univariate analysis of all available patient- and hospital-level variables, multivariate modeling, and model validation. SAS, version 9.2 (SAS Institute), was used for data analysis. After completion of the primary analysis, endpoints were altered to include only complex (deep incisional and organ/space) SSIs detected at hospitalization and rehospitalization to develop models appropriate for public reporting, consistent with the 2008 National Quality Forum (NQF) recommendation to exclude superficial SSIs and those detected through surveillance after discharge from the hospital.19
NHSN Risk Index Model
The NHSN risk index comprises 3 dichotomous variables: ASA score (3, 4, or 5), wound classification (contaminated or dirty), and procedure duration in minutes (>75th percentile). Each risk factor represents 1 point; thus, the NHSN SSI risk index ranges from 0 (lowest risk) to 3 (greatest risk).8 Logistic regression of SSIs against the NHSN risk index was used to build the NHSN risk index models by procedure category.
New Procedure-Specific Risk Model
The new model incorporates the 3 NHSN risk index variables and additional data elements currently collected in the NHSN. These are variables of convenience in that they are routinely reported to the NHSN as part of the existing SSI surveillance methodology. Variables were dichotomous (general anesthesia, emergency procedure, gender, trauma association, and medical school affiliation), ordinal (ASA score), categorical (wound classification and number of hospital beds), or continuous (age and procedure duration; Table 1).
|All||Gender, age, emergency, trauma, general anesthesia, ASA score, wound classification, duration, medical school affiliation, no. of hospital beds, endoscope, outpatient|
|HPRO||Type of surgery (total primary, partial primary, partial revision, total revision)|
|KPRO||Type of surgery (revision, primary)|
|CSEC||Labor, blood loss, body mass index|
|FUSN/RFUSN||Approach, spinal level, diabetes|
Procedure-specific supplemental variables include primary versus revision arthroplasty for HPRO and KPRO; total or partial hip arthroplasty for HPRO; body mass index (BMI), history of labor, and estimated blood loss for CSEC; and diagnosis of diabetes, spinal level, and surgical approach for FUSN and RFUSN (Table 1).
Among the variables common to all 849,659 procedures, 7 variables had missing values in 1,304 (0.15%) of the procedures. Variables with missing values were medical school affiliation (931 [0.11%]), trauma (219 [0.03%]), general anesthesia (89 [0.01%]), ASA score (23 [<0.01%]), endoscope (20 [<0.01%]), wound classification (12 [<0.01%]), and emergency (10 [<0.01%]). Among procedure-specific variables, missing values included the following: for CSEC, BMI (242 [0.78%]) and history of labor (5 [0.02%]); for HPRO, type of surgery (16 [0.01%]); for KPRO, type of surgery (15 [0.01%]); and for spine procedures, diabetes (157 [0.37%]), spinal level (3 [0.01%]), and surgical approach (3 [0.01%]; Table 2).
|Procedure code, characteristic||No. (%) of procedures|
|HPRO, type of surgery|
|Total primary||99,046 (75.09)|
|Partial primary||19,658 (14.90)|
|Total revision||10,518 (7.97)|
|Partial revision||2,661 (2.02)|
|KPRO, type of surgery|
|CSEC, blood loss|
|≤400 mL||2,310 (7.48)|
|401–800 mL||23,854 (77.22)|
|>800 mL||4,725 (15.30)|
|Anterior and posterior||1,229 (2.90)|
|Lateral transverse||1,004 (2.37)|
|Not specifieda||6,623 (15.65)|
|FUSN/RFUSN, spinal level|
|Not specifieda||5,777 (13.65)|
The χ2 test was used to test for each individual variable’s association with SSI. Ordinal variables were collapsed into a single group if the χ2 test showed no significant difference between them. For categorical variables, multiple categorizations were used, and only the category most significantly associated with SSI risk was presented as the result of univariate analysis. Continuous variables were divided into quartiles and were compared by means of the χ2 test; continuous variables were coded as binary variables if a significant cutoff point was found. Otherwise, the continuous variable “duration” was coded as “duration10” for every 10-minute increase in duration, and “age” was coded as “age10” for every 10-year increase in age. Variables from the univariate analysis with were considered potential independent variables and entered into the logistic regression model as candidate variables for inclusion.
Stepwise logistic regression was used to develop the model. For all regression analyses, the referent category was the one that conferred the least risk of SSI. Variables were eligible for inclusion if the likelihood ratio test (LRT) and removed at LRT significance. For variables with multiple categorical, ordinal, or dichotomous cutoff values, the one with the smallest LRT P value was included.
Final Model Variable Selection Procedure
To confirm the appropriateness of the final models, we performed the same stepwise model selection with all variables included regardless of their significance levels in univariate analysis. The interaction terms were tested and kept at LRT significance.
Training and Validation Samples
The models were validated using a bootstrap sample following the steps described in Appendix B.
The predictive performances of the new and existing NHSN risk index models were assessed by constructing receiver-operating characteristic (ROC) curves and calculating the c-index for the separate logistic regression models. An ROC curve is constructed by plotting the sensitivity (y-axis) versus 1 minus specificity (x-axis) over the range of scores for a given index. The area under the ROC curve (AUC) is the c-index. The c-index is a measure of predictive performance and represents the proportion of instances in which a patient who acquires an SSI is assigned a higher probability of SSI than a patient who does not acquire an SSI. Values for the c-index range from 0.5 (null) to 1.0 (perfect predictive ability).20 The difference in c-index was tested using the method described by Hanley and McNeil.21
Prediction Models for Possible Public Reporting
To be consistent with proposed measures submitted to the NQF regarding public reporting of SSI, we also evaluated the performance characteristics of procedure-specific models for the subset of SSIs classified as deep or organ/space and detected only during the hospitalization during which the surgical procedure was performed or upon rehospitalization at the same facility. To perform this task, we repeated all of the methodologies described for all incisional SSIs for the subset of SSIs classified as complex (deep incisional or organ/space) detected during hospitalization or after rehospitalization at the same hospital. These models are referred to as predictive of complex SSI for public reporting.
From January 1, 2006, through December 31, 2008, 847 hospitals reported to the NHSN a total of 849,659 procedures and 16,147 SSIs at the primary incision site. The overall risk of SSI was 1.90 per 100 procedures, ranging from 0.26 (THYR) to 13.83 (LTP). The variability in patient and hospital characteristics for some of the main procedure-specific variables is summarized in Table 2.
A list of the significant variables for each of the 39 procedures is summarized in Table 3. As an example, univariate analysis results are shown for hip prostheses (HPRO), for which there were 10 potential independent variables identified for inclusion in the multivariate modeling (Table 4).
|Procedure code||Description||List of variables|
|AAA||Abdominal aortic aneurysm||Emergency, wound class, ASA score, duration|
|AMP||Limb amputation||Bed size, duration|
|APPY||Appendectomy||Emergency, endoscope, gender, ASA score, wound class|
|AVSD||Arteriovenous shunt for dialysis||Age, duration|
|BILI||Bile duct, liver or pancreatic surgery||Emergency, endoscope, ASA score, wound class, bed size, duration|
|BRST||Breast surgery||ASA score, bed size, duration|
|CABG||Coronary artery bypass graft||Anesthesia, gender, medical school affiliation, ASA score, bed size, age, duration|
|CARD||Cardiac surgery||ASA score, wound class, age, duration|
|CHOL||Cholecystectomy||Emergency, endoscope, ASA score, wound class, age, duration|
|COLO||Colon surgery||Anesthesia, endoscope, gender, ASA score, wound class, bed size, age, duration|
|CRAN||Craniotomy||Trauma, bed size, age, duration|
|CSEC||Cesarean delivery||Body mass index, age, anesthesia, ASA, duration, labor, bed size, wound class, emergency|
|FUSN||Spinal fusion||Anesthesia, gender, medical school affiliation, trauma, wound class, diabetes, approach, spinal level, duration, ASA score|
|FX||Open reduction of long bone fracture||ASA score, age, duration, outpatient|
|GAST||Gastric surgery||Emergency, endoscope, gender, ASA score, wound class, age, duration|
|HER||Herniorrhaphy||Anesthesia, emergency, endoscope, gender, medical school affiliation, trauma, ASA score, wound class, duration, outpatient|
|HPRO||Hip arthroplasty||Anesthesia, emergency, gender, trauma, ASA score, wound class, bed size, age, duration, total/primary/partial/revision|
|HYST||Abdominal hysterectomy||Anesthesia, endoscope, ASA score, wound class, duration|
|KPRO||Knee arthroplasty||Anesthesia, gender, trauma, ASA score, wound class, age, duration, primary/revision|
|KTP||Kidney transplant||Bed size, age, duration|
|LAM||Laminectomy||Anesthesia, endoscope, gender, ASA score, age, duration|
|LTP||Liver transplant||Anesthesia, emergency, trauma, age, duration|
|NECK||Neck surgery||Wound class, duration|
|OVRY||Ovarian surgery||ASA score, wound class|
|PRST||Prostate surgery||ASA score|
|PVBY||Peripheral vascular bypass surgery||Gender, ASA score, age, duration|
|REC||Rectal surgery||Endoscope, gender, trauma, wound class, bed size, duration|
|RFUSN||Refusion of spine||Trauma, duration, diabetes, spinal level, approach|
|SB||Small-bowel surgery||Bed size, duration|
|THYR||Thyroid and/or parathyroid surgery||Age|
|VHYS||Vaginal hysterectomy||Medical school affiliation, bed size, age, duration|
|VSHN||Ventricular shunt||Emergency, wound class, bed size, age|
|XLAP||Exploratory abdominal surgery||Bed size, duration|
|Variable, class||No. of procedures||No. of SSIs||Risk||P|
|Type of surgeryb||<.0001|
|Medical school affiliation||.1784|
Procedure-Specific Risk Prediction Model
Table 5 shows the results for models of all SSIs identified at primary incision sites for the 39 procedure categories. Multivariate modeling strategies defined new procedure-specific models for each of the 39 procedure categories. The 3 most common variables included were procedure duration, ASA score, and age (30, 21, and 20 models, respectively). Other common variables were the number of hospital beds (16 models), wound class (8), general anesthesia (6), endoscope (5), medical school affiliation (5), emergency (4), and trauma (4). All procedure-specific supplemental variables, except estimated blood loss, were selected for inclusion into the final model. No variables were selected at the level for 4 procedures (ie, intercept-only models): carotid endarterectomy (CEA), heart transplant (HTP), pacemaker placement (PACE), and splenectomy (SPLE). The observed number of SSIs for these 4 procedures during the study period was small, ranging from 6 to 15 (Table 5).
|Procedure code||No. of procedures||No. of SSIs||Effect||Estimate||OR (95% CI)||P||PSM||RIM||Pr>|t||
|Duration, >82 vs ≤82||1.09||2.97 (1.43–6.18)||.0036|
|Bed size, >200 vs ≤200||3.04||20.96 (2.83–154.99)||.0029|
|Emergency, Y vs N||0.61||1.84 (1.14–2.99)||.0135|
|Gender, M vs F||0.53||1.70 (1.07–2.68)||.024|
|Bed size, >500 vs ≤500||0.77||2.15 (1.38–3.34)||.0007|
|Wound class, CO vs C/CC||0.63||1.89 (1.07–3.33)||.0294|
|Wound class, D vs C/CC||1.26||3.53 (2.04–6.09)||<.0001|
|ASA, ≤3 vs >3||1.02||2.76 (1.29–5.89)||.0087|
|Bed size, 201–500 vs ≤200/>500||0.93||2.54 (1.51–4.29)||.0005|
|ASA, >2 vs ≤2||0.92||2.50 (1.57–4.00)||.0001|
|Bed size, ≤200/>500 vs 201–500||0.93||2.54 (1.29–4.98)||.0068|
|ASA (1/2, 3, 4/5)||0.28||1.33 (1.22–1.44)||<.0001|
|Gender, F vs M||2.16||<.0001|
|Bed size, ≤200/>500 vs 201–500||0.15||1.16 (1.08–1.26)||.0001|
|ASA, >3 vs ≤3||0.49||1.63 (1.28–2.07)||<.0001|
|Age, >52 vs ≤52||0.44||1.55 (1.05–2.29)||.0272|
|ASA (1, 2, 3/4/5)||0.60||1.82 (1.30–2.56)||.0005|
|Endoscope, N vs Y||0.43||1.54 (1.07–2.20)||.0191|
|Wound class, CO/D vs C/CC||0.68||1.97 (1.18–3.27)||.0093|
|Anesthesia, Y vs N||0.38||1.47 (1.02–2.12)||.0405|
|ASA, >2 vs ≤2||0.30||1.35 (1.26–1.46)||<.0001|
|Endoscope, N vs Y||0.13||1.14 (1.04–1.25)||.0063|
|Medical school affiliation, N vs Y||0.14||1.15 (1.06–1.25)||.0008|
|Bed size, >500 vs ≤500||0.26||1.30 (1.19–1.41)||<.0001|
|Wound class, CO/D vs C/CC||0.09||1.10 (1.01–1.19)||.0369|
|ASA, >2 vs ≤2||0.32||1.38 (1.04–1.82)||.0243|
|Bed size, >500 vs ≤500||0.45||1.57 (1.18–2.09)||.0022|
|Trauma, Y vs N||0.54||1.72 (1.12–2.65)||.0141|
|Age, ≤26 vs >26||0.27||1.31 (1.11–1.55)||.0017|
|Anesthesia, Y vs N||0.42||1.52 (1.15–2.00)||.0032|
|ASA (1, 2, 3/4/5)||0.28||1.32 (1.10–1.59)||.0026|
|Emergency, Y vs N||0.21||1.23 (1.03–1.47)||.0214|
|Labor, Y vs N||0.41||1.51 (1.27–1.80)||<.0001|
|Wound class, CO/D vs C/CC||0.74||2.09 (1.39–3.15)||.0004|
|Approach, B/L/P vs A||0.93||2.52 (1.96–3.25)||<.0001|
|ASA (1/2, 3, 4/5)||0.61||1.83 (1.60–2.10)||<.0001|
|Diabetes, Y vs N||0.42||1.52 (1.23–1.87)||.0001|
|Medical school affiliation, Y vs N||0.32||1.38 (1.14–1.68)||.0011|
|Spinal level, CD or DL vs AX, AC, or CV||0.67||1.96 (1.41–2.72)||<.0001|
|Spinal level, LL vs AX, AC, or CV||0.52||1.68 (1.32–2.14)||<.0001|
|Wound class, CO/D vs C/CC||0.84||2.31 (1.06–5.03)||.035|
|Trauma, Y vs N||0.60||1.83 (1.23–2.71)||.0026|
|Age, >25 vs ≤25||0.72||2.05 (1.29–3.28)||.0026|
|ASA (1, 2, 3/4/5)||0.29||1.34 (1.07–1.68)||.0119|
|Duration, >138 vs ≤138||0.77||2.16 (1.58–2.95)||<.0001|
|Bed size, 201–500 vs ≤200/>500||0.37||1.45 (1.07–1.95)||.0153|
|Outpatient, N vs Y||1.51||4.52 (1.11–18.36)||.0349|
|ASA, >2 vs ≤2||0.47||1.60 (1.06–2.40)||.0245|
|Emergency, Y vs N||0.64||1.90 (1.19–3.04)||.0074|
|Age, ≤71 vs >71||0.74||2.09 (1.42–3.07)||.0002|
|ASA (1, 2, 3/4/5)||0.76||2.15 (1.68–2.74)||<.0001|
|Gender, F vs M||0.83||2.30 (1.73–3.04)||<.0001|
|Outpatient, N vs Y||0.59||1.80 (1.28–2.53)||.0008|
|Anesthesia, Y vs N||0.11||1.12 (1.01–1.25)||.0383|
|ASA, 3 vs 1/2||0.80||2.23 (2.01–2.49)||<.0001|
|ASA, 4/5 vs 1/2||1.07||2.91 (2.45–3.46)||<.0001|
|Type of surgerya||0.26||1.29 (1.22–1.38)||<.0001|
|Bed size, >500 vs ≤500||0.19||1.21 (1.09–1.34)||.0004|
|Trauma, Y vs N||0.36||1.43 (1.24–1.65)||<.0001|
|Anesthesia, Y vs N||0.68||1.97 (1.26–3.07)||.003|
|ASA (1, 2, 3/4/5)||0.86||2.37 (2.10–2.67)||<.0001|
|Endoscope, N vs Y||0.35||1.43 (1.17–1.74)||.0005|
|Bed size, ≤500 vs >500||0.22||1.25 (1.06–1.47)||.0065|
|Age, ≤58 vs >58||0.30||1.34 (1.21–1.49)||<.0001|
|Anesthesia, Y vs N||0.11||1.12 (1.01–1.24)||.0383|
|ASA (1/2, 3, 4/5)||0.48||1.62 (1.49–1.76)||<.0001|
|Gender, M vs F||0.20||1.22 (1.11–1.34)||<.0001|
|Revision vs primary||0.63||1.89 (1.64–2.17)||<.0001|
|Bed size, >200 vs ≤200||0.11||1.12 (1.01–1.25)||.039|
|Trauma, Y vs N||0.69||1.99 (1.31–3.03)||.0013|
|Age, >59 vs ≤59||0.77||2.16 (1.32–3.54)||.0021|
|ASA, >3 vs ≤3||0.51||1.67 (1.01–2.75)||.0452|
|Bed size, ≤500 vs >500||1.30||3.65 (2.19–6.09)||<.0001|
|Anesthesia, Y vs N||0.71||2.03 (1.04–3.94)||.0371|
|ASA (1, 2, 3/4/5)||0.50||1.64 (1.43–1.89)||<.0001|
|Endoscope, Y vs N||1.35||3.85 (1.57–9.49)||.0033|
|Age, ≤43 vs 44–58||1.07||2.92 (1.81–4.71)||<.0001|
|Age, >58 vs 44–58||0.62||1.86 (1.10–3.15)||.0215|
|Duration, >320 vs ≤320||1.01||2.74 (1.75–4.30)||<.0001|
|Emergency, Y vs N||0.64||1.90 (1.22–2.93)||.0042|
|ASA, >2 vs ≤2||1.38||3.99 (1.47–10.82)||.0065|
|Duration, >178 vs ≤178||1.62||5.07 (1.11–23.25)||.0367|
|ASA, >2 vs ≤2||0.57||1.77 (1.16–2.69)||.0076|
|Gender, F vs M||0.32||1.38 (1.12–1.69)||.0021|
|Medical school affiliation, N vs Y||0.23||1.26 (1.02–1.56)||.0338|
|Endoscope, Y vs N||0.58||1.78 (1.08–2.95)||.0242|
|Gender, M vs F||0.48||1.61 (1.01–2.58)||.0464|
|Wound class, CO/D vs C/CC||0.82||2.26 (1.42–3.61)||.0006|
|Approach, B/L/P vs A||2.12||8.35 (1.12–62.16)||.038|
|Diabetes, Y vs N||1.09||2.98 (1.16–7.69)||.024|
|Duration, >209 vs ≤209||1.25||3.48 (1.39–8.69)||.008|
|Duration, >125 vs ≤125||0.90||2.46 (1.87–3.24)||<.0001|
|Bed size, >200 vs ≤200||0.96||2.60 (1.76–3.84)||<.0001|
|Duration, >187 vs ≤187||1.40||4.04 (1.72–9.46)||.0013|
|Bed size, >500 vs ≤500||1.03||2.79 (1.18–6.60)||.0198|
|Age, ≤44 vs >44||0.66||1.94 (1.43-2.64)||<.0001|
|ASA, >2 vs ≤2||0.42||1.51 (1.03–2.23)||.0363|
|Duration, >100 vs ≤100||0.50||1.65 (1.22–2.23)||.0011|
|Medical school affiliation, Y vs N||0.89||2.42 (1.76–3.34)||<.0001|
|Age, ≤1 vs >1||0.77||2.16 (1.69–2.75)||<.0001|
|Medical school affiliation, Y vs N||0.69||2.00 (1.23–3.23)||.005|
|Bed size, ≤200/>500 vs 201–500||1.66||5.24 (2.92–9.40)||<.0001|
|Wound class, C vs CC/CO/D||0.82||2.27 (1.29–4.01)||.0045|
|Duration, >197 vs ≤197||0.66||1.93 (1.28–2.92)||.0017|
|Bed size, >500 vs ≤500||0.53||1.71 (1.13–2.57)||.0104|
For the NHSN risk index models, the c-index ranged from 0.51 (VSHN) to 0.77 (NECK), compared with 0.59 (COLO) to 0.85 (THYR) for the new procedure-specific risk models (resultant increase in the c-index from 0 to 0.2). For 33 procedures, the new models yielded a higher c-index than did the NHSN index models, and for 28 of these, the improvement was statistically significant (; Table 5).
The subset analysis of only complex (deep incisional and organ/space) SSIs that occurred during hospitalization or rehospitalization at the same hospital resulted in prediction models that, overall, had a c-index similar to or higher than that for all SSIs, but 9 procedures had intercept-only models, which was more than what was observed in all SSIs models (Table 6).
|Procedure code||No. of procedures||No. of SSIs||Effect||Estimate||OR (95% CI)||P||c-index|
|Wound class, CO/D vs C/CC||2.37||10.72 (3.19–36.07)||.0001|
|Emergency, Y vs N||0.87||2.38 (1.21–4.67)||.0116|
|Gender, M vs F||0.84||2.31 (1.22–4.38)||.0099|
|Bed size, >500 vs ≤500||0.94||2.56 (1.44–4.54)||.0013|
|Wound class, CO/D vs C/CC||1.07||2.90 (1.64–5.15)||.0003|
|ASA, ≤3 vs >3||1.34||3.83 (1.36–10.82)||.0113|
|Bed size, 201–500 vs ≤200/>500||1.25||3.49 (1.97–6.20)||<.0001|
|ASA, >2 vs ≤2||1.45||4.25 (1.84–9.79)||.0007|
|Bed size, ≤200/>500 vs 201–500||1.51||4.51 (1.05–19.32)||.0422|
|ASA (1/2, 3, 4/5)||0.38||1.47 (1.31–1.65)||<.0001|
|Gender, F vs M||2.29||<.0001|
|Medical school affiliation, Y vs N||0.19||1.21 (1.08–1.36)||.0009|
|Age, ≤56 vs >56 years||0.35||1.42 (1.09–1.85)||.0093|
|Duration, >306 vs ≤306||0.61||1.83 (1.40–2.40)||<.0001|
|Emergency, Y vs N||0.48||1.61 (1.07–2.41)||.0215|
|Age, >52 vs ≤52||0.79||2.21 (1.18–4.13)||.0131|
|ASA, >2 vs ≤2||0.62||1.86 (1.03–3.35)||.0382|
|Bed size, >200 vs ≤200||0.96||2.61 (1.41–4.82)||.0022|
|Age, ≤75 vs >75||0.15||1.16 (1.03–1.30)||.0137|
|ASA, >2 vs ≤2||0.33||1.39 (1.26–1.54)||<.0001|
|Endoscope, N vs Y||0.18||1.19 (1.05–1.36)||.0088|
|Medical school affiliation, N vs Y||0.16||1.18 (1.06–1.31)||.0028|
|Bed size, >200 vs ≤200||0.21||1.23 (1.10–1.37)||.0004|
|Wound class, CO/D vs C/CC||0.19||1.21 (1.08–1.36)||.0013|
|Bed size, >500 vs ≤500||0.56||1.75 (1.24–2.46)||.0013|
|Anesthesia, Y vs N||0.55||1.74 (1.09–2.78)||.0209|
|ASA (1, 2, 3/4/5)||0.53||1.69 (1.21–2.37)||.0023|
|Labor, Y vs N||0.83||2.29 (1.65–3.18)||<.0001|
|Bed size, >200 vs ≤200||0.84||2.32 (1.53–3.52)||<.0001|
|Wound class, CO/D vs C/CC||1.07||2.91 (1.50–5.65)||.0015|
|Approach, B/L/P vs A||0.94||2.56 (1.85–3.55)||<.0001|
|ASA (1/2, 3, 4/5)||0.60||1.82 (1.54–2.16)||<.0001|
|Diabetes, Y vs N||0.39||1.48 (1.13–1.93)||.0045|
|Medical school affiliation, Y vs N||0.34||1.41 (1.10–1.80)||.0067|
|Spinal level, CD/DL vs AX/AC/CV||0.82||2.26 (1.51–3.40)||<.0001|
|Spinal level, LL vs AX/AC/CV||0.49||1.63 (1.20–2.22)||.002|
|Age, >25 vs ≤25||0.83||2.29 (1.36–3.86)||.0018|
|Duration, >138 vs ≤138||0.92||2.52 (1.72–3.69)||<.0001|
|Bed size, >200 vs ≤200||0.55||1.73 (1.17–2.56)||.0064|
|Age, ≤71 vs >71||0.93||2.53 (1.36–4.71)||.0035|
|ASA (1, 2, 3/4/5)||0.75||2.12 (1.39–3.22)||.0005|
|Gender, F vs M||0.85||2.35 (1.50–3.69)||.0002|
|Bed size, >200 vs ≤200||0.83||2.30 (1.31–4.01)||.0035|
|Anesthesia, Y vs N||0.17||1.19 (1.03–1.36)||.016|
|ASA, 3 vs 1/2||0.82||2.27 (1.98–2.59)||<.0001|
|ASA, 4/5 vs 1/2||1.07||2.91 (2.34–3.61)||<.0001|
|Type of surgerya||0.35||1.43 (1.32–1.54)||<.0001|
|Medical school affiliation, Y vs N||0.19||1.21 (1.07–1.37)||.003|
|Bed size, >200 vs ≤200||0.31||1.37 (1.20–1.56)||<.0001|
|Trauma, Y vs N||0.24||1.27 (1.05–1.53)||.0126|
|ASA (1, 2, 3/4/5)||0.73||2.08 (1.73–2.50)||<.0001|
|Bed size, ≤500 vs >500||0.33||1.39 (1.07–1.80)||.0137|
|Age, ≤58 vs >58||0.34||1.41 (1.24–1.61)||<.0001|
|ASA (1/2, 3, 4/5)||0.49||1.64 (1.47–1.82)||<.0001|
|Gender, M vs F||0.35||1.42 (1.26–1.60)||<.0001|
|Revision vs primary||0.78||2.18 (1.85–2.58)||<.0001|
|Medical school affiliation, Y vs N||0.16||1.18 (1.04–1.33)||.0096|
|Bed size, >200 vs ≤200||0.18||1.20 (1.04–1.38)||.01|
|Trauma, Y vs N||0.68||1.97 (1.18–3.31)||.0099|
|ASA, >3 vs ≤3||0.87||2.39 (1.09–5.22)||.0292|
|ASA (1, 2, 3, 4/5)||0.52||1.68 (1.38–2.03)||<.0001|
|Medical school affiliation, N vs Y||0.66||1.94 (1.37–2.76)||.0002|
|Bed size, >500 vs ≤500||0.61||1.84 (1.32–2.56)||.0003|
|Age, ≤43 vs 44–58||1.30||3.66 (2.18–6.16)||<.0001|
|Age, >58 vs 44–58||0.78||2.18 (1.23–3.86)||.0074|
|Duration, >320 vs ≤320||1.14||3.12 (1.92–5.06)||<.0001|
|Age, ≤58 vs >58||0.56||1.75 (1.27–2.39)||.0005|
|ASA, >3 vs ≤3||0.39||1.47 (1.07–2.02)||.0173|
|Medical school affiliation, N vs Y||0.62||1.86 (1.36–2.55)||.0001|
|Gender, M vs F||1.06||2.87 (1.39–5.92)||.0043|
|Bed size, >500 vs ≤500||1.24||3.46 (1.43–8.40)||.006|
|Duration, >209 vs ≤209||1.37||3.94 (1.46–10.63)||.0069|
|Duration, >125 vs ≤125||0.92||2.51 (1.73–3.64)||<.0001|
|Bed size, 201–500 vs ≤200||0.99||2.8 (1.40–5.12)||.0028|
|Bed size, >500 vs ≤200||1.08||2.96 (1.71–5.12)||.0001|
|Duration, >187 vs ≤187||1.93||6.85 (2.10–22.35)||.0014|
|Medical school affiliation, Y vs N||0.87||2.38 (1.61–3.53)||<.0001|
|Age, ≤1 vs >1||0.76||2.14 (1.67–2.75)||<.0001|
|Medical school affiliation, Y vs N||0.62||1.86 (1.15–3.02)||.012|
|Bed size, ≤200/>500 vs 201–500||1.77||5.87 (3.11–11.11)||<.0001|
|Wound class, C vs CC/CO/D||0.75||2.12 (1.20–3.74)||.0094|
Risk models based on the NHSN risk index, although simple in design, showed poor predictive performance for many procedures. New procedure-specific predictive models developed with currently collected NHSN data elements significantly improved the predictive performance for most procedures, including all of the most common procedures reported to the NHSN.
This study represents a large and robust data set of almost 850,000 surgical procedures among 39 procedure categories reported since 2006 by 847 hospitals in 43 states. Most of the potential predictive factors included have been previously identified as risk factors in other studies.6,22-32 The c-indices also approximate what has been reported in other studies,6,25,32 which suggests some reproducibility in these findings.
We found that the procedure duration was the most common of the 3 traditional NHSN risk index parameters selected by 30 of the 39 models; ASA score was the next most common (21 models). Age, which is not a component of the traditional NHSN risk index, was the third most commonly selected factor (included in 18 models). Because patient-specific variables available for analysis were limited, we also included hospital-level variables. These likely serve as proxy indicators for patient case mix or possibly for surgical programs. We incorporated hospital-specific information, including the number of hospital beds (16 models) and medical school affiliation (5). Including these latter variables as well as procedure duration could introduce some risk adjustment for surgical performance (ie, surgical residents performing at teaching facilities) and/or for patient case mix (higher risk patients cared for at teaching facilities). Until further patient- (eg, BMI and diabetes) and procedure-specific data are available to allow comparable risk adjustment without including such proxy indicators or intermediate outcomes (like duration), we decided to maximize risk adjustment using all of the information available.
Although SSI prediction improved considerably with our models, the resulting c-indices still remained relatively low. This may result from the characteristics of the NHSN surveillance data in which, for most procedures, there are no procedure-specific risk factors. For the 5 procedures for which procedure-specific data elements were collected, improvement was noted. For example, in addition to those factors collected across all procedures, our CSEC model included BMI and whether the patient was in labor. This resulted in a model with significant improvement in predictive performance, compared with that reported by Brandt et al6 (0.66 vs 0.55), which included only ASA score, procedure duration, age, and wound class. Further improvement can be expected with additional patient- and procedure-specific factors, such as diabetes, duration of preoperative hospital stay, indication for surgery, and the number of discharge diagnoses.33 Beginning in January 2013, the NHSN will require submission of BMI and diabetes information for all procedures.
Our findings indicate that, with use of the currently available NHSN data, the new procedure-specific risk models significantly improved SSI prediction. This justifies their use in facility-specific performance comparisons with an external benchmark, which serve as guides for internal quality improvement efforts. To enable NHSN users to take advantage of the new procedure-specific risk models, the CDC has incorporated them into the NHSN application. The new models supersede the NHSN risk index for procedures in which the traditional NHSN risk index has little discriminatory power. Improved risk adjustment may provide SSI data that are more convincing to clinicians and thus more effective in guiding changes in infection-prevention practices. In addition, separate models for predicting the subset of SSIs classified as complex (deep incisional or organ/space infections) detected during initial hospitalization or upon rehospitalization at the same hospital were developed (Table 6). These models may be more acceptable for public reporting, because there may be less variability to detect this subset between facilities when excluding those infections detected by surveillance after hospital discharge and superficial infections. In addition, the model fit, as measured by the c-index, was improved for a number of the procedures, which indicates that perhaps the identified risk factors are better at predicting this subset of SSIs. However, any models developed for public reporting will need frequent reevaluation as more information becomes available and the quality measure environment changes.19 Likewise, even for the overall SSI models, caution should also be exercised when evaluating some of these models. Specifically, 9 procedure categories (AVSD, CEA, HTP, NEPH, OVRY, PACE, PRST, SPLE, and THYR) had fewer than 20 SSI events, and for 4 of these (CEA, HTP, PACE, and SPLE), we were able to construct intercept-only models. The intercept-only models produce essentially unadjusted infection rates for comparison, and for the other models, the risk estimates might not be stable because of an insufficient number of SSI cases. These models can and should be modified as additional information on methods to improve risk adjustment (eg, the addition of specific patient-level variables) for specific procedures and the ability to reliably and effortlessly acquire these variables from surgical or facility information systems as part of routine SSI surveillance improve. This is an ongoing, deliberate, and iterative process. The NHSN is committed to pursue additional efforts to present the best available risk-adjusted SSI data to reporting facilities and to make accurate overall assessments of the status of SSI prevention efforts in the United States.
We thank the NHSN participants for their ongoing efforts to monitor healthcare-associated infections and improve patient safety and our colleagues in the Division of Healthcare Quality Promotion for their tireless support of this unique public health network.
Potential conflicts of interest. All authors report no conflicts of interest relevant to this article.
Outlier exclusion rules:
(1) Exclude all procedures where duration in minutes = 0 ();
(2) Exclude all procedures where patient was less than 1 day old or greater than 109 years old ();
(3) Exclude all procedures where wound class was undefined ();
(4) Exclude all procedures with duration between 0 and 5 minutes or more than 5 times the interquartile range ().
Bootstrap resampling steps:
(1) For each procedure category, 100 independent samples of the same size as the original sample were obtained, each of which was a simple random sample with replacement;
(2) Logistic regression was applied to each sample using selected risk factors;
(3) The 95% confidence intervals based on 100 independent samples for the estimated effects (of the risk factors) were calculated;
(4) If the effects at the 2.5th percentile and the 97.5th percentile were both positive (being risk factors) or negative (being protective factors), the effects were deemed to be significant; if the lower and the upper bound of the effects pointed to different directions (one being positive and the other being negative), the effect was deemed to be nonsignificant;
(5) Nonsignificant effect was removed from the models, and the stepwise model selection was run to see whether other new effects could enter the models with this effect absent. The above bootstrapping process was repeated to validate the new models.
(6) If several effects were found to be nonsignificant through bootstrapping, we removed the least significant effect in step 5.
- 1. Klevens RM, Edwards JR, Richards CL Jr, et al. Estimating health care–associated infections and deaths in U.S. hospitals. Public Health Reports 2002;122:160–167.
- 2. Mangram AJ, Horan TC, Pearson ML, Silver LC, Jarvis WR. Guideline for prevention of surgical site infection. Infect Control Hosp Epidemiol 1999;20:250–280.
- 3. Scott II RD. The direct medical costs of healthcare-associated infections in U.S. hospitals and the benefits of prevention. 2009. http://www.cdc.gov/ncidod/dhqp/pdf/Scott_CostPaper.pdf. Accessed August 12, 2011.
- 4. Haley RW, Culver DH, White JW, et al. The efficacy of infection surveillance and control programs in preventing nosocomial infections in United States hospitals. Am J Epidemiol 1985;121:182–205.
- 5. Gaynes R, Richards C, Edwards JR, et al. Feeding back surveillance data to prevent hospital-acquired infections. Emerg Infect Dis 2001;7:295–298.
- 6. Brandt C, Hansen S, Sohr D, Daschner F, Ruden H, Gastmeier P. Finding a method for optimizing risk adjustment when comparing surgical-site infecions rates. Infect Control Hosp Epidemiol 2004;25:13–18.
- 7. Anderson DJ, Kaye KS, Classen D, et al. Strategies to prevent surgical site infections in acute care hospitals. Infect Control Hosp Epidemiol 2008;(suppl 1):S51–S61.
- 8. Culver DH, Horan TC, Gaynes RP, et al. Surgical wound-infection rates by wound class, operative procedure, and patient risk index. Am J Med 1991;91:S152–S157.
- 10. US Department of Health and Human Services (HHS). HHS action plan to prevent healthcare associated infections: appendices. Appendix G. http://www.hhs.gov/ophs/initiatives/hai/appendices.html Accessed July 28, 2010.
- 11. Anderson DJ, Chen LF, Sexton DJ, Kaye KS. Complex surgical site infections and the devilish details of risk adjustment: important implications for public reporting. Infect Control Hosp Epidemiol 2008;29:941–946.
- 12. Shahian DM, O’Brien SM, Filardo G, Ferraris VA, Haan CK, Rich JB, et al. The Society of Thoracic Surgeons 2008 cardiac surgery risk models: part 1—coronary artery bypass grafting surgery. Ann Thorac Surg 2009;88:S2–S22.
- 13. Friedman ND, Bull AL, Russo PL, Gurrin L, Richards M. Performance of the National Nosocomial Infections Surveillance risk index in predicting surgical site infection in Australia. Infect Control Hosp Epidemiol 2007;28:55–59.
- 14. Paul M, Raz A, Leibovici L, Madar H, Holinger R, Rubinovitch B. Sternal wound infection after coronary artery bypass graft surgery: validation of existing risk scores. J Thorac Cardiovasc Surg 2007;133:397–403.
- 15. Roy MC, Herwaldt LA, Embrey R, Kuhns K, Wenzel RP, Eerl TM. Does the Centers for Disease Control’s NNIs system risk index stratify patients undergoing cardiothoracic operations by their risk of surgical-site infection. Infect Control Hosp Epidemiol 2000;21:186–190.
- 16. Edwards JR, Peterson KD, Mu Y, et al. National Healthcare Safety Network (NHSN) report: data summary for 2006 through 2008, issued December 2009. Am J Infect Control 2009;37:783–805.
- 18. Centers for Disease Control and Prevention. National Healthcare Safety Network: surgical site infection (SSI) event. http://www.cdc.gov/nhsn/PDFs/pscManual/9pscSSIcurrent.pdf. Accessed June 26, 2010.
- 19. National Quality Forum. National voluntary consensus standards for the reporting of healthcare-associated infections data. National Quality Forum, 2008. http://www.qualityforum.org/Publications/2008/03/National_Voluntary_Consensus_Standards_for_the_Reporting_of_Healthcare-Associated_Infection_Data.aspx. Accessed August 9, 2010.
- 20. Hanley JA, McNeil BJ. The meaning and use of the area under a receive operating characteristic (ROC) curve. Radiology 1982;143:29–36.
- 21. Hanley JA, McNeil BJ. A method of comparing the areas under receiver operating characteristic curves derived from the same cases. Radiology 1983;148:839–843.
- 22. Abboud CS, Wey SB, Baltar VT. Risk factors for mediastinitis after cardiac surgery. Ann Thorac Surg 2004;77:676–683.
- 23. de Boer AS, Mintjes-de Groot AJ, Severijnen AJ, van den Berg JM, van Pelt W. Risk assessment for surgical-site infections in orthopedic patients. Infect Control Hosp Epidemiol 1999;20:402–407.
- 24. Gaynes RP. Surgical-site infections (SSI) and the NNIS SSI risk index, part II: room for improvement. Infect Control Hosp Epidemiol 2001;22:268–272.
- 25. The Parisian Mediastinitis Study Group. Risk factors for deep sternal wound infections after sternotomy: a prospective, multicenter study. J Thorac Cardiovasc Surg 2006;111:1200–1207.
- 26. Harrignton G, Russo P, Spelman D, et al. Surgical-site infection rates and risk factor analysis in coronary artery bypass graft surgery. Infect Control Hosp Epidemiol 2004;25:472–476.
- 27. Killian CA, Graffunder EM, Vinciguerra TJ, Venezia RA. Risk factors for surgical-site infections following cesarean section. Infect Control Hosp Epidemiol 2001;22:613–617.
- 28. Kivi M, Mannien J, Wille JC, van den Hof S. Surgical site infection surveillance and the predictive power of the National Nosocomial Infection Surveillance index as compared with alternative determinants in the Netherlands. Am J Infect Control 2008;36:S27–S31.
- 29. Neumayer L, Hosokawa P, Itani K, El-Tamer M, Henderson WG, Khuri SF. Multivariable predictors of postoperative surgical site infection after general and vascular surgery: results from the patient safety in surgery study. J Am Coll Surg 2007;204:1178–1187.
- 30. Olsen MA, Lock-Buckley P, Hopkins D, Polish LB, Sundt TM, Fraser VJ. The risk factors for deep and superficial chest surgical-site infections after coronary artery bypass graft surgery are different. J Thorac Cardiovasc Surg 2002;124:136–145.
- 31. Russo PL, Spelman DW. A new surgical-site infection risk index using risk factors identified by multivariate analysis for patients undergoing coronary artery bypass graft surgery. Infect Control Hosp Epidemiol 2002;23:372–376.
- 32. Tran TS, Jamulitrat S, Chongsurvivatwong V, Geater A. Risk factors for postcesarean surgical site infection. Obstet Gynecol 2000;95:367–371.
- 33. Geubbels ELPE, Grobbee DE, Vandenbroucke-Grauls CMJE, Wille JC, de Boer AS. Improved risk adjustment for comparison of surgical site infection rates. Infect Control Hosp Epidemiol 2006;27:1330–1339.