Reliability and Validity of a Standardized Measure of Influenza Vaccination Coverage among Healthcare Personnel
(See the commentary by Sickbert-Bennett and Weber, on pages 346–348.)
Objective. To evaluate the reliability and validity of a standardized measure of healthcare personnel (HCP) influenza vaccination.
Setting. Acute care hospitals, long-term care facilities, ambulatory surgery centers, physician practices, and dialysis centers from 3 US jurisdictions.
Participants. Staff from 96 healthcare facilities randomly sampled from 234 facilities that completed pilot testing to assess the feasibility of the measure.
Methods. Reliability was assessed by comparing agreement between facility staff and project staff on the classification of HCP numerator (vaccinated at facility, vaccinated elsewhere, contraindicated, declined) and denominator (employees, credentialed nonemployees, other nonemployees) categories. To assess validity, facility staff completed a series of case studies to evaluate how closely classification of HCP groups aligned with the measure’s specifications. In a modified Delphi process, experts rated face validity of the proposed measure elements on a Likert-type scale.
Results. Percent agreement was high for HCP vaccinated at the facility (99%) and elsewhere (95%) and was lower for HCP who declined vaccination (64%) or were medically contraindicated (64%). While agreement was high (more than 90%) for all denominator categories, many facilities’ staff excluded nonemployees for whom numerator and denominator status was difficult to determine. Validity was lowest for credentialed and other nonemployees.
Conclusions. The standardized measure of HCP influenza vaccination yields reproducible results for employees vaccinated at the facility and elsewhere. Adhering to true medical contraindications and tracking declinations should improve reliability. Difficulties in establishing denominators and determining vaccination status for credentialed and other nonemployees challenged the measure’s validity and prompted revision to include a more limited group of nonemployees.
For more than 2 decades, the Advisory Committee on Immunization Practices (ACIP) has recommended annual influenza vaccination of all healthcare personnel (HCP)1 to reduce nosocomial spread of influenza, influenza-associated mortality, and HCP absenteeism.2-5 However, HCP vaccination coverage was estimated to be 64% for the 2010–2011 influenza season,6 far below the Healthy People 2020 objective of 90%.7 Measurement of influenza vaccination rates and feedback to healthcare facility staff and administration provides actionable data that may facilitate institutional efforts to increase HCP vaccination. Measurement is recommended by the ACIP, the Healthcare Infection Control Practices Advisory Committee of the Centers for Disease Control and Prevention (CDC), the Society for Healthcare Epidemiology of America, the National Foundation for Infectious Diseases, and the Association of Professionals in Infection Control and Epidemiology,8-12 among others. Beginning July 1, 2012, the Joint Commission strengthened accreditation standard IC.02.04.0113—which required facilities to offer influenza vaccination to staff—to additionally require measurement and reporting of vaccination rates, as well as goal setting consistent with attaining the 2020 objective.7 The standard will be extended in a phased approach from hospitals and long-term care facilities (LTCFs) to a variety of inpatient and outpatient facility types.14
To reduce the substantial variation in facility-level measurement of HCP influenza vaccination coverage,15 the CDC sponsored the development of a standardized measure receiving provisional endorsement from the National Quality Forum (NQF) in 2008.16 The CDC-sponsored measure, which aimed to produce HCP vaccination data that are comprehensive within a single facility and comparable across facilities, was pilot tested during the 2010–2011 influenza season with the goal of obtaining full NQF endorsement.17,18 In this article, we report the reliability and validity results of the pilot test.
Four pilot jurisdictions across the United States (California, New Mexico, New York City, and western Pennsylvania) recruited participants from 5 types of healthcare facilities: acute care hospitals (ACHs), LTCFs, ambulatory surgery centers (ASCs), physician practices, and dialysis centers. Of 318 facilities recruited, staff from 234 (74%) completed 3 quantitative surveys on individual facility characteristics and the feasibility of implementing the CDC-sponsored measure (including the groups of HCP they were unable to include in data reporting during the pilot test). Sampling and data collection methods and data on reported barriers and facilitators are described elsewhere.17,18
Participating facilities reported the total number of HCP who worked at the facility for at least 1 day during the period October 1, 2010, to March 31, 2011, in mutually exclusive numerator (vaccination at facility, vaccination elsewhere, contraindication, documented declination) and denominator (employees, credentialed nonemployees, other nonemployees) categories according to the CDC-sponsored measure specifications (Table 1). The percentage of facilities reporting data for all numerator and denominator cells was calculated. We selected questions from the pilot quantitative surveys to assess the completeness and perceived accuracy of the reported numerator and denominator data.
|Vaccinated at facility||Received an influenza vaccine at this healthcare facility since August 2010|
|Vaccinated elsewhere||Received an influenza vaccine elsewhere since August 2010a|
|Medically contraindicated||Have a medical contraindicationa,b|
|Declined||Declined to receive the influenza vaccine for any reason|
|Employees||All persons who receive a paycheck from the healthcare facility, regardless of whether they have direct patient contact|
|Credentialed nonemployees||Licensed practitioners affiliated with the healthcare facility who do not receive a paycheck from the healthcare facility, regardless of whether they have direct patient contact (eg, physicians or other providers with clinical or admitting privileges at the healthcare institution and nonemployee technicians or therapists with professional credentialing)|
|Other nonemployees||Noncredentialed persons affiliated with the healthcare facility who do not receive a paycheck from the facility, regardless of whether they have direct patient contact (eg, contractors, students or trainees, resident physicians or fellows not paid by the institution, and volunteers)|
Reliability and Case Study Sample
Data collection instruments were field tested in Pennsylvania facilities. Facilities completing the pilot test in California, New Mexico, and New York City (for anonymity, referred to in no particular order as jurisdictions A, B, and C) were recruited for reliability testing of their 2010–2011 influenza season data submitted during the pilot. Within each jurisdiction, facilities were randomly selected with the goal of recruiting 8 facilities from each of the 5 facility types. If jurisdictions were unable to recruit 8 of each facility type, they could recruit more than 8 facilities of another type. Ninety-six facilities agreed to participate.
Interrater reliability was assessed by comparing agreement between 2 raters (facility staff vs project staff in pilot jurisdictions) on the classification of HCP into numerator and denominator categories. Staff were instructed to select 60 HCP records (20 employees, 20 credentialed nonemployees, and 20 other nonemployees) using systematic or simple random sampling. At facilities with fewer than 20 HCP in each category, staff selected more than 20 records from a different HCP category, if available, to meet the goal of 60 records per facility. Facility records were reviewed and classified into numerator and denominator categories, including the categories “missing” and “uncertain.” Electronic or paper records were reviewed on the basis of the data system used by the facility.
Project staff conducted site visits in jurisdictions A and B (referred to as method 1), while facility staff mailed records without identifying information to project staff in jurisdiction C (referred to as method 2), because of travel limitations. A HCP record was excluded from reliability calculations if jurisdiction C project staff did not receive documentation or received only a log sheet from facility staff.
For each jurisdiction and facility type, we calculated percent agreement and Cohen’s κ statistic, which measures the proportion of agreement beyond that expected by chance.19 Possible values of κ range from −1 (no agreement) to 1 (perfect agreement), with a value of 0 indicating agreement no better than that expected by chance. We constructed 95% confidence intervals around the sample estimate using the standard error of κ. Reported κ coefficients and percent agreement were adjusted by assigning a weight of 0 to records classified by facility staff as missing or uncertain to control for the possible overestimation of agreement when both raters selected missing/uncertain.
All statistical analyses were completed using SAS 9.3 (SAS Institute).
Twenty-three brief case studies were used to confirm that measure specifications were understood by pilot facilities. Facilities participating in the reliability assessment were asked to classify the hypothetical case described in each scenario into the appropriate numerator or denominator category. The proportion of facilities selecting the correct response on the basis of measure specifications was calculated.
Face validity, which assesses the suitability of a measurement instrument as a source of data using common-sense criteria,20 was evaluated using a 2-round modified Delphi process.21 A panel of 9 experts in influenza vaccination measurement and quality improvement from multiple public and private organizations individually rated elements of the CDC-sponsored measure using a Likert-type scale of 1 (unlikely to produce an accurate measurement) to 9 (very likely to produce an accurate measurement). Results from the pilot test feasibility surveys,18 interrater reliability assessment, and case studies were provided to the panel. During a moderated telephone conference, experts discussed the aggregated results from the first round of ratings. After the call, experts again individually rated a revised set of elements (round 2 ratings).
Characteristics of Participating Facilities
Demographic characteristics and institutional HCP vaccination policies of the 96 facilities recruited for reliability testing are summarized in Table 2. Forty-five percent of these facilities had a policy requiring a declination statement for unvaccinated HCP. One-third of the facilities had no experience measuring HCP influenza vaccination rates prior to the 2010–2011 influenza season. While 95% of facilities collected data on employee vaccinations at the facility during the measure pilot test (2010–2011), only 73% tracked vaccinations among credentialed nonemployees, and 67% tracked vaccinations among other nonemployees. Twenty-eight percent tracked employee vaccinations using electronic administrative data, electronic occupational health records, or registry databases.
|Acute care hospitals (n = 27)||Long-term care facilities (n = 19)||Ambulatory surgery centers (n = 13)||Physician practices (n = 19)||Dialysis centers (n = 18)||Overall (n = 96)|
|Private, for profit||26 (7)||32 (6)||100 (13)||53 (10)||28 (5)*||43 (41)|
|Private, not for profit||52 (14)||32 (6)||0 (0)||21 (4)||67 (12)*||38 (36)|
|Public, federal/state/local||22 (6)||37 (7)||0 (0)||26 (5)||6 (1)||20 (19)|
|Urban||59 (16)||58 (11)||70 (9)||53 (10)||33 (6)||54 (52)|
|Suburban||11 (3)||16 (3)||23 (3)||11 (2)||39 (7)||19 (18)|
|Rural||30 (8)||26 (5)||8 (1)||37 (7)||28 (5)||27 (26)|
|Size, median (range)|
|No. of total HCP||2,083 (44–9,572)||256 (40–1,657)||48 (23–366)||14 (2–293)||25 (4–93)||97 (2–9,572)|
|% of HCP who are nonemployeesa||24 (9–50)||11 (0–27)||51 (0–69)||0 (0–67)||11 (0–28)||15 (0–69)|
|Facility requires a declination statement for unvaccinated HCP|
|Yes||74 (20)||63 (12)||38 (5)||16 (3)||17 (3)*||45 (43)*|
|Prior experience measuring HCP vaccination|
|Never measured||0 (0)||21 (4)||92 (12)||58 (11)||28 (5)||33 (32)|
|During 2009–2010 only||0 (0)||11 (2)||0 (0)||11 (2)||17 (3)||7 (7)|
|2–4 years||56 (15)||16 (3)||8 (1)||11 (2)||6 (1)||23 (22)|
|5 or more years||44 (12)||53 (10)||0 (0)||21 (4)||53 (9)||36 (35)|
|Collected data on HCP vaccinated at the facility, 2010–2011b|
|Employees||100 (26)||100 (16)||92 (12)||95 (18)||82 (15)||95 (86)|
|Credentialed nonemployees||96 (25)||58 (7)||64 (7)||75 (6)||46 (6)||73 (51)|
|Other nonemployees||96 (25)||67 (8)||60 (3)||67 (4)||14 (2)||67 (42)|
|Employee vaccination tracked electronicallyc|
|Yes||48 (13)||21 (4)||8 (1)||32 (6)||17 (3)||28 (27)|
These facilities were representative of the overall 234 pilot facilities18 with the exception of dialysis centers. Among dialysis centers, private, nonprofit facilities were overrepresented in our sample. Significantly fewer dialysis centers participating in the reliability study had a policy requiring declination statements (17%) compared with all pilot dialysis centers (42%; ; data not shown).18
Reliability data were collected from 90% (86/96) of facilities. Ten facilities did not have documentation available for review because HCP influenza vaccinations were not recorded and staff collected data for the pilot by self-report because of the small number of total HCP. Additionally, of the 931 HCP records selected at facilities in jurisdiction C (method 2), 170 (18%) numerator records and 196 (21%) denominator records were excluded because facility staff mailed only a log list for certain types of HCP (eg, students, contracted nurses) or failed to mail records to project staff.
Overall adjusted agreement between facility staff and project staff was 91% (2,682/2,934) for numerator data (adjusted ) and 96% (3,208/3,356) for denominator data (adjusted ). Rates of agreement differed moderately by jurisdiction and facility type (Table 3).
|Adjusted % agreementa||Adjusted κa (95% CI)||Adjusted % agreementa||Adjusted κa (95% CI)|
|Overall||91||0.82 (0.80–0.83)||96||0.91 (0.90–0.93)|
|A||85||0.73 (0.69–0.76)||98||0.97 (0.95–0.98)|
|B||93||0.83 (0.80–0.87)||98||0.96 (0.94–0.98)|
|Cb||97||0.94 (0.91–0.96)||85||0.77 (0.74–0.81)|
|Acute care hospitals||99||0.96 (0.94–0.98)||95||0.93 (0.91–0.95)|
|Long-term care facilities||96||0.93 (0.90–0.96)||99||0.96 (0.94–0.98)|
|Ambulatory surgery centers||87||0.72 (0.65–0.79)||79||0.51 (0.42–0.61)|
|Physician practices||84||0.63 (0.58–0.69)||95||0.44 (0.27–0.67)|
|Dialysis centers||77||0.55 (0.50–0.60)||99||0.96 (0.92–1.00)|
Raters agreed on the numerator status for 99% (2,070/2,082) of the records selected for HCP vaccinated at the facility, 95% (199/210) of HCP vaccinated elsewhere, 64% (27/42) of HCP with medical contraindications, and 64% (386/600) of HCP who declined vaccination (Table 4). Most numerator disagreements (84% [211/252]) resulted from facility staff classifying HCP as having declined vaccination when no documentation of declination existed rather than categorizing them as missing/uncertain, in accordance with pilot protocol. This type of numerator disagreement occurred almost exclusively in ASCs (), physician practices (), and dialysis centers () and accounted for the lower κ values observed in these outpatient settings.
|Numerator category determined by jurisdiction project staff|
|Numerator category reported by facility staff||Vaccinated at institution||Vaccinated elsewhere||Medical contraindication||Declined vaccination||Missing/uncertain||Total||% agreement|
|Vaccinated at institution||2,070||7||0||0||5||2,082||99|
The second major source of numerator disagreement was in the misclassification of medical contraindications. Of the 42 contraindications recorded by facility staff during record review, 15 were classified by project staff as declined or missing/uncertain because the healthcare worker did not cite a reason that was a true medical contraindication for inactivated influenza vaccine according to ACIP guidelines:22 1 “had flu already” and 5 cited pregnancy, including “[physician] recommended avoiding vaccination while pregnant.” According to surveys completed during pilot testing, of the 91 facilities participating in reliability testing and offering inactivated influenza vaccine, 29 (32%) reported including pregnancy in the medical contraindications numerator category. Additionally, 46% of facilities included moderate to severe illness, and 13% included age more than 50 years.
Raters agreed on the denominator status of 97% (2,201/2,270) of employee records, 91% (479/526) of credentialed nonemployees, and 94% (528/560) of other nonemployees (Table 5). Most denominator disagreements (70% [104/148]) resulted from project staff classifying records as missing/uncertain rather than in other categories. The other 44 disagreements resulted from varying interpretations of “credentialed” nonemployees.
|Denominator category determined by jurisdiction project staff|
|Denominator category reported by facility staff||Employee||Credentialed nonemployee||Other nonemployee||Missing/uncertain||Total||% agreement|
Completeness and Perceived Accuracy
Many facilities, including more than 40% of ACHs, were unable to report complete numerator or denominator data for credentialed nonemployees and other nonemployees (Table 6); facility staff cited unclear definitions of credentialed nonemployees and an inability to determine credentialed status and vaccination status from existing records. Contracted clinical personnel, contracted custodial staff, and construction workers were commonly omitted from reported denominators either because none worked in the facility or because they could not be tracked. Perceived accuracy of the reported nonemployee denominators was also low. Because of these constraints (and the percentage of HCP working in each facility who are nonemployees), the majority of records selected by facility staff for reliability testing were for employees: 45% of records from ACHs, 80% from LTCFs, 70% from ASCs, 93% from physician practices, and 82% from dialysis centers (data not shown). Overall numerator and denominator agreement was similar when recalculated only among those facilities including nonemployee records as well as employee records during reliability testing.
|Acute care hospitals (n = 27)||Long-term care facilities (n = 2,719)||Ambulatory surgery centers (n = 2,713)||Physician practices (n = 2,719)||Dialysis centers (n = 2,718)||Overall (n = 2,796)|
|Reported complete numerator and denominator dataa|
|Employees||85 (23)||79 (15)||92 (12)||100 (19)||100 (18)||91 (87)|
|Credentialed nonemployees||59 (16)||58 (11)||54 (7)||100 (19)||83 (15)||71 (68)|
|Other nonemployees||52 (14)||58 (11)||69 (9)||100 (19)||61 (11)||67 (64)|
|Groups commonly excluded from reported data|
|Contracted clinical personnel|
|None working here||11 (3)||47 (9)||77 (10)||79 (15)||89 (16)||55 (53)|
|Could not trackb||21 (5)||10 (1)||33 (1)||0 (0)||0 (0)||16 (7)|
|Contracted custodial staff|
|None working here||37 (10)||47 (9)||39 (5)||68 (13)||22 (4)||43 (41)|
|Could not trackb||12 (2)||20 (2)||75 (6)||50 (3)||71 (10)||42 (23)|
|None working here||26 (7)||53 (10)||69 (9)||84 (16)||72 (13)||57 (55)|
|Could not trackb||50 (10)||67 (6)||75 (3)||67 (2)||80 (4)||71 (19)|
|Accuracy of reported denominator data: very accuratec|
|Employees||74 (20)||58 (11)||92 (12)||89 (17)||72 (13)||76 (73)|
|Credentialed nonemployees||30 (8)||23 (5)||46 (6)||84 (16)||78 (14)||51 (49)|
|Other nonemployees||11 (3)||16 (3)||54 (7)||89 (17)||33 (6)||38 (36)|
Ninety participants (94%) completed the case studies. The majority of respondents across all facility types identified the correct numerator category in each case, with the exception of the 4 scenarios describing HCP who deferred vaccination all season or verbally refused vaccination for a variety of reasons (Table A1). Respondents tended to classify these as declinations, but because they were not documented as per the pilot protocol, the correct choice was “none of the above.”
Denominator cases studies were correctly classified by the majority of respondents with the exception of the following: only 44.4% (40/89) of staff correctly categorized physicians who are credentialed by the facility but are not on the facility’s payroll and who had not admitted patients to the facility in the prior 12 months (Table A2); staff tended to categorize facility owners who are physicians and work part time as credentialed nonemployees rather than as employees (47.8% [43/90]), although 81.3% (13/16) of staff from private practices identified the correct answer (data not shown).
The modified Delphi panel of 9 experts reached consensus on the validity of the following numerator definitions: influenza vaccination received at the facility, documented receipt of influenza vaccination outside the facility, documented receipt of a medical contraindication to vaccination, and documented declination of vaccine for nonmedical reasons, including religious exemptions (Table B1). There was considerable disagreement among panelists regarding the ability of the measure to produce valid results if undocumented self-reported data were included.
Experts reached the strongest consensus on the validity of the following denominator definitions: credentialed nonemployees defined as nonemployee physicians, advanced practice nurses, and physician assistants working at the facility for 30 or more days between October 1 and March 31 of the following year; other nonemployees defined as students and volunteers working at the facility for 30 or more days between October 1 and March 31 of the following year; and other nonemployees defined as all nonemployees who are required by the facility to undergo a periodic skin test for tuberculosis (Table B2).
This study evaluated the reliability and validity of a CDC-sponsored standardized measure of HCP influenza vaccination coverage for use across various healthcare settings. Results demonstrate high reliability and validity for numerator data reported for employees vaccinated at the facility and elsewhere. Outpatient settings were able to accurately discern these numerators, despite reporting less prior experience measuring vaccination. Across all facility types, employee denominators were reported with high reliability and validity. While the majority of nonemployee records were classified correctly, many facilities’ reports did not include nonemployee HCP for whom this information could not be determined. Establishing complete denominators for nonemployees can be difficult,23 because volunteers and contracted staff are not usually included in a facility’s human resources database24 and there is a lack of existing infrastructure or precedent for tracking vaccination in these groups.15 Additionally, our study found that inclusion of all credentialed nonemployees in the piloted measure decreased the comparability of results, as credentialing policies of HCP vary by state and facility.
To address these concerns about the validity of the measure as piloted and to reduce institutional barriers to reporting,17,18 denominator specifications were revised in accordance with Delphi panel ratings and guidance from the project’s Steering Committee to include a more limited number of HCP among whom vaccination could currently be more reliably and accurately measured: employees, licensed independent practitioners (physicians, advanced practice nurses, and physician assistants only), and adult students/trainees and volunteers. Nonemployee groups were not defined by the requirement for periodic tuberculosis testing because of variable testing requirements among facilities in different states.
To balance feasibility of measure implementation with concerns raised by the Delphi panel experts regarding the accuracy of self-reported data, the revised measure specifications require vaccinations received outside the facility to be documented but allow for self-report of declinations and medical contraindications. Policies requiring declination statements for unvaccinated HCP are nearly universal among hospitals in the United States with an institutional influenza vaccination requirement.25 While studies evaluating the effect of declination forms on vaccination rates have mixed results,26-29 the forms facilitate differentiation of HCP who decline from those who are not reached by existing vaccination campaigns.30 The requirement for declinations to be documented during the pilot accounted for the majority of case study numerator misclassifications and the lower numerator κ values observed in outpatient settings because, unlike the ACHs in our study, the majority of these outpatient facilities did not use a declination form and facility reporters likely relied on personal knowledge of refusers, which could not be validated by project staff. The revised measure includes verbal declinations in the “declined” numerator category and adds an “unknown” category to provide facilities with actionable data on unvaccinated HCP who may not have purposely declined.
Our study corroborates published literature suggesting that while the true incidence of medical contraindications to inactivated influenza vaccine among HCP is below 2%,31 reported rates are sometimes higher,32 and medical exemptions are requested for conditions that are not true contraindications. Ours is not the first study to identify physician recommendation against influenza vaccination in pregnant women as a reason for HCP to decline influenza vaccination,30,31 despite ACIP recommendations.1 While the CDC-sponsored measure allows for self-report of contraindications, reported medical exemptions that reflect misinformation about the vaccine threaten the accuracy of reported data and the integrity of the vaccination program. Moreover, the increased risk for severe complications from influenza infection among pregnant women, persons with chronic medical conditions, and the elderly1 underscores the value in monitoring contraindication rates to enable facilities to recognize unusually high contraindication rates and target educational efforts appropriately.
This study has limitations. Facilities surveyed may not be representative of all US healthcare facilities. However, usability data gathered in the pilot test indicated that the measure is appropriate for both inpatient and outpatient settings.18 We did not interview HCP or access medical records; such assessments were outside the scope of our evaluation. Exclusion of certain types of HCP by facilities in reporting data elements limited our ability to analyze reliability and validity in these groups. Future work is needed to evaluate best practices for recording and reporting vaccination of nonemployee HCP, including contract staff.
The revised CDC measure (NQF 0431) received full endorsement from the NQF in May 2012.16 While all staff present in a healthcare facility may not be included in the current version of the measure, vaccination is recommended for all HCP.1 As outlined in the NQF’s National Voluntary Consensus Standards for Influenza and Pneumococcal Immunizations, “[Measure] specifications are not intended to replace current guidelines; rather, they are intended to capture the most important aspects of the guidelines that can be feasibly measured.”33(p6) According to the recently published Centers for Medicaid and Medicare Services (CMS) final rule, facilities participating in the CMS’s Hospital Inpatient Quality Reporting program will be required to report annual influenza vaccination coverage using NQF measure 0431 to receive payment.34 Data reporting using the CDC’s National Healthcare Safety Network began January 1, 2013, and facility-level data will be made publicly available at http://www.medicare.gov/hospitalcompare/.10
The results presented here guided revisions to the CDC-sponsored measure of HCP influenza vaccination coverage to produce facility-level data comparable across a variety of healthcare settings to benchmark facilities, improve rates of HCP influenza vaccination, and track progress toward achieving the objective of 90% coverage.7
We thank staff from the California Department of Public Health (Dr Robert Harrison, Dr Jon Rosenberg, Patricia McLendon, and Erica Boston), the New Mexico Department of Health (Lisa Bowdey and Dr Joan Baumbach), the New York City Department of Health and Mental Hygiene (Edward Wake and Dr Jane Zucker), and the University of Pittsburgh Medical Center for assisting with the implementation of the pilot test and reliability study. We acknowledge the members of the pilot project’s Steering Committee and the Delphi panel of experts for guiding the pilot test and informing the specifications of the final endorsed measure. The pilot test was determined to be public health nonresearch by the Centers for Disease Control and Prevention’s human subject office and local jurisdictions, as appropriate.
Financial support. This work was funded by the US Centers for Disease Control and Prevention.
Potential conflicts of interest. All authors report no conflicts of interest relevant to this article. All authors submitted the ICMJE Form for Disclosure of Potential Conflicts of Interest, and the conflicts that the editors consider relevant to this article are disclosed here.
|Distribution of answers by facility staff (n = 90)|
|Case study scenario by correct numerator category||Vaccinated at facility||Vaccinated elsewhere||Medically contraindicated||Declineda||None of the above||Don’t know||% with correct answer|
|Vaccinated at facility|
|Employee vaccinated at facility’s employee health services||86||4||0||0||0||0||95.6|
|Employee submits documentation of vaccine received from provider in August 2010||3||78||0||1||3||3||88.6|
|Physician received vaccination from provider but does not provide documentation||1||68||0||3||13||5||75.6|
|Verbal report of anaphylactic reaction to egg||0||0||79||3||5||2||88.8|
|Volunteer’s vaccination declination form lists history of fever following previous flu vaccination||0||0||9||75||2||4||83.3|
|None of the above|
|Refused vaccination at facility and provided no other information||0||0||0||36||44||9||49.4|
|Deferred vaccine all season||0||0||0||56||30||4||33.3|
|Verbal declination because wants to build natural immunity||0||0||0||74||10||3||11.5|
|Verbal declination because of history of redness and swelling at site||0||0||9||71||9||1||10.0|
|Distribution of answers by facility staff (n = 90)|
|Case study scenario by correct denominator category||Employee||Credentialed nonemployee||Other nonemployee||Not included||Don’t know||% with correct answer|
|Nurse employed part time who also works at another facility||87||0||1||0||1||97.8|
|Part-time billing employees with rare patient contact||83||2||3||1||1||92.2|
|Full-time employee on maternity during October and November||71||0||0||17||2||78.9|
|Janitors with rare patient interaction||68||2||9||5||4||77.3|
|Medical residents on facility payroll||64||7||5||7||6||71.9|
|Facility’s owners who are physicians and work part time||43||31||6||3||7||47.8a|
|Independent physicians credentialed by facility||8||68||2||8||2||77.3|
|Physicians credentialed by contracting agency who occasionally admit to facility||3||51||8||15||12||66.3b|
|Allied health staff employed by national company but credentialed by facility||7||58||9||9||5||65.9|
|Nurse practitioners not credentialed by facility||2||25||32||18||13||63.3b|
|Physicians credentialed by facility who have not admitted patients to facility in 12 monthsc||1||40||0||45||4||44.4|
|Construction workers employed by contractor and working at facility October 2010 through February 2011||2||3||59||22||4||65.6|
|Nursing students on 4-week rotations at facility between October 2010 and March 2011||2||5||59||20||4||65.6|
|Distribution of 9 expert ratingsa regarding accuracy of proposed numerator elements for employeesb|
|Proposed numerator element||Inaccurate||Neutral||Accurate|
|Received an influenza vaccination at the institution from the time vaccine became available through March 31||0||0||9|
|Received an influenza vaccination elsewhere from the time vaccine became available through March 31|
|Self-report of vaccination||2||4||3|
|Vaccine not received because of a medical contraindication/condition|
|Self-report of medical contraindication/condition||1||5||3|
|Documented medical contraindication/condition||0||1||8|
|Self-report to be accepted only if occupational health records attesting to the contraindication are not available||1||2||6|
|Documented declination for reasons other than medical contraindications, including religious exemptions||1||0||8|
|Distribution of 9 expert ratingsa regarding accuracy of proposed denominator elements and time period|
|Worked 1 or more daysb||Worked 30 or more daysb|
|Proposed denominator element||Inaccurate||Neutral||Accurate||Inaccurate||Neutral||Accurate|
|Credentialed nonemployees (eg, physicians, nurse practitioners, physician assistants)||4||1||4||2||0||7|
|Contracted custodial/maintenance workers||4||1||4||3||1||5|
|Contracted cafeteria workers||4||0||5||3||1||5|
|Contracted construction workers||5||1||3||3||2||4|
|Vendors (eg, pharmaceutical, medical device)||4||1||4||3||0||6|
|Allied health staff (eg, physical/occupational/speech/respiratory therapists, lab/radiology technicians)||3||1||5||2||1||6|
|Anyone required to have tuberculosis skin test||4||0||5||2||0||7|
|Except for visitors, all nonemployees||3||3||3||3||1||5|
- 1. Advisory Committee on Immunization Practices, Centers for Disease Control and Prevention. Immunization of health-care personnel: recommendations of the Advisory Committee on Immunization Practices (ACIP). MMWR Recomm Rep 2011;60(RR-7):1–45.
- 2. Carman WF, Elder AG, Wallace LA, et al. Effects of influenza vaccination of health-care workers on mortality of elderly people in long-term care: a randomised controlled trial. Lancet 2000;355(9198):93–97.
- 3. Wilde JA, McMillan JA, Serwint J, Butta J, O’Riordan MA, Steinhoff MC. Effectiveness of influenza vaccine in health care professionals: a randomized trial. JAMA 1999;281(10):908–913.
- 4. Salgado CD, Giannetta ET, Hayden FG, Farr BM. Preventing nosocomial influenza by improving the vaccine acceptance rate of clinicians. Infect Control Hosp Epidemiol 2004;25(11):923–928.
- 5. Potter J, Stott DJ, Roberts MA, et al. Influenza vaccination of health care workers in long-term-care hospitals reduces the mortality of elderly patients. J Infect Dis 1997;175(1):1–6.
- 6. Influenza vaccination coverage among healthcare personnel—United States, 2010–11 influenza season. MMWR Morb Mortal Wkly Rep 2011;60(32):1073–1077.
- 7. US Department of Health and Human Services. Healthy People 2020 Summary of Objectives: Immunization and Infectious Diseases. http://www.healthypeople.gov/2020/topicsobjectives2020/pdfs/Immunization.pdf. Accessed May 8, 2012.
- 8. Talbot TR, Babcock H, Caplan AL, et al. Revised SHEA position paper: influenza vaccination of healthcare personnel. Infect Control Hosp Epidemiol 2010;31(10):987–995.
- 10. National action plan to prevent healthcare-associated infections: roadmap to elimination; influenza vaccination of healthcare personnel. US Department of Health and Human Services website. http://www.hhs.gov/ash/initiatives/hai/hcpflu.html. Accessed July 25, 2012.
- 11. Dash GP, Fauerbach L, Pfeiffer J, et al. APIC position paper: improving health care worker influenza immunization rates. Am J Infect Control 2004;32(3):123–125.
- 12. Pearson ML, Bridges CB, Harper SA; Healthcare Infection Control Practices Advisory Committee; Advisory Committee on Immunization Practices. Influenza vaccination of health-care personnel: recommendations of the Healthcare Infection Control Practices Advisory Committee (HICPAC) and the Advisory Committee on Immunization Practices (ACIP). MMWR Recomm Rep 2006;55(RR-2):1–16.
- 13. New infection control requirement for offering influenza vaccination to staff and licensed independent practitioners. Jt Comm Perspect 2006;26(6):10–11.
- 14. Joint Commission revises influenza vaccination standard: applicability extending to all accreditation programs in 2012. Jt Comm Perspect 2011;31(12):4–5.
- 15. Lindley MC, Yonek J, Ahmed F, Perz JF, Torres GW. Measurement of influenza vaccination coverage among healthcare personnel in US hospitals. Infect Control Hosp Epidemiol 2009;30(12):1150–1157.
- 16. Influenza vaccination coverage among healthcare personnel. NQF 0431. National Quality Forum website. http://www.qualityforum.org/MeasureDetails.aspx?actid=0&SubmissionId=511#k=0431. Accessed July 2, 2012.
- 17. MacCannell T, Shugart A, Schneider A, et al. A qualitative assessment of a performance measure for reporting influenza vaccination rates among healthcare personnel. Infect Control Hosp Epidemiol 2012;33(9):945–948.
- 18. Lindley MC. Pilot-testing a standardized measure for reporting healthcare personnel influenza vaccination, 2010–2011. Paper presented at: National Influenza Vaccine Summit; May 9, 2011; San Diego.
- 19. Sim J, Wright CC. The kappa statistic in reliability studies: use, interpretation, and sample size requirements. Phys Ther 2005;85(3):257–268.
- 20. Miller-Keane, Miller BF, O’Toole M. Miller-Keane Encyclopedia and Dictionary of Medicine, Nursing and Allied Health. 7th ed. Philadelphia: Elsevier Health Sciences, 2003.
- 21. Fitch K, Bernstein S, Aguilar N, et al. The RAND/UCLA Appropriateness Method User’s Manual. Santa Monica, CA: Rand, 2001.
- 22. Fiore AE, Uyeki TM, Broder K, et al. Prevention and control of influenza with vaccines: recommendations of the Advisory Committee on Immunization Practices (ACIP), 2010. MMWR Recomm Rep 2010;59(RR-8):1–62.
- 23. Melia M, O’Neill S, Calderon S, et al. Development of a flexible, computerized database to prioritize, record, and report influenza vaccination rates for healthcare personnel. Infect Control Hosp Epidemiol 2009;30(4):361–369.
- 24. Russell ML, Henderson EA. The measurement of influenza vaccine coverage among health care workers. Am J Infect Control 2003;31(8):457–461.
- 25. Miller BL, Ahmed F, Lindley MC, Wortley PM. Institutional requirements for influenza vaccination of healthcare personnel: results from a nationally representative survey of acute care hospitals—United States, 2011. Clin Infect Dis 2011;53(11):1051–1059.
- 26. Talbot TR. Do declination statements increase health care worker influenza vaccination rates? Clin Infect Dis 2009;49(5):773–779.
- 27. Polgreen PM, Septimus EJ, Parry MF, et al. Relationship of influenza vaccination declination statements and influenza vaccination rates for healthcare workers in 22 US hospitals. Infect Control Hosp Epidemiol 2008;29(7):675–677.
- 28. Polgreen PM, Polgreen LA, Evans T, Helms C. A statewide system for improving influenza vaccination rates in hospital employees. Infect Control Hosp Epidemiol 2009;30(5):474–478.
- 29. Polgreen PM, Chen Y, Beekmann S, et al. Elements of influenza vaccination programs that predict higher vaccination rates: results of an emerging infections network survey. Clin Infect Dis 2008;46(1):14–19.
- 30. Ribner BS, Hall C, Steinberg JP, et al. Use of a mandatory declination form in a program for influenza vaccination of healthcare workers. Infect Control Hosp Epidemiol 2008;29(4):302–308.
- 31. Babcock HM, Gemeinhart N, Jones M, Dunagan WC, Woeltje KF. Mandatory influenza vaccination of health care workers: translating policy to practice. Clin Infect Dis 2010;50(4):459–464.
- 32. Wu SS, Yang P, Li HY, Ma CN, Zhang Y, Wang QY. The coverage rate and obstructive factors of influenza vaccine inoculation among residents aged above 18 years in Beijing from 2007 to 2010. Zhonghua Yu Fang Yi Xue Za Zhi 2011;45(12):1077–1081.
- 33. National Voluntary Consensus Standards for Influenza and Pneumococcal Immunizations: A Consensus Report. Washington, DC: National Quality Forum, 2008.
- 34. Centers for Medicare and Medicaid Services. Medicare program; hospital inpatient prospective payment systems for acute care hospitals and the long-term care hospital prospective payment system and FY 2012 rates; hospitals’ FTE resident caps for graduate medical education payment. Final rules. Fed Regist 2011;76(160):51476–51846.