Developing a Valid Surveillance Design

(Or…Is there life after mandatory reporting?)

Ona MontgomeryOna MontgomeryAmarillo VA Health Care SystemAmarillo VA Health Care System

Proposed Initial Mandatory Reporting Components CL-BSICL-BSI

NICU - birthweight stratifiedNICU - birthweight stratified ICU – stratified by type of ICUICU – stratified by type of ICU Special Care Units – stratified by type of unitSpecial Care Units – stratified by type of unit

SSI SSI CMS procedures CMS procedures Stratified by NHSN indexStratified by NHSN index ASC and general hospitalsASC and general hospitals

HCA-RSV HCA-RSV Pedi hospitalsPedi hospitals General hospitals with pedi unitsGeneral hospitals with pedi units

Other Surveillance Drivers

Regulations and requirements Regulations and requirements JCAHOJCAHO CMSCMS Corporate mandatesCorporate mandates

Internal strategic plansInternal strategic plans

Single acute care facility or part of larger Single acute care facility or part of larger system?system?

How do we develop a scientifically valid and rational surveillance design when so many aspects seem out of our control?

(As well as so much competition for available personnel and other finite resources)

Approach with an epidemiologic perspective.

Recommended Practices for Surveillance

1.1. Assess the populationAssess the population2.2. Select the outcome or process to surveySelect the outcome or process to survey3.3. Apply surveillance definitionsApply surveillance definitions4.4. Collect surveillance dataCollect surveillance data5.5. Calculate rates & analyze surveillance Calculate rates & analyze surveillance

findingsfindings6.6. Apply risk stratification methodsApply risk stratification methods7.7. Report and use surveillance informationReport and use surveillance information

APIC Surveillance Initiative Working Group, 1998 (Under revision)

Key Epidemiologic Principle of SurveillanceFocus on Focus on

populations or populations or individuals at risk individuals at risk for infection.for infection.

Organizational population assessment should be comprehensive

Healthcare VenuesHealthcare Venues Acute careAcute care ICUsICUs LTCLTC RehabRehab LTACLTAC AmbulatoryAmbulatory Home careHome care

Patient PopulationsPatient Populations ElderlyElderly Trauma Trauma Diabetic Diabetic Oncology Oncology DialysisDialysis Spinal cord injurySpinal cord injury Other Important risk Other Important risk

populationspopulations

Epidemiologic Principle

Focus on priority problems/improvement Focus on priority problems/improvement objectivesobjectives Greatest potential for improvement (high Greatest potential for improvement (high

volume, high risk, high intervention impact) volume, high risk, high intervention impact) Significant outcomeSignificant outcome Organizational objectives Organizational objectives Cost-effectivenessCost-effectiveness Risk managementRisk management Existence of standards and mandatesExistence of standards and mandates

Surveillance Practice 1: Assess the Population

Obtain data to describe / understand Obtain data to describe / understand YOUR patient population YOUR patient population

Establish priorities for surveillance Establish priorities for surveillance through your annual risk assessmentthrough your annual risk assessment

Which patients or staff in your Which patients or staff in your organization are at increased risk for organization are at increased risk for infection? infection?

Surveillance Practice 2: Select Outcome or Process to SurveyConsider...Consider...• Relative frequency of the event (high volume)Relative frequency of the event (high volume)• Cost or impact of a specific negative outcome (high Cost or impact of a specific negative outcome (high

risk)risk)• Preventability (Preventability (high intervention impact)high intervention impact)• Cost-benefit Cost-benefit • Customer needs (e.g. priorities set by facility plan, Customer needs (e.g. priorities set by facility plan,

regulatory compliance)regulatory compliance)• Organizational mission / strategic goalsOrganizational mission / strategic goals• Available resources (administrative commitment) Available resources (administrative commitment)

Cost-effectivenessCost-effectiveness

The Surveillance Process

Assessing populations for riskAssessing populations for risk Choosing problem or event to be studiedChoosing problem or event to be studied Often done simultaneouslyOften done simultaneously

Factor in mandatory reporting componentsFactor in mandatory reporting components

Outcome and Process Measures OutcomesOutcomes

InfectionsInfections EventsEvents

ProcessesProcesses Known to be related to outcomeKnown to be related to outcome Key patient care stepsKey patient care steps BundlesBundles

Search of literature, opinions of experts, or Search of literature, opinions of experts, or practice standards may point to appropriate practice standards may point to appropriate process indicatorsprocess indicators

Process Measures - Examples Prophylactic antibiotic timing in ambulatory surgeryProphylactic antibiotic timing in ambulatory surgery Antibiotic selection in treatment of outpatient Antibiotic selection in treatment of outpatient

pneumoniapneumonia Immunization rates in all settings - staff and patientsImmunization rates in all settings - staff and patients Adherence to disinfection processes in dialysis centerAdherence to disinfection processes in dialysis center Compliance with bundles prevent ventilator pneumoniaCompliance with bundles prevent ventilator pneumonia Device utilization ratios - with or without device-Device utilization ratios - with or without device-

associated infection rates in home care, LTC, ICUsassociated infection rates in home care, LTC, ICUs PPD skin testing compliance of patients and/or staff - in PPD skin testing compliance of patients and/or staff - in

conjunction with outcome measure of PPD skin test conjunction with outcome measure of PPD skin test conversionsconversions

Surveillance Practice 4:Collect Surveillance Data

Data collection methods must be Data collection methods must be systematicsystematic

Utilize available information systems Utilize available information systems (including electronic data)(including electronic data)

The team approach is critical - The team approach is critical - negotiate for assistancenegotiate for assistance

Provide surveillance personnel with Provide surveillance personnel with standardized trainingstandardized training

Intensity of surveillance must be Intensity of surveillance must be maintained over time!maintained over time!

Surveillance Practice 5: Calculate Rates and Analyze Findings Plan for data analysis. Plan for data analysis. Express surveillance information in numerical Express surveillance information in numerical

terms (I.e. incidence density, rates) terms (I.e. incidence density, rates) Turn surveillance “data” into “information”Turn surveillance “data” into “information” Use statistical probability methods to determine Use statistical probability methods to determine

whether observed differences in rates are whether observed differences in rates are meaningful (i.e. significance testing; p values)meaningful (i.e. significance testing; p values)

Provide analysis and interpretation Provide analysis and interpretation

Use scientifically sound, well understood methods in data collection and analysis

Calculate rates only when numbers are Calculate rates only when numbers are sufficientsufficient

Numerator analysis and reporting may be Numerator analysis and reporting may be appropriate for infrequent events appropriate for infrequent events

Control chart theory has been applied to Control chart theory has been applied to rate- and event-based datarate- and event-based data

0

50

100

150

200

250

300

350

03/0

1/01

06/0

6/01

08/2

1/01

12/2

0/01

05/0

6/02

07/1

8/02

08/2

3/02

10/2

1/02

11/0

5/02

02/1

9/03

04/0

2/03

06/1

0/03

09/0

4/03

04/2

1/04

09/2

7/04

11/1

7/04

12/0

2/04

12/0

8/04

01/1

1/05

03/0

9/05

08/0

9/05

08/1

6/05

08/2

4/05

09/1

4/05

01/0

4/06

01/1

1/06

02/2

2/06

Date of Surgery

# Cas

es b

etw

een

Mean = 86

Post-op PneumoniaTrend Analysis

mean calc based on FY01 - FY04

GOOD

Surveillance Practice 6: Apply Risk Stratification MethodsRisk stratification: subdividing (stratifying) Risk stratification: subdividing (stratifying)

your surveillance population into groups at your surveillance population into groups at similar levels of infection risk prior to similar levels of infection risk prior to performing any analyses or comparisons.performing any analyses or comparisons.

To ensure comparing “apples to apples”

CDC NNIS Risk Index for SSI Surveillance

Patient-specific Risk ScorePatient-specific Risk Score Total 0-3 points Total 0-3 points

Wound class class III or IVWound class class III or IV 1 point1 point ASA score 3, 4, 5ASA score 3, 4, 5 1 point1 point Duration of surgery > cutpointDuration of surgery > cutpoint 1 point 1 point

CDC NNIS Risk Stratification for High Risk Nursery (HRN)

Surveillance Stratification by Birthweight Categories:Stratification by Birthweight Categories:

• </= 500 grams</= 500 grams• 501 - 1000 grams501 - 1000 grams• 1001-1500 grams1001-1500 grams• 1501-2500 grams1501-2500 grams• >2500 grams>2500 grams

Surveillance Practice 7:Report and Use Surveillance Data

Plans for the distribution of Plans for the distribution of surveillance information should be surveillance information should be incorporated into the development incorporated into the development of each surveillance objectiveof each surveillance objective

Provide feedback to Provide feedback to cliniciansclinicians!! Reporting should be timelyReporting should be timely Present surveillance findings using Present surveillance findings using

graphs and easy-to-read tablesgraphs and easy-to-read tables

Present surveillance data in a manner to Present surveillance data in a manner to stimulate ideas for process improvement.stimulate ideas for process improvement.

Perform follow-up surveillance to monitor for Perform follow-up surveillance to monitor for improvement following changes (“close the improvement following changes (“close the loop”).loop”).

“Surveillance without action should be abandoned.”

Compared to what?

Internal changes over timeInternal changes over time External comparisons such as:External comparisons such as:

NNIS/NHSNNNIS/NHSN IHI publicationsIHI publications State reporting dataState reporting data

Pooled means and percentiles of the distribution of device-associated infection rates, by type of

ICU, NNIS, 1/95-6/03

Central line-associated BSI rate** PercentileNo. of Central Line- Pooled 10% 25% 50% 75% 90%

Type of ICU Units Days Mean (median)

Coronary 114 363,976 4.2 0.0 1.9 4.2 5.8 8.4Cardiothoracic 71 598,118 2.9 0.4 1.3 2.2 3.5 4.9Medical 143 975,318 5.7 2.1 3.4 5.0 6.8 9.6Medical-Surgical Major teaching 133 936,223 5.0 2.2 3.0 4.9 6.3 7.7 All others 187 1,295,477 3.7 0.0 1.8 3.3 5.0 6.8Neurosurgical 52 180,581 4.8 0.0 2.5 4.1 6.5 9.0Pediatric 79 428,104 7.3 0.7 3.8 5.9 8.8 11.5Surgical 160 1,267,959 5.2 1.1 2.6 4.7 6.9 9.3Trauma 28 178,179 7.8 2.5 5.2 6.6 10.0 12.3** Number of central line-associated BSI Number of central line-days

X 1000

Pooled means and percentiles of the distribution of device utilization ratios, by type of ICU,

NNIS, 1/95-6/03

Central line utilization** PercentileNo. of Pooled 10% 25% 50% 75% 90%

Type of ICU Units Patient-Days Mean (median)

Coronary 115 1,120,967 0.32 0.13 0.21 0.29 0.42 0.58Cardiothoracic 71 751,547 0.80 0.57 0.70 0.82 0.91 0.95Medical 143 1,905,674 0.51 0.30 0.37 0.52 0.64 0.75Medical-Surgical Major teaching 133 1,688,840 0.55 0.35 0.45 0.55 0.64 0.73 All others 187 2,770,191 0.47 0.25 0.34 0.47 0.57 0.63Neurosurgical 52 401,236 0.45 0.26 0.38 0.49 0.55 0.63Pediatric 82 936,169 0.46 0.20 0.30 0.41 0.53 0.60Surgical 160 1,958,691 0.65 0.44 0.55 0.67 0.76 0.86Trauma 28 280,074 0.64 0.47 0.57 0.65 0.75 0.85** Number of central line-days Number of patient-days

Surgical Prophylaxis (% given < 1 hr. before incision)

0102030405060708090

100

1QFY02

2QFY02

3QFY02

4QFY02

3-Jan 3-Feb 3-Mar

% compliance

Development of your Hospital Surveillance Plan

Plan must be written!Plan must be written! Reflect logical risk assessmentReflect logical risk assessment Address internal and external prioritiesAddress internal and external priorities Describe each surveillance componentDescribe each surveillance component

• PopulationsPopulations• Outcomes of concernOutcomes of concern• Rationale (e.g., risk assessment rating)Rationale (e.g., risk assessment rating)• Measurement/analysisMeasurement/analysis• Interventions and reporting planInterventions and reporting plan

Risk Rating Example

Probability Risk Current Systems

4 Expect it to happen5 Loss of life/

limb/function 5 None in place currently

3 Likely4 Temp loss of

function 4 Poor

2 Maybe3 Prolonged

hospitalization 3 Fair

1 Rare2 Moderate clinical /

financial 2 Good

0 Never1 Minimal clinical /

financial1 Solid system in place

currently

Plan ExamplePriorit

yRisk

Score

Surveillance focus

(Populations affected)

Outcome Goal Intervention Plans Evaluation Methods Progress

1 38 Influenza Prevention and Pandemic Preparedness

(Populations: Acute, long term, & ambulatory care; HCWs)

Influenza prevention in all populations

SEASONAL INFLUENZA1. Pre-season CNE2. Revise NH on-line module3. Staff immunization

strategies4. Prevention/control of NH

outbreaksPANDEMIC PREPAREDNESS- Tests of systems- Supply stockpile

Vaccination acceptance in long term care, ambulatory (including COBCs), inpatients, and staff

Preparedness for local epidemic and/or pandemic (surge capacity)

Detection, prevention and control of nursing home outbreaks

Development of internal pandemic response plan linked to community plan

2 25.5 CL-BSI Risk Reduction

(Populations: Acute, long term & ambulatory care)

Prevention of Central Line associated Blood-stream Infection

1. Expand bundle compliance monitoring to entire facility

2. Revise policies & procedures3. Improve maintenance

process (e.g. pt. education, dressings, showers, securement)

4. Implement CHG dressing change kits

Outcome: Central Line associated Bloodstream Infection in ICU

Process: Compliance with the “bundle” of prevention measures

Outpatient processes and outcomes

Explore feasibility of non-ICU line day collection

3 25 Dialysis (if initiated in

FY07)(Population: Acute

care; HCW)

Prevention of transmission of BBP and other dialysis-related infection

1. Monitoring c/w CDC NHSN system

2. Facilitate IC involvement in P&P development

3. Implement H2O monitoring

Outcome: BSI in Dialysis patients

Process: Compliance with defined procedures

Key Principle

Include clinicians in all steps of Include clinicians in all steps of planningplanning and and implementing surveillance plan.implementing surveillance plan. Improved buy-in and participationImproved buy-in and participation Non-acute settings -- technical staffNon-acute settings -- technical staff

ExamplesExamples ICU - front line staff (bundles)ICU - front line staff (bundles) LTC – CNAs as well as nursesLTC – CNAs as well as nurses Home health - family educatorsHome health - family educators

Key Principle

Report data to clinicians who can influence Report data to clinicians who can influence patient care practices.patient care practices. Important tie between surveillance and clinical Important tie between surveillance and clinical

carecare Reinforces staff involvementReinforces staff involvement Allows completion of surveillance cycleAllows completion of surveillance cycle

Fostering Effective Interventions Use stakeholders and content expertsUse stakeholders and content experts

Effective? Effective? Feasible?Feasible? Barriers?Barriers?

ICPs should strive to facilitate, not dictate ICPs should strive to facilitate, not dictate or implementor implement

Document improvement activities Document improvement activities undertaken by all participantsundertaken by all participants

Follow-up

Continued surveillance for internal Continued surveillance for internal comparisons to measure improvementcomparisons to measure improvement

Outcome monitoring - look for improved Outcome monitoring - look for improved infection ratesinfection rates

Process monitoring - look for improved rates Process monitoring - look for improved rates of compliance - important even without of compliance - important even without corresponding change in outcome ratescorresponding change in outcome rates

“A good surveillance system does not guarantee you will make the right decisions, but it reduces the chances of making the wrong ones.”

Dr. Alexander Langmuir

N Eng J Med 1963;268:182-92.

References and Resources Guidance on Public Reporting of Healthcare-Associated Infections

AJIC Feb. 2005 http://www.cdc.gov/ncidod/dhqp/pdf/hicpac/

PublicReportingGuide.pdf Lee TB, Baker OG, Lee JT, Scheckler WE, et al. Recommended

Practice for Surveillance. American Journal of Infection Control 1998;26:277-288.http://www.apic.org/AM/Template.cfm?

Section=Surveillance_Definitions_Reports_and_Recommendations&Template=/CM/ContentDisplay.cfm&ContentFileID=2710

NHSN Member site (surveillance protocols, etc.):http://www.cdc.gov/ncidod/dhqp/nhsn_members.html

Gaynes R, Richards C, Edwards J, et al. Feeding Back Surveillance Data to Prevent Hospital-acquired Infections. Emerging Infectious Diseases Mar-Apr 2001http://www.cdc.gov/ncidod/eid/vol7no2/pdfs/gaynes.pdf

Haley RW. Surveillance by objective: a new priority-directed approach to the control of nosocomial infections. Am J Infect Control 1985;13:78-89.