70.3 Basics of Performance Measurement for Quality ...

Funded by HRSA HIV/AIDS Bureau

The Basics of Performance Measurement for Quality

Improvement

Nancy Showers, DSW

888-NQC-QI-TANationalQualityCenter.org

2 National Quality Center (NQC)

Learning Objectives

• At the conclusion of the workshop, participants will:

Understand the balance between performance measurement and quality improvement activities and be able to identify and implement key performance measurement steps

Develop clinical and non-clinical performance indicators and data collection strategies.

Know how to access existing resources: clinical and non-

clinical indicators, HIVQUAL software, Performance Measurement Guide, etc.


Infrastructure

Linking Performance Measurement and Quality Improvement


How to Go in Circles

Measure

CQI

ChangeCQI


Measuring performanceSteps to measure performance:1. Select a quality of care indicator.2. Define the measurement population.3. Define the measure.4. Create a data collection plan.5. Develop data collection tools and

instructions.6. Train medical record abstractors.7. Run a pilot test.8. Collect data.9. Analyze data and plan QI activities.10. Display and distribute data.11. Evaluate the measurement process and determine how to improve it.

http://images.google.com/imgres?imgurl=http://www.ebinteractive.com/Legaldocs/CD-DVD%2520Label%2520small.jpg&imgrefurl=http://www.ebinteractive.com/legaldocs.html&h=1373&w=1373&sz=78&tbnid=zZe_6RVuqaEJ:&tbnh=150&tbnw=150&hl=en&start=3&prev=/images%3Fq%3Dcd%26svnum%3D10%26hl%3Den%26lr%3D


Quiz 170% of eligible patients had a PPD read in the past year. This

performance is in the

A. top 10% of programsB. top 25% of programsC. upper 50% of programsD. lower 50% of programs

80% of patients with CD4 count < 200 were on PCP prophylaxis. This performance is in the


Based on 2005 National HIVQUAL Data, representing 103 (??) HIV programs


Basics of Performance Measurement

• Why measure?• What to measure?• When to measure?• How to measure?• Strategic planning for measurement


Why Measure?


Reasons to Measure

• Separates what you think is happening from what really is happening

• Establishes a baseline: It’s ok to start out with low scores!

• Determines whether changes actually lead to improvements

• Avoids slippage


Reasons to Measure (cont.)

• Ongoing / periodic monitoring identifies problems as they emerge

• Measurement allows for comparison across sites, programs, EMAs and states

• The Ryan White Reauthorization Act of 2000 mandates performance measurement

• The HIV/AIDS Bureau places strong emphasis on quality management


Quiz 2

Why should you measure annual PAP performance?

A. All women should have a yearly PAPB. Cervical cancer is more common in

women with HIV C. This is a “best practices” indicator for

HIV careBased on 2005 National HIVQUAL Data, representing 103 (??) HIV programs


What to Measure


What is a Quality Indicator?

An indicator is a surrogate for direct measurement of quality

• Quality cannot be measured directly• An indicator is a measure thought to

contribute to or reflect quality


Dimensions of Quality

Technical Quality Provider Perception of Quality of HIV Care

Experience QualityConsumer

Perception of Quality of HIV Care

Leonard Berry, Texas A&M University, IHI conference 2001


Topics for Outcome Indicators• Patient Health Status

Intermediate outcomes like immune and virological status Survival Symptoms Co-Infection Disease progression Disability Subjective health status Hospital and ER visits

• Patient Satisfaction• Public Health Outcomes

Retention in Care Access to Care


Processes

Actions that providers take in delivering care. Procedures for achieving the best outcomes

Patient’s subjective sense of providers’ skills, including interpersonal skills

Wu, et. Al. “Quality-of-Care Indicators for HIV/AIDS”. Disease Management Health Outcomes. June, 2000.


Process Indicator Topic Areas

• Medical processes• Case management processes• Clinic / agency / EMA / state processes• Patient utilization of care

Underutilization Overutilization Misutilization

• State, EMA, network common processes• Coordination of care processes


A Few Process Questions…

• Is necessary care provided? Is the care provided necessary?

• Are services provided in a timely fashion? Are those services easily available?

• Is care provided in the most efficient manner? • Is the care provided as intended?• Are customers satisfied with how services are

provided? Are there patterns of complaints and concerns?


HIVQUAL Indicator Topics – Adults and Adolescents

• ARV THERAPY MANAGEMENT• HIV MONITORING (CD4 and VL

testing)• HIV SPECIALIST CARE• ANTIRETROVIRAL THERAPY

MEDICATION• TREATMENT EDUCATION• ADHERENCE TO ARV

THERAPY• PCP PROPHYLAXIS• MAC PROPHYLAXIS• GYNECOLOGY EXAMS

• TUBERCULOSIS SCREENING (PPD)

• SYPHILIS SCREENING• HEPATITIS C SCREENING• VACCINATION• SUBSTANCE USE• MENTAL HEALTH CARE• DENTAL CARE• OPHTHALMOLOGICAL CARE• LIPID SCREENING• BASIC PATIENT EDUCATION


HIVQUAL Indicator Topics – Pediatric

• ARV THERAPY MANAGEMENT

• HIV MONITORING (CD4 and VL testing)

• HIV PEDIATRIC SPECIALIST CARE

• ANTIRETROVIRAL THERAPY

• MEDICATION ADHERENCE

• PCP PROPHYLAXIS

• MAC PROPHYLAXIS

• ROUTINE VACCINATIONS

• NEURODEVELOPMENTAL ASSESSMENTS

• MULTIDISCIPLINARY CARE PLAN


HIVQUAL Indicator Topics – Case Management

• Complete psychosocial assessment

• Patient knowledge screening

• Treatment adherence assessment

• Service care plan & coordination of care

• Access and Continuity

• Self-management: Client participation in care planning


What is a Good Indicator?

• Relevance-How Important is the Indicator? Does the indicator affect a lot of people or

programs? Does the indicator have a great impact on the

programs or patients/clients in your EMA, state, network or clinic?

• Measurability Can the indicator realistically and efficiently be

measured given finite resources?


What is a Good Indicator? (Cont’d)

• Accuracy Is the indicator based on accepted guidelines or

developed through formal group-decision making methods?

• Improvability Can the performance rate associated with the

indicator realistically be improved given the limitations of your services and population?


Considerations for Selecting an Indicator

• Best practices vs. minimum standards Ex. Mental health screening every visit vs. annual Ophthalmology exam for CD4 < 200 vs. <50

• Overall performance vs. PDSA Ex. Annual PAP rate vs. all patients needing a PAP got one

in the past two weeks

• Balance of measures Ex. Reduction of ER visits vs. increase of post ER

hospitalizations


Quiz 31. 75% of patients had a PAP in the past year. This

performance is in the A. top 10% of programsB. top 25% of programsC. upper 50% of programsD. lower 50% of programs

2. 80% of eligible patients had a medical visit at least every 4 months last year. This performance is in the




Eligible Patients/Sample(pts seen in the clinic in the last 12 months)

Denominator(pts with CD4 counts < 200)

Numerator(pts with prescribed PCP prophylactic therapy)


Develop Criteria to Define Your Eligible Population

• Location: all sites, or only some?• Gender: men, women, or both?• Age: any limits?• Client conditions: all HIV-infected clients, or

only those with a specific diagnosis?• Treatment status?• Number of visits?

Sampling Records


Eligibility definition by the National HIVQUAL Project

• HIV+ ambulatory patients, who have had at least 2

primary care visits in last 12 months; with at least 1

primary care visit within the last 6 months.

Sampling Records


Indicator Definition

Eligibility (Patients over 12 years of age seen in the clinic in the last 12 months)Numerator (# of pts over 12 with CD4 < 200 cells/mm3 on prescribed PCP prophylactic therapy)

Denominator (# of patients over 12 with CD4 < 200 cells/mm3)


Steps in Defining an Indicator

1. Specify the reasonable requirement for careEx: “ New patients should be seen within 5

business days of referral/request for care”2. Define the indicator

Ex: “% of new patients seen within 5 business days of referral/request for care”


Steps in Defining the Indicator (Cont’d)

3. Set the denominator: Specifically which patients should be getting the care? (What is the eligible population?)

Ex. The number of patients referred or requesting care in the past 3 months

4. Set the numerator. Which patients got the care?:

Ex. The number of patients referred or requesting care in the last 3 months

who were seen within 5 business days5. Measurement: Divide the numerator by the

denominator to get the performance percentage


Indicator Definition Tips

1. Base the indicator on guidelines and standards of care when possible

2. Be inclusive (of staff and consumers) when developing an indicator to create ownership

3. Be clear in terms of patient / program characteristics (gender, age, patient condition, provider type, etc.)

4. Set specific time-frames in indicator definitions


Quiz 4

1. 70% of eligible patients had a dental exam in the past year. This performance is in the


2. 70% of eligible ARV patients had a lipid profile in the last year. This performance is in the




Indicator Exercise

• Define an indicator for measuring no show performance in a primary medical clinic

• Define an indicator for measuring effectiveness of case management care plans in a Title IV program

• Define an internal indicator for measuring the effectiveness of a Title I Quality Management program

• Define an indicator for measuring underutilization of care across a state

• Define an indicator for measuring integrity of a clinical data system


Indicator Development Exercise


How to Measure


Create a Plan

• Decide on a sampling plan (sample size, eligible records, draw a random sample)

• Develop data collection tools and instructions• Train data abstractors• Run pilot test (adjust after a few records)• Inform other staff of the measurement process• Check for data accuracy • Remain available for guidance• Make a plan for display and distribution of data


Using a Random Sample

• Use a random sample if the entire population can’t easily be measured

• “Random selection” means that each record has an equal chance of being included in the sample.

• The easiest way to select records randomly is to find a random number table and pull each record in the random sequence.


Over-sampling

• Over-sampling may be done to account for records that turn out to be ineligible or are unavailable for review.

• The HIVQUAL sample size chart includes some “over-sampling,” or pulling “extra” charts.


Construct your Sample

1. Identify Eligible Patients• Review all records for eligibility. Eligibility for

HIVQUAL review is defined as all HIV+ patients who meet the following visit criteria: At least two medical visits during the study period (i.e.,

1/1/2005 through 12/31/2005) including At least one medical visit during the last six months of the

study period (i.e., 7/1/2005 through 12/31/2005).• Patients who died prior to the end of the review

period are still eligible if the above conditions are met.

Sampling Records


Construct your Sample (cont.)

2. Identify the Number of Eligible Male and Female Records

• Separate the list of eligibles into two lists, one list only for males and one list for females.

• Count the number of eligible male and female records.

• Sequentially order these two lists, either alphabetically, by medical record, or client number.

Sampling Records



3. Determine the minimum number of male and female records to be reviewed

• Using the number of eligible female patients, determine the minimum number of female records needed from the Minimum Sample Table.

• Determine the minimum number of male records needed by subtracting the minimum female records from the total minimum records.

Sampling Records


Example

A facility has 80 eligible female patients and 120 eligible male patients. Therefore the total eligible population is 200 patients.

(1) Based on the total sample table, the minimum total number of records needed is 75.

(2) Based on the female sample table, the minimum number of female records needed is 46.

(3) Therefore, the minimum number of male patients needed is 29 (75-46 = 29).

Sampling Records



4. Select Charts Randomly for Review Obtain a random number set equal to the number

of females needed Obtain a random number set equal to the number

of males needed Apply the random number sets to the lists of

eligible males and females using the sequence you created when numbering your lists.

Sampling Records


Resources to Randomize Your Records

• “Measuring Clinical Performance: A Guide for HIV Health Care Providers” (includes random number tables)

• A useful website for the generation of random numbers is www.randomizer.org

• Common spreadsheet programs, such as MS Excel

Sampling Records


Quiz 5

1. 80% of patients with CD4 < 200 were on HAART. This performance is in the


2. 80% of eligible patients with VL > 100,000 were on HAART. This performance is in the




Strategies

• Measure multiple indicators simultaneously to identify priorities for improvement projects

• Focus on one or more areas of agreed upon needed improvement.

• Measure an entire population• Measure a random sample


Strategies Depend on Resources

• Data systems enhance capability More indicators can be measured Indicators can be measured more often Entire populations can be measured Alerts, custom reports help manage care

• Personnel resources Person power for chart reviews, logs, other

means of measurement Expertise in electronic / manual measurement


Collect “Just enough” Data

• The goal is to improve care, not prove a new theorem

• 100% is not needed• Maximal power is not needed• In most cases, a straightforward sample will

do just fine


Tips for the Electronic Era• Strategically plan for the electronic era

Add data needed for denominators and numerators Plan for needed queries and reports For EMRs, focus heavily on adding HIV specific data and

aggregate reports• Don’t defer improvement projects while implementing

electronic plans. • Don’t expect your electronic system to entirely replace the

need for manual systems• Use the electronic system to manage as well as measure care


Careware Potential for Measuring Outcomes: Examples

• If you add mental health scores to the screening module, you can begin to look at interventions that improve scores as well as change impact on things like adherence and viral loads

• If you add self-management interventions, you can look at intervention impact on adherence with appointments, adherence with meds, and lab values


Encounter Report-Screenings


Quiz 6

1. 90% of ARV patients had a quantitative adherence assessment every 4 months This performance is in


2. 90% of patients had a mental health screen last year This performance is in




When to Measure


Frequency

• You don’t need to measure everything all of the time. You can sample a short period of time and extrapolate the results

• Balance the frequency of measurement against the cost in resources

• If limited resources, measure areas of concern more frequently, others less frequently

• Consider the audience. How will frequency best assist in setting priorities and generating change?


Quiz 7

1. 80% of patients had a CD4 every four months. This performance is in the


2. 80% of patients had a viral load every four month. This performance is in the




National HIVQUAL Data Reports

• Show national trends based on self-reported data by participating HIVQUAL grantees

• Provide an opportunity to compare program performance with national data to highlight areas of opportunity


The HIVQUAL Project 2005 Performance Data

Title III and Title IV Programs


Questions for Data Follow-up• What are the results for key indicators?• What are the major findings based on the

generated data reports and your data analysis? What is the frequency of patients /

programs not getting care? What is the impact of not getting the care? How does the performance compare with

benchmark data? What is the feasibility of improving the

care?


Key Questions for Data Follow Up (Cont’d)

• How can you best share the data results with your key stakeholders (HIV QI committee, facility-wide QI committee, staff, consumers, CAB, EMA programs, state providers,etc.)?

• How do you generate ownership among staff and consumers?

• Are you considering initiating a QI project to address the data findings? Who will be responsible and what are the next steps?


Evaluate

• What lessons did you learn about the measurement process? What went particularly well? What went less well?

• What changes will you make next time?• Have you kept a balance between

measurement and focus on change?


Resources for Performance Measurement in HIV Care


Resources: Indicator Definitions

• New York State Department of Health AIDS Institute - www.HIVQUAL.org

• New York State Department of Health AIDS Institute – NationalQualityCenter.org

• National Quality Measures Clearinghouse - www.qualitymeasures.ahrq.gov


Resources: HIV Guidelines

• New York State DOH AIDS Institute - www.hivguidelines.org

• HIV/AIDS Treatment Information Service www.aidsinfo.nih.gov

• Johns Hopkins AIDS Service www.hopkins-aids.edu

• Infectious Diseases Society of America www.hivma.org/HIV/tocCEN.htm


HIVQUAL3 Software

HIVQUAL3: • software package in Microsoft Access• incorporates HIV clinical indicators to measure HIV

care • provides immediate reporting of performance data

for use in internal quality programs• external reporting for comparisons with other

participating HIVQUAL programs


HIVQUAL3 Software


Integration of HIVQUAL Software into CAREWare

• Objective: To fully integrate HIVQUAL indicators and reporting features into HRSA’s CAREWare software.

• Benefits: Create reports through CAREWare that can be submitted to

HIVQUAL Allow for single data entry Produce data reports for the entire clinic population instead

of a retrospective sample of patients HIVQUAL indicators will be available to all CAREWare users

(not just HIVQUAL participants) Allow for CAREWare web-based data entry


National Quality Center (NQC)NYSDOH AIDS Institute90 Church Street—13th FloorNew York, NY 10007-2919888-NQC-QI-TAfax 212 [email protected]

NQC Contact Information


CQICQI HeavenHeaven

Measure

ChangeCQICQI

On Our Way to…

70.3 Basics of Performance Measurement for Quality ...

Business

Transcript of 70.3 Basics of Performance Measurement for Quality ...