1

Using Data for Decision Making in Health Care

“Plot the dots.”

Greg Ogrinc, MD, MS

Dartmouth Medical School

27 June 2009

2

Objectives

Investigate how healthcare systems are measured and the effect of this measurement

Identify how understanding and measuring variation is integral to the model for improvement

Apply the rules of analysis for evaluating data over time from a system

3

Agenda Storytime! (10 min)

P4P with buzzgroups and discussion (10 min)

Evidence, improvement, and measurement (30 min)

Data, group work, and discussion

Introduction to variation and data over time (time ordered data) (40 min)

Data, group work, and discussion

4

Storytime…

5

Percent Normotensive (<140/<90) (Avg=74.39, UCL=83.66, LCL=65.11 for subgroups Jun-07-May-08)

Avg

UCL

LCL

Target

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

May-05 Jun-05 Jul-05 Aug-05 Nov-05 Jan-06 May-06 Jul-06 Sep-06 Oct-06 Jan-07 Mar-07 Jun-07 Jul-07 Sep-07 Nov-07 Jan-08 Mar-08 May-08

Date

% p

atie

nts

with

nor

mal

BP

6

HTN letter

High blood pressure can lead to heart disease and stroke if not well controlled. We review the risks and benefits of changing their blood pressure treatment regimen on a regular basis. Checks of your Blood Pressure show that it is higher than 140/90, so we need to get this under better control to reduce your risk of heart disease and stroke.

You can do the following things to help keep your blood pressure under control:

- Reach or maintain a normal body weight

- Eat a healthy diet without too much salt.

- Eat plenty of fruits and vegetables

- Limit the amount of caffeine

- Include regular physical activity in your schedule

- Do not drink more then 2 ounces of liquor or 2 glasses of beer or wine in one day.

7

HTN letterIn order to help you get better control of your blood

pressure, I recommend the following:

(1) Discontinue HYDROCHLOROTHIAZIDE.

(2) Start CHLORTHALIDONE 100MG each morning. I have made the changes in the computer and the meds will be mailed to you.

(3) Please come to the WHITE MOUNTAIN FIRM for a blood pressure check in 2 weeks. You do not need an appointment.

(4) Stop by the laboratory for blood work on the same day you come for the blood pressure check.

8

P4P in NHSDoran et al, NEJM, 2006

Evaluated first year of P4P for >8000 family practices in England

Measured proportion of patients eligible for whom the indicator was met

Diabetes – % blood pressure <145/<85

Hypothyroid – % blood check within 15 mos

Stroke – % cholesterol < 193

Able to request exceptions for some patients

9

10

P4P in NHSDoran et al, NEJM, 2006

Median level of achievement was 83%

Level of achievement was affected by

Age and socioeconomic of patients

Size of practice, # patients per practitioner, age of practitioner, whether practitioner was trained in the UK

Exception reporting (1% increase in exception reporting increased achievement by 0.6%)

Small number of practices achieved high scores by excluding large number of patients

More common in small practices

11

Buzz Groups Turn to the person next to you

Use the stories you just heard as well as your knowledge about pay for performance (P4P) – news, articles, journals, internet.

1. What are the benefits of this approach to improving quality?

2. What might be some drawbacks to this approach?

12

Evidence-based Improvement

Generalizable Scientific Evidence + Particular

Patient

Measured PerformanceImprovement

Batalden, 2003

13

Evidence-based Improvement

Generalizable Scientific Evidence + Particular

Context

Measured PerformanceImprovement

Batalden, 2003

14

Evidence-based Improvement

Generalizable Scientific evidence + Particular

Context

Measured PerformanceImprovement

• control for context• generalize across contexts• experimental design• statistics

• understand system “particularities”

• learn structures, processes, patterns

• culture and context of changes

• balanced measures• clinical• functional• satisfaction• costs

Batalden, 2003

choosing best plan

executing locally

15

Individual Patient Care versusSystems of Care

Individual patient System

Initial work-up History, physical exam, chart review

Your experience within system, discuss with others

Further work-up

Labs, xrays, ultrasound, functional tests

Process & cause-effect diagrams, outcomes data

Therapy Surgery, medications, watchful waiting

model for improvement, PDSA, root cause analysis

16

“quality improvement”

The combined and unceasing efforts of everyone – health care professionals,

patients and their families, researchers, administrators, payers, planners, educators – to make changes that will lead to better

patient outcome, better system performance, and better professional

development.

Batalden and Davidoff, QSHC, 2007

17

Betterpatient (population)

outcome

Bettersystem

performance

Betterprofessionaldevelopment

Everyone

Batalden and Davidoff, QSHC, 2007

18

Betterpatient (population)

outcome

Bettersystem

performance

Betterprofessionaldevelopment

Everyone

Batalden and Davidoff, QSHC, 2007

20

Better patient /

populationoutcome

Bettersystem

performance

Betterprofessionaldevelopment

Everyone

Measures?Patient knowledge?Variation?Causes?

Competence?Accreditation / certification / licensure?Faculty / curricula development?Professional school admission / selection?Interprofessional cooperation?Joy / creativity / pride?

Hiring / orientation?Supervision?Accountability?Participation /

commitment?Recognition /

reward?

Linking / leadership?Org. development?Governance?Financing?

What might be the foci of inquiry?

Linking / leadership / supervisory development?

Leadership performance review?

Recognition / reward?

Measures?Options / methods?Reliability / failure?Standards?

21

A Model for ImprovementWhat are we trying to accomplish?

How will we know that a change is an improvement?

What change can we make that will result in improvement?

PLAN

DOSTUDY

ACT

Langley et al. , The Improvement Guide, 1996

22

Assumptions about Measurement for QI

Measurement has become “popular” and arises in many venues (e.g., P4P, country comparisons, etc)

Measurement and measures fit into the model for improvementPart of the cycle

Necessary element

Understanding the measures means understanding the level of the system to which the measures relate

How you display and data can have a significant effect on how it is interpreted and usedKnow how the measures relate to the system

2323

What are the Challenges of Measurement?

Time consuming/added work.

Threatening, especially when it is used against you.

Making sure the data are accurate and consistent.

Too many indicators; not the appropriate indicators.

Using the data you collect to actually take action.

Manual versus automated data collection systems.

Results don’t match management’s view of reality.

The indicators were given to me by my manager and I had no input.

Ownership of the data collection process and the results.

Lack of training in data collection methods and analysis.

2424

What are the Benefits of Measurement?

Helps you make decisions; build confidence.

Allows you to keep tabs on what is going on.

Sets the stage for improvement/identifies problem areas.

Provides a common frame of reference for staff and management.

Identify patterns and trends in the data.

See how well performance matches our goals.

Helps us focus on what is important.

Helps you “sell” your ideas to management.

Understand interrelationships between departments and units.

Moves you away from anecdotes and one person’s view.

2525

AIM (Why are you measuring?)

Concept

Measure

Operational Definitions

Data Collection Plan

Data Collection

Analysis ACTIONACTION

The Quality Measurement Journey

Source: Lloyd, R. Quality Health Care. Jones and Bartlett Publishers, Inc., 2004: 62-64.

2626

AIM – freedom from harm for hospitalized patients

Concept – reduce patient falls

Measure – falls rate (falls per 1000 patient days)

Operational Definitions - # falls/inpatient days

Data Collection Plan – monthly; no sampling; all IP units

Data Collection – unit submits data to RM; RM assembles and send to QM for analysis

Analysis – control chart

Tests of Tests of ChangeChange

The Quality Measurement Journey

27

Relating the Three Faces ofPerformance Measurement to your work

27

Improvement Judgment

Research

Not mutually exclusive silos

All three areas must be understood as a system – interdependent. Individuals need to build skills in all three areas.

Organizations need translators who are able to speak the language of each approach.

Individuals often identify with one of these approaches and dismiss the value of the other two.

28

Primer on Heart Attacks Often called “MI” or “AMI”

Symptoms include chest pain or pressure, shortness of breath, sweating, nausea and may come on with exertion Individuals often have specific risk factors (genetics, smoking,

diabetes, high blood pressure or cholesterol…)

Treatment aimed at opening up blocked artery to restore blood flow to part of the heart and reducing the work the heart has to do

Treatments may include Catheter with stent to open artery or medication to dissolve

clot

Meds to thin blood (reduce chance new clot will form) like aspirin and others

Meds to reduce the work of the heart like beta-blockers and ACE inhibitors

29

What Happens to People Having Heart Attacks?

PatientSymptoms

EmergencySystem

CardiacCare Unit

EmergencyDepartment

Where should we give aspirin?

30

Opportunities To Give Aspirin

PatientSymptoms

EmergencySystem

CardiacCare Unit

EmergencyDepartment

Aspirin

•Pre-print admit orders

•MD orders•Nurse protocol

Aspirin

•EMT gives•MD orders by radio

Aspirin

•Take when calls•Family education•Before symptoms start

Aspirin

31

32

Dartmouth Atlas

Hospital Composite AMI CHF Pneumonia

California Pacific Med Ctr

85.5 92.8 93.5 72.3

Hartford Hospital 80.1 90.2 84.0 60.7

BIDMC 92.5 96.4 94.5 86.5

Catholic Med Ctr 93.2 98.8 93.0 86.3

DHMC 88.3 96.0 88.5 78.5

Fletcher Allen 85.7 92.0 84.5 78.5

CMS Technical Process Quality Measures Scores(0-100; low–hi)

Medicare claims data, 2005

33

Dartmouth Atlas

State Composite AMI CHF Pneumonia

California 85.4 92.3 88.1 73.5

Connecticut 88.6 92.7 88.6 81.6

Massachusetts 89.2 94.7 89.2 80.0

New Hampshire 90.7 95.3 90.8 83.1

Vermont 87.9 91.3 84.3 84.5

CMS Technical Process Quality Measures Scores(0-100; low–hi)

Medicare claims data, 2005

34

Group Work, Part 1

Review the data table from the first hand-out

Discuss the questions in your group

Be prepared to discuss in the large group setting

3535

“If I had to reduce my message for management to just a few words, I’d say it all had to do with reducing variation.”

W. Edwards Deming

36

If you don’t understand the variation that lives in your data, you will be tempted to ...

Deny the data (It doesn’t fit my view of reality!)

See trends where there are no trends

Try to explain natural variation as special events

Blame and give credit to people for things over which they have no control

Distort the process that produced the data

Kill the messenger!

37

Demonstrating Variation: Example #1

37

The figure below shows how two individuals, Mary and Bill, placed ten shots on a target. Which individual is the better

shot?

BillMary

38

Demonstrating Variation: Example #1

38

• Mary shots are clustered• Bill hit the bulls eye, but his pattern is erratic • Mary merely has to adjust her sights down and to the left. Then, her shots should all cluster near the center of the target. • People take the most current data point and assume that this represents the process's performance• If you are really sincere about understanding where your processes have been, where they are now, and where they can be in the future, you must become knowledgeable about the types of variation and how to depict them.

Mary Bill

39

Demonstrating Variation: Example #2

39

Using whatever method you have for telling time (watch, phone, Blackberry, PC, position of the sun, etc.) record the

exact time right now and decide what time it is at your table!

4040

The Problem

Aggregated data presented in tabular formats or with summary

statistics will not help you measure the impact of process improvement/redesign efforts.

Aggregated data leads to judgment, not to improvement.

41

Types of VariationCommon Cause Variation

Is inherent in the design of the process

Is due to regular, natural or ordinary causes

Affects all the outcomes of a process

Results in a “stable” process that is predictable

Also known as random or unassignable variation

Special Cause Variation

Is due to irregular or unnatural causes that are not inherent in the design of the process

Affect some, but not necessarily all aspects of the process

Results in an “unstable” process that is not predictable

Also known as non-random

or assignable variation

42

Key Point …

Common Cause does NOT mean “Good Variation.”

• It only means that the process is stable and predictable.

• If a patient’s systolic blood pressure averaged around 165 and was usually between 160 and 170 mmHg, this might be stable and predictable but completely unacceptable.

Special Cause variation does NOT mean “Bad Variation”

• A special cause may represent a very good result (e.g., a low turnaround time), which you would want to emulate.

•Special Cause merely means that the process is unstable and unpredictable.

Knowledge of the process helps you decide

if the output of the process is acceptable!

43

Balancing under and over acting to minimize net loss

X loss

loss X

Actiontaken

Actionnot taken

Actionneeded

Action notneeded

44

Elements of a Run Chart

X (CL)

Mea

sure

Time ordered observations

The centerline (CL) on a Run

Chart is the Median

Four simple tests are used to determine if special cause

variation is present

45

Time-ordered observations (1 n)

Measured value (“x”)

Natural Process Limits

Mean

Elements of a statistical process control (SPC) chart

46

Tests to Identify Special Causes on a Run Chart

Test #1: Too few or too many runs Test #2: A shift in the process

7 data points on one side of the median

Test #3: A trend7 data points constantly going up or down depending

on how many data points you have on the chart)

Test #4: A saw toothed pattern

47

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29

Point Number

Po

un

ds o

f R

ed

Ba

g W

aste

3.25

3.50

3.75

4.00

4.25

4.50

4.75

5.00

5.25

5.50

5.75

6.00

Median=4.610

Run Chart: Medical WasteTotal data points = 29

Data points on the Median = 2

Number of “useful observations” = 27

The number of runs = 14

Points on the Median (don’t count these as

“useful observations”)

48

Does this chart allow us to understand variation?

49

How about this one?

50

Is this one any better?

Isn’t this a downward trend?

51

This must surely be an upward trend!

52

Maybe this one is a better chart?

If so, why?

Is this the Median?

53

Chronic Obstructive Pulmonary Disease Run Chart

COPD Length of Stay

0.0

2.0

4.0

6.0

8.0

10.0

12.0

14.0

Month

Da

ys

spell los

1. Find the Median

2. Determine the “useful observations”

3. Apply the run test rules

33 data points with 2 on the median

We have 31 useful observations

54

Chronic Obstructive Pulmonary Disease Run Chart

COPD Length of Stay

0.0

2.0

4.0

6.0

8.0

10.0

12.0

14.0

Month

Da

ys

spell los

12 runs (should be between 11 and 21 runs)

Are there more than 8 points in a run above or below the median (shift) ?

Are there 7 data points constantly increasing (trend)?

1. Find the Median

2. Determine the “useful observations”

3. Apply the run test rules

55

Group Work, Part 2

Review the run charts on the hand-out

Discuss the questions in your group

Be prepared to discuss in the large group setting

56

% Beta Blockers after Heart Attack, Site 1

30%

40%

50%

60%

70%

80%

90%

57

% Beta Blockers after Heart Attack, Site 2

30%

40%

50%

60%

70%

80%

90%

58

% Beta Blockers after Heart Attack, Site 3

30%

40%

50%

60%

70%

80%

90%

59

Summary How measures are used have a strong influence

on the care that is delivered (P4P)

To be most effective, the measures must be clearly connected to an aim/goal, the process, and changes that are tried

Measuring data over time (and using run charts or statistical process control charts) creates useful measurement that provides insight into the process

Measurement for improvement and decision-making, not just for judgment and research