Population and Patient Centered Care: Unintended ......Population and Patient Centered Care:...

Population and Patient Centered Care:Unintended Consequences of Measurement

11/11/19

David C. Aron, MD, MS

Louis Stokes Cleveland Dept. of Veterans Affairs Medical CenterDept. of Medicine, Div. of Clinical and Molecular Endocrinology,

School of Medicine andDept. of Organizational Behavior, Weatherhead School of Management,

Case Western Reserve University Cleveland, OH

[email protected]

1

The following presentation is rated R (for opining and sarcasm which may “inadvertently” seep in). Viewer discretion is advised, but feel free to challenge everything I say.

2

3

Disclaimer: The opinions expressed in the following presentation are solely those of the presenter, and do not represent those of any agency or organization

Disclosures: • Grant funding from Dept. of Veterans Affairs• Endocrine Society Representative to NCQA/AMA on

Performance Measures• NO money from Big Pharma, medical device companies,

etc.

Outline

• A story of a performance measure gone wrong• Unintended or unanticipated consequences• Causes of unintended consequences• Dealing with unintended consequences

• Identify possible consequences beforehand• Prevention• Detection• Mitigation

4

• Two new Comprehensive Diabetes Care measures assess whether members with diabetes have their blood pressure controlled to 135/85, and whether their HbA1c levels are controlled to less than 7 percent, the nationally accepted standard of adequate HbA1c control.

• Piloted in 2005, initiated in 2006

5

A brief history of the A1c

Age- and sex-standardized rate of severe hypoglycemia resulting in hospital

admission, observation stay, or emergency department visit per 100

person-years, 2006–2013.

Kasia J. Lipska et al. Dia Care 2017;40:468-475

©2017 by American Diabetes Association

What happened? Frequency of ED visits and hospitalizations for hypoglycemia increased.

6

What was behind the rise in rates of hypoglycemia rise?

7

InputsOutputsProcesses

Sensor

Other influences on the Inputs Feedback

Cybernetic System

Effector

8

Sensor Comparator Glucose Level

Effector

Feedback

Target Organs

Other Factors,

e.g., meals

Set Point

β-Cell

Cybernetic View of Glucose Control by the β-Cell

9

Sensor Comparator A1cEffector

Rx Adjust-

ment in a patient

with diabetes

Set Point

Physician

Feedback System (Self-Performance Management)

1. How effective is the feedback?

2. What about the set point?

3. What other actions might the physician take (or not take)?

10

If Set Point is set too LOW

If Set Point is set too HIGH

Patient’s A1c > set point

DIAGNOSIS: Inadequate

Glycemic Control

OVER-Rx

UNDER-Rx

Where you set the set point, matters.

What influences the choice of set point?

11

Sensor Comparator A1cEffector

Rx Adjust-

ment in a patient

with diabetes

Set Point

Physician

Feedback System (Self-Performance Management)

1. How effective is the feedback?

2. What about the set point?

3. What other actions might the physician take (or not take)?

12

Sensor Comparator Effector

Set Point

Physician

Health Care System

Desired Performance

Pay for Performance

Influence

Feedback

A1c

Rx Adjust-

ment in a patient

with diabetes

Patient Advocacy

GroupsPayersProfessional

SocietiesPharmaceutical/ Medical Device

Industry

Quality Measurement

Industry

Stakeholders

13

14

• A national PSA campaign sponsored by the Conference of Mayors in partnership with Aim. Believe. Achieve. The Diabetes A1C Initiative(tm).

http://usmayors.org/uscm/home.asp

New NCQA Quality Diabetes Measure was promoted by NDEP (4/2004)

• “An A1C

WHO RAN THE CAMPAIGN?

17

What happened to the NCQA A1c

Assessing potential glycemic overtreatment in persons on insulin and/or sulfonylureas at high hypoglycemic risk. Tseng et al. JAMA Int Med 2014: 174; 259-268

a Study population: patients with diabetes, on insulin or sulfonylureas, having HbA1c in FY 2009.bCI: Cognitive Impairments. cD: Dementia. dAdvanced diabetes complications: end-stage renal disease, amputations, advanced retinopathy. eDiminished life expectancy: cancer, end-stage hepatic disease.fMajor neurologic disorders: gastro paresis, Parkinsons, aphasia, dysphagia, hemiplegia, apraxia, epilepsy, transient ischemic attack. gCardiovascular diseases: myocardial infarction, chronic heart failure, ischemic vascular disease.

Group

High risk patientsIncreased number of patients

Number of patients in denominator

% of study population a(n=652,738)

Overtreatment measures (in %). % with A1c

Overtreatment in groups with high risk of hypoglycemia (an unintended consequence of focus on undertreatment)

Target measure A1c

“There are known knowns; there are things we know we know. We also know there are known unknowns; that is to say we know there are some things we do not know. But there are also unknown unknowns – the ones we don't know we don't know.” Donald Rumsfeld

21

If not an intended consequence, then what?

Unintended versus Unanticipated consequences

De Zwart F. Unintended but not unanticipated consequences. Theory and Society. 2015 May 1;44(3):283-97.

http://www.katinamichael.com/research/tag/users

Consequences Anticipated Unanticipated

Intended Known knowns Unknown knowns i.e., ignorance, denial

Unintended Known unknowns Unknown unknowns

(Hanlon’s Razor)

22

L. Gregory Pawlson, MD, MPH; and Thomas H. Lee Jr, MD Am J Mgd Care 2010; 16: 16-17 Letter in reponse to paper by Choe et al. re: hypertension overtreatment

23

“It is important to note that the National Committee for Quality Assurance (NCQA) includes a strong advisory that 100% performance is not the goal and that clinical judgment should be used in applying a measure…Although physicians should be aware of guidelines and measures, they need also to apply more nuanced approaches when seeing individual patients. To imply that clinicians would knowingly put patients in harm so they could perform marginally better than other physicians on a clinical performance measure provides a rather dim view of medical practice.”

http://www.ajmc.com/publications/issue/2010/2010-01-vol16-n01http://www.ajmc.com/publications/issue/2006/2006-06-vol12-n6/Jun06-2318p309-310

knowingly put patients in harm?

• In response to a vignette of a 77y/o with serum creatinine of 2.4 mg/dl on glipizide 5 mg day with A1c 6.5 :

• 55% state that patients would be harmed by relaxing control• 42% state that relaxing control would fall outside measures • 27% state that relaxing control would be difficult to recommend, 22% believe patients would find it difficult to accept

24

Caverly TJ, Fagerlin A, Zikmund-Fisher BJ, Kirsh S, Kullgren JT, Prenovost K, Kerr EA. Appropriate Prescribing for Patients With Diabetes at High Risk for Hypoglycemia: National Survey of Veterans Affairs Health Care Professionals. JAMA Intern Med. 2015 Dec;175(12):1994-6

• To the Editor: Drs Pogach and Aron used the Toyota example of failure to quickly “pull the cord” at the first sign of a safety or quality problem to suggest leading guideline and measure development organizations did not respond to changing evidence on control of glycated hemoglobin (HbA1c) levels in patients with diabetes.

25

Margaret E. O’Kane, National Committee for Quality Assurance Washington, DC; Greg Pawlson, Department of Medicine George Washington University School of MedicineWashington, DC JAMA May 18, 2011, Vol 305, No. 19, pp96-7

In response to: Pogach L, Aron DC. Sudden acceleration of diabetes quality measures. JAMA 2011 Feb 16;305(7):709-710.

https://blog.toyota.co.uk/andon-toyota-production-system

• In fact, the National Committee for Quality Assurance (NCQA) did respond quickly when the National Institutes of Health suspended the ACCORD (Action to Control Cardiovascular Risk in Diabetes) study intervention group in February 2008 and the ACCORD and ADVANCE (Action in Diabetes and Vascular Disease: Preterax and Diamicron Modified Release Controlled Evaluation) studies were published later that year. Within 2 weeks of the suspension, NCQA suspended use of the measure and launched an expedited review of the HbA1c measures…

26


27

In Reply: In the mid 2000s, several measurement developers endorsed optimal measures for all persons with diabetes aged 18 to 74 years, even though the ACCORD and other studies were ongoing and, consequently, the outcomes still unknown. The strength of evidence supporting recommendations varied widely…The andon cord analogy highlights the fact that the developers ignored concerns voiced by the National Diabetes Quality Improvement Alliance and a subsequent Agency for Healthcare Research and Quality–funded conference years before ACCORD and other trials reported their findings—early warnings.

Although NCQA re-evaluated the scientific validity of their measures, it did so only subsequent to termination of the glycemic treatment group of ACCORD.

Pogach and Aron


Choosing Wisely. An initiative of the ABIM. American Geriatrics Society

• Avoid using medications to achieve hemoglobin A1c

In response to all of this, VA launched the Choosing Wisely/Hypoglycemia Safety Initiative (CW/HSI) in 2014

• National voluntary program• Lists

– High risk: A1c < 7 and on Insulin or Sulfonylurea who:• Are age 75 or greater• OR renal impairment (creatinine >2.0)

– Ultra high risk: A1c

Results: Facility Rates of Over and Under Rx

measure 2013 2016mean±1SD range mean±1SD range

A1c

Change in Overtreatment Rate (A1c9%)

Change (%) in Undertreatment RateDecreasing Under RxIncreasing Under Rx

Chan

ge (%

) in

Ove

rtre

atm

ent R

ate

Decr

easin

g O

ver R

xIn

crea

sing

Ove

r Rx

R= -0.653

P

Ignore UnderRx

OverRx UnderRx

Focus on OverRx

Ignore OverRx

Focus on UnderRx

• Promotion of overtreatment reduction may be associated with an increase in undertreatment in patients with diabetes.

• Systems should include balancing measures to identify potential unintended harms.

Aron DC, Wilson B, Tseng CL, Soroka O, Pogach LM. Positive Deviance in Health Care: Beware of Pseudo-Equifinality. In: Putting Systems and Complexity Sciences Into Practice 2018 (pp. 189-198). Springer, Cham.32

Undertreatment rates over time among facilities with lowest rates of overtreatment

Aron DC, Wilson B, Tseng CL, Soroka O, Pogach LM. Positive Deviance in Health Care: Beware of Pseudo-Equifinality. In: Putting Systems and Complexity Sciences Into Practice 2018 (pp. 189-198). Springer, Cham.

average rate among all VA facilities.

33

Outline



34

Goodhart’s Law

•"When a measure becomes a target, it ceases to be a good measure."

35

A taxonomy of unintended consequences, • Tunnel vision • Measure fixation • Acontextual actions or adverse selection • Misrepresentation• Gaming • Myopia • Suboptimization• Ossification is the inhibition of innovation.

36

• Smith P. On the unintended consequences of publishing performance data in the public sector. International journal of public administration. 1995 Jan 1;18(2-3):277-310.

• Rambur B, Vallett C, Cohen JA, Tarule JM. Metric-driven harm: an exploration of unintended consequences of performance measurement. Appl Nurs Res. 2013;26(4):269-72.

Adam A. Powell, PhD, Katie M. White, Melissa R. Partin, Ph , Krysten Halek, MA , Jon B. Christianson, PhD , Brian Neil, MD , Sylvia J. Hysong, PhD , Edwin J. Zarling, MD , and Hanna E. Bloomfield, MD J Gen Intern Med 27(4):405–12

37

Unintended Consequences of Implementing a National Performance Measurement System into Local Practice

• Bevan G, Hamblin R. Hitting and missing targets by ambulance services for emergency calls: effects of different systems of performance measurement within the UK. Journal of the Royal Statistical Society: Series A (Statistics in Society). 2009 Jan;172(1):161-90.

NHS Ambulance resonse times

38

39

Bloomrosen M, Starren J, Lorenzi NM, Ash JS, Patel VL, Shortliffe EH. Anticipating and addressing the unintended consequences of health IT and policy: a report from the AMIA 2009 Health Policy Meeting. Journal of the American Medical Informatics Association. 2011 Jan 1;18(1):82-90.

Campbell's law

"The more any quantitative social indicator is used for social decision-making, the more subject it will be to corruption pressures and the more apt it will be to distort and corrupt the social processes it is intended to monitor.”

By Source, Fair use, https://en.wikipedia.org/w/index.php?curid=4155776

40

Outline



41

Sterman J. Business dynamics: systems thinking and modeling for a complex world. Boston, MA: Irwin McGraw-Hill, 2000.

Single-Decision “Open Loop” View

Problem Results

Goals

Situation

Decision

“SideEffects”

Feedback ViewGoals

Environment

Actions

Goals ofOthers

Actions ofOthers

“SideEffects”

Delay Delay

Delay

Delay

DelayDelay

Delay

DelayDelay

Delay

Delay

Delay

42

A System…

• An interconnected set of elements that is coherently organized in a way that achieves something.

• Elements• Interconnectedness• Function or purpose

-43-

Meadows, D. Thinking in Systems: A Primer.

Dividing a cow in half does not give you two smaller cows

44

The picture can't be displayed.

Todd Little

Properties of Complex Systems • Large collections of relatively simple components• Components interact locally and nonlinearly• Limited centralized control• Components have limited access to global information• Components have ‘bounded rationality’• System as a whole has emergent, collective behavior

• Complex dynamics• Information processing and computation• Adaptation and learning

45Mitchell M. Complexity: A guided tour. Oxford University Press; 2009.

Complex Environment

46

Thank you to whoever came up with this series of figures. Please let me know who you are so I can give you proper attribution.

Intervention

47Insert action into the system

Unpredictable Outcomes

48

EMERGENCE

I need some volunteers

49

Schelling Model

50

https://www.youtube.com/watch?v=JjfihtGefxk

https://www.youtube.com/watch?v=JjfihtGefxk

51

• Complex systems are inherently uncontrollable

• They cannot be understood sufficiently even while they are unfolding

• So prediction and control of complex systems is very hard, if not impossible, to do

52http://static.guim.co.uk/sys-images/Guardian/Pix/site_furniture/2010/4/28/1272477292847/Afghanistan-powerpoint-gr-008.jpg

Outline



53

THUMB PINNING EXERCISE

Julie Johnson, PhD

• Goal: pin your partner as many times as possible in 15 seconds. I will tell you when to start.

54

Question your assumptions.

55https://thegreatoutdoorsman.net/2019/07/09/the-far-side/

P4P

Rx of DM Structure, process, outcomes, cost

“Quality” of Care“Value” (Q/cost)

“Set Points”

Assumption: Individual variation is caused by variation in motivation - i.e. that poor performance is a matter of intention, and that change in motivation is therefore likely to improve performance.

Identify possible consequences beforehand: 1. Question your assumptions, i.e. your mental model.

Identify possible consequences beforehand: 2. Systems thinking

• Meadows, D. Thinking in Systems: A Primer. 2008• De Savigny D, Adam T, editors. Systems thinking for health systems

strengthening. World Health Organization; 2009. • Systems Sciences and Population Health; El-Sayed and Galea, editors; 2017 57

System DynamicsSimulating Dynamic Complexity

Good at Capturing• Differences between short- and long-term consequences of an action• Time delays (e.g., incubation period, time to detect, time to respond)• Accumulations (e.g., prevalences, resources, attitudes)• Behavioral feedback (reactions by various actors)• Nonlinear causal relationships (e.g., threshold effects, saturation effects)• Differences or inconsistencies in goals/values among stakeholders

Forrester JW. Industrial Dynamics. Cambridge, MA: MIT Press; 1961.

Sterman JD. Business Dynamics: Systems Thinking and Modeling for a Complex World. Boston, MA: Irwin/McGraw-Hill; 2000.

Outline



59

Prevention 1. Use good measuresWhat Makes a Good Quality Measure (in addition to accuracy and validity)?

• Target patients most likely to benefit• Help providers do the “right” thing• Incorporate (or at least don’t ignore) patient

preferences• Acknowledge limitations of current data sources

and resulting measures (and motivate collection and use of clinically detailed data)

• Avoid unintended consequences

http://www.ahrq.gov/QUAL/diabetescare/ 60

Outline



61

In addition to making sure you are measuring what is important:

Avoiding and mitigating unintended consequences.

• Balancing measures• Indirect – What might doctors NOT do in order for them to do what you

want them to do.• Direct – Reciprocal; If you are assessing undertreatment, then also

asses overtreatment• Continuous rather than dichotomous measures

62

8

9

10

7.5

7

6

CHOOSING WISELY : A1c ≥7.5%Ages ≥65 on medication other than metformin alone

NQF MEASURE: A1c 65-75; no exclusion criteria

NQF MEASURE: A1c>9%UNDER TREATMENT (UT) Applies to ages >65-75; no exclusion criteria

In Range (IR)Out of Range (OOR)

DHHS PROPOSED OVER TREATMENT (OT) : A1c65 on hypoglycemic agents

8.5GUIDELINES: Limited life expectancy, significant co-morbid conditions

Pogach L, Tseng CL, Soroka O, Maney M, Aron D. A Proposal for an Out-of-Range Glycemic Population Health Safety Measure for Older Adults With Diabetes. Diabetes Care. 2017 Apr;40(4):518-525.

Out of Range Measure

63

Actions have consequences, both intended and unintended. The implementation of an improvement initiative or a performance measure are actions. When adverse unintended consequences can be anticipated, it is incumbent upon systems to include mitigating actions such as counterbalancing measures to ensure that unintended harms are avoided.

Summary

Aron DC. No "Black swan": unintended but not unanticipated consequences of diabetes performance measurement. Jt. Comm J Qual. Patient. Saf. 2013;39(3):106-8. 64

“It is easy to dodge our responsibilities, but we cannot dodge the consequences of dodging our responsibilities.”

Josiah Stamp

65

Additional References• Mannion R, Braithwaite J. Unintended consequences of performance

measurement in healthcare: 20 salutary lessons from the English National Health Service. Intern Med J. 2012;42(5):569-74.

• Hysong SJ, SoRelle R, Broussard Smitham K, Petersen LA Reports of unintended consequences of financial incentives to improve management of hypertension. PLoS ONE 2017; 12(9): e0184856.

• Nescolarde-Selva JA, Gash H, Usó-Domenech JL. What are unintended and adverse consequences?. Kybernetes. 2019 Feb 4;48(2):226-37.

• Franco-Santos M, Otley D. Reviewing and theorizing the unintended consequences of performance management systems. International Journal of Management Reviews. 2018 Jul;20(3):696-730.

• Mica, A., Sociology as Analysis of the Unintended From the Problem of Ignorance to the Discovery of the Possible. Routledge. 2019.

66

Population and Patient Centered Care:�Unintended Consequences of Measurement�11/11/19��David C. Aron, MD, MS��Louis Stokes Cleveland Dept. of Veterans Affairs Medical Center�Dept. of Medicine, Div. of Clinical and Molecular Endocrinology,�School of Medicine and�Dept. of Organizational Behavior, Weatherhead School of Management,�Case Western Reserve University Cleveland, OH��[email protected]�Slide Number 2Slide Number 3OutlineSlide Number 5Slide Number 6What was behind the rise in rates of hypoglycemia rise?Slide Number 8Slide Number 9Slide Number 10Slide Number 11Slide Number 12Slide Number 13Slide Number 14New NCQA Quality Diabetes Measure was promoted by NDEP (4/2004)Slide Number 16WHO RAN THE CAMPAIGN?What happened to the NCQA A1c

Population and Patient Centered Care: Unintended ......Population and Patient Centered Care:...

Documents

Transcript of Population and Patient Centered Care: Unintended ......Population and Patient Centered Care:...