Challenges for institutional performance measures...Challenges for institutional performance...
Transcript of Challenges for institutional performance measures...Challenges for institutional performance...
![Page 1: Challenges for institutional performance measures...Challenges for institutional performance measures Responsible Data Science in health care Nicolette de Keizer Dept Medical Informatics](https://reader034.fdocuments.net/reader034/viewer/2022042311/5ed96c5ff59b0f56f45f76cd/html5/thumbnails/1.jpg)
Challenges for institutional performance measures
Responsible Data Science in health care
Nicolette de Keizer Dept Medical Informatics
Academic Medical Center, Amsterdam
![Page 2: Challenges for institutional performance measures...Challenges for institutional performance measures Responsible Data Science in health care Nicolette de Keizer Dept Medical Informatics](https://reader034.fdocuments.net/reader034/viewer/2022042311/5ed96c5ff59b0f56f45f76cd/html5/thumbnails/2.jpg)
2
Data science in health care
Data reuse: • Management information • Quality assurance • Research • Surveillance
![Page 3: Challenges for institutional performance measures...Challenges for institutional performance measures Responsible Data Science in health care Nicolette de Keizer Dept Medical Informatics](https://reader034.fdocuments.net/reader034/viewer/2022042311/5ed96c5ff59b0f56f45f76cd/html5/thumbnails/3.jpg)
3
Learning Health System
Data reuse: • Management information • Quality assurance • Research • Surveillance • Financial reimbursement
![Page 4: Challenges for institutional performance measures...Challenges for institutional performance measures Responsible Data Science in health care Nicolette de Keizer Dept Medical Informatics](https://reader034.fdocuments.net/reader034/viewer/2022042311/5ed96c5ff59b0f56f45f76cd/html5/thumbnails/4.jpg)
Quality registries in health care >150 registries in health care Aims: – Accountability
• Government • Insurance companies • Patients
– Quality assessment and improvement – Scientific research
![Page 5: Challenges for institutional performance measures...Challenges for institutional performance measures Responsible Data Science in health care Nicolette de Keizer Dept Medical Informatics](https://reader034.fdocuments.net/reader034/viewer/2022042311/5ed96c5ff59b0f56f45f76cd/html5/thumbnails/5.jpg)
Accountability
![Page 6: Challenges for institutional performance measures...Challenges for institutional performance measures Responsible Data Science in health care Nicolette de Keizer Dept Medical Informatics](https://reader034.fdocuments.net/reader034/viewer/2022042311/5ed96c5ff59b0f56f45f76cd/html5/thumbnails/6.jpg)
Accountability Is the data fair, accurate, transparent?
![Page 7: Challenges for institutional performance measures...Challenges for institutional performance measures Responsible Data Science in health care Nicolette de Keizer Dept Medical Informatics](https://reader034.fdocuments.net/reader034/viewer/2022042311/5ed96c5ff59b0f56f45f76cd/html5/thumbnails/7.jpg)
Accountability Is the data fair, accurate, transparent? Large consequences ….. – Loss of faith – Demotivation – Loss of budget
![Page 8: Challenges for institutional performance measures...Challenges for institutional performance measures Responsible Data Science in health care Nicolette de Keizer Dept Medical Informatics](https://reader034.fdocuments.net/reader034/viewer/2022042311/5ed96c5ff59b0f56f45f76cd/html5/thumbnails/8.jpg)
8
![Page 9: Challenges for institutional performance measures...Challenges for institutional performance measures Responsible Data Science in health care Nicolette de Keizer Dept Medical Informatics](https://reader034.fdocuments.net/reader034/viewer/2022042311/5ed96c5ff59b0f56f45f76cd/html5/thumbnails/9.jpg)
9
History and development
Founded in 1996 by and for intensivists
0
100000
200000
300000
400000
500000
600000
700000
800000
900000
1000000
0
10
20
30
40
50
60
70
80
90
100
1997199819992000200120022003200420052006200720082009201020112012201320142015
# admissions
# IC
Us
![Page 10: Challenges for institutional performance measures...Challenges for institutional performance measures Responsible Data Science in health care Nicolette de Keizer Dept Medical Informatics](https://reader034.fdocuments.net/reader034/viewer/2022042311/5ed96c5ff59b0f56f45f76cd/html5/thumbnails/10.jpg)
Quality assessment and improvement Benchmarking
Observed difference = Difference in quality of care Onverklaarde
verschillen Onverklaarde verschillen
Onverklaarde verschillen
Registratie verschillen
Patiënten kenmerken
Toeval
Kwaliteit van zorg
0,40
0,50
0,60
0,70
0,80
0,90
1,00
A B C D E F G H
indi
cato
r sco
re
ICU/institution
![Page 11: Challenges for institutional performance measures...Challenges for institutional performance measures Responsible Data Science in health care Nicolette de Keizer Dept Medical Informatics](https://reader034.fdocuments.net/reader034/viewer/2022042311/5ed96c5ff59b0f56f45f76cd/html5/thumbnails/11.jpg)
Variation
Observed diference
Unexplained differences
Onverklaarde verschillen
Onverklaarde verschillen
Patiënten kenmerken
Toeval
Kwaliteit van zorg
Case mix
Mortality 20% 17%
Age 68 57
Comorbidity 40% 5%
![Page 12: Challenges for institutional performance measures...Challenges for institutional performance measures Responsible Data Science in health care Nicolette de Keizer Dept Medical Informatics](https://reader034.fdocuments.net/reader034/viewer/2022042311/5ed96c5ff59b0f56f45f76cd/html5/thumbnails/12.jpg)
Variation
Observed difference
Unexplained differences
Onverklaarde verschillen
Case mix
Toeval
Kwaliteit van zorg
Unexplained differences
Registration, definition differences
![Page 13: Challenges for institutional performance measures...Challenges for institutional performance measures Responsible Data Science in health care Nicolette de Keizer Dept Medical Informatics](https://reader034.fdocuments.net/reader034/viewer/2022042311/5ed96c5ff59b0f56f45f76cd/html5/thumbnails/13.jpg)
Variation
Observed difference
Unexplained differences
Uncertainty
Quality of care
Unexplained differences
Unexplained differences
Case mix
Registration, definition differences
![Page 14: Challenges for institutional performance measures...Challenges for institutional performance measures Responsible Data Science in health care Nicolette de Keizer Dept Medical Informatics](https://reader034.fdocuments.net/reader034/viewer/2022042311/5ed96c5ff59b0f56f45f76cd/html5/thumbnails/14.jpg)
Mortality as quality indicator
14
Input (1st 24 hour) Output
ICU Hospital
Prognostic models: APACHE II en IV, SAPSII…
![Page 15: Challenges for institutional performance measures...Challenges for institutional performance measures Responsible Data Science in health care Nicolette de Keizer Dept Medical Informatics](https://reader034.fdocuments.net/reader034/viewer/2022042311/5ed96c5ff59b0f56f45f76cd/html5/thumbnails/15.jpg)
15
Standardized Mortality Ratio (SMR)
Expected mortality depends on prognostic model
Observed in-hospital mortality
Expected mortality
SMR=
![Page 16: Challenges for institutional performance measures...Challenges for institutional performance measures Responsible Data Science in health care Nicolette de Keizer Dept Medical Informatics](https://reader034.fdocuments.net/reader034/viewer/2022042311/5ed96c5ff59b0f56f45f76cd/html5/thumbnails/16.jpg)
Ranking institutions
A common procedure is to rank institutions by SMR, providing a league table The top/bottom 10% or 25% are sometimes labelled as excellent/poor performers
Marshall EC, Spiegelhalter DJ. BMJ 1998;316(7146):1701-4.
![Page 17: Challenges for institutional performance measures...Challenges for institutional performance measures Responsible Data Science in health care Nicolette de Keizer Dept Medical Informatics](https://reader034.fdocuments.net/reader034/viewer/2022042311/5ed96c5ff59b0f56f45f76cd/html5/thumbnails/17.jpg)
ICU ranking accurate?
Ranks of 40 Dutch ICUs (n=86,427) SMRs based on Apache II, SAPSII, MPM24II model Rank CIs computed by bootstrap sampling (10,000 replications) Excellent performance: with 95% certainty among top 25% institutes Very poor performance: with 95% certainty among bottom 25% institutes
Bakhshi-Raiez F et al. Crit Care Med 2007
![Page 18: Challenges for institutional performance measures...Challenges for institutional performance measures Responsible Data Science in health care Nicolette de Keizer Dept Medical Informatics](https://reader034.fdocuments.net/reader034/viewer/2022042311/5ed96c5ff59b0f56f45f76cd/html5/thumbnails/18.jpg)
0 10 20 30 40
215
251836223
1033408
1539141911132779
384
322
313537231729126
2016281
34243026
0 10 20 30 40
518141021258
367
3319222
39383
4030374
1527281229119
176
162431321
263413202335
0 10 20 30 40
18255
10403339363738192
2122157
1432172949
2338
1227306
3416112831351
13242620
Apache II SAPS II MPM24 II
Bak
hshi
-Rai
ez F
et a
l. C
rit C
are
Med
200
7
![Page 19: Challenges for institutional performance measures...Challenges for institutional performance measures Responsible Data Science in health care Nicolette de Keizer Dept Medical Informatics](https://reader034.fdocuments.net/reader034/viewer/2022042311/5ed96c5ff59b0f56f45f76cd/html5/thumbnails/19.jpg)
Results
20 ICU significantly differ in rank (2-19 positions) by 1 or more pair of models 3 ICUs rated as performance outlier by one model while others excluded this possibility with 95% certainty
![Page 20: Challenges for institutional performance measures...Challenges for institutional performance measures Responsible Data Science in health care Nicolette de Keizer Dept Medical Informatics](https://reader034.fdocuments.net/reader034/viewer/2022042311/5ed96c5ff59b0f56f45f76cd/html5/thumbnails/20.jpg)
215
251836223
1033408
1539141911132779
384
322
3135372317
518141021258
367
3319222
39383
4030374
1527281229119
176
18255
10403339363738192
2122157
1432172949
2338
1227306
Apache II SAPS II MPM24 II
![Page 21: Challenges for institutional performance measures...Challenges for institutional performance measures Responsible Data Science in health care Nicolette de Keizer Dept Medical Informatics](https://reader034.fdocuments.net/reader034/viewer/2022042311/5ed96c5ff59b0f56f45f76cd/html5/thumbnails/21.jpg)
215
251836223
1033408
1539141911132779
384
322
3135372317
518141021258
367
3319222
39383
4030374
1527281229119
176
18255
10403339363738192
2122157
1432172949
2338
1227306
Apache II SAPS II MPM24 II
![Page 22: Challenges for institutional performance measures...Challenges for institutional performance measures Responsible Data Science in health care Nicolette de Keizer Dept Medical Informatics](https://reader034.fdocuments.net/reader034/viewer/2022042311/5ed96c5ff59b0f56f45f76cd/html5/thumbnails/22.jpg)
215
251836223
1033408
1539141911132779
384
322
3135372317
518141021258
367
3319222
39383
4030374
1527281229119
176
18255
10403339363738192
2122157
1432172949
2338
1227306
Apache II SAPS II MPM24 II
![Page 23: Challenges for institutional performance measures...Challenges for institutional performance measures Responsible Data Science in health care Nicolette de Keizer Dept Medical Informatics](https://reader034.fdocuments.net/reader034/viewer/2022042311/5ed96c5ff59b0f56f45f76cd/html5/thumbnails/23.jpg)
0 10 20 30 40
81539141911132779
384
322
313537231729126
2016281
34243026
0 10 20 30 40
19222
39383
4030374
1527281229119
176
162431321
263413202335
0 10 20 30 40
192
2122157
1432172949
2338
1227306
3416112831351
13242620
Apache II SAPS II MPM24 II
![Page 24: Challenges for institutional performance measures...Challenges for institutional performance measures Responsible Data Science in health care Nicolette de Keizer Dept Medical Informatics](https://reader034.fdocuments.net/reader034/viewer/2022042311/5ed96c5ff59b0f56f45f76cd/html5/thumbnails/24.jpg)
Benchmark –SMR fair?
Benchmark on in-hospital mortality or long term mortality? Why choose for hospital mortality? – Sooner and easily available – Mortality not related to ICU admission
Why choose for longterm mortality? – More relevant for patients – Less influence by discharge policy
![Page 25: Challenges for institutional performance measures...Challenges for institutional performance measures Responsible Data Science in health care Nicolette de Keizer Dept Medical Informatics](https://reader034.fdocuments.net/reader034/viewer/2022042311/5ed96c5ff59b0f56f45f76cd/html5/thumbnails/25.jpg)
In-hospital mortality vs long term mortality
0 0,5 1 1,5 2
32
4
2
2
2
42
323
22
2
41925
2110
67
1311161824251217
32030412722281423263840153732
83519393634332942433144
SMR 3 months
0 0,5 1 1,5 2
2
3
2
34
4
2
2
3
2222
2
123456789
1011121314151617181920212223242526272829303132333435363738394041424344
SMR In-hospital
S. B
rinkm
an e
t al.
Inte
nsiv
e C
are
Med
. 201
3 N
ov;3
9(11
):192
5-31
![Page 26: Challenges for institutional performance measures...Challenges for institutional performance measures Responsible Data Science in health care Nicolette de Keizer Dept Medical Informatics](https://reader034.fdocuments.net/reader034/viewer/2022042311/5ed96c5ff59b0f56f45f76cd/html5/thumbnails/26.jpg)
26
Other Challenges regarding FACT
Observed difference
Unexplained differences
Uncertainty
Quality of care
Unexplained differences
Unexplained differences
Case mix
Registration, definition differences
![Page 27: Challenges for institutional performance measures...Challenges for institutional performance measures Responsible Data Science in health care Nicolette de Keizer Dept Medical Informatics](https://reader034.fdocuments.net/reader034/viewer/2022042311/5ed96c5ff59b0f56f45f76cd/html5/thumbnails/27.jpg)
27
Other Challenges regarding FACT The way of data collection influences SMR – Sample frequency – Coded data versus free text
Interpretation and definition of QI – Expl IGZ: Mean duration of mechanical ventilation
• Different types of ventilation • Duration in hours or calendar days • Mean based on all ICU patients or ventilated patients
![Page 28: Challenges for institutional performance measures...Challenges for institutional performance measures Responsible Data Science in health care Nicolette de Keizer Dept Medical Informatics](https://reader034.fdocuments.net/reader034/viewer/2022042311/5ed96c5ff59b0f56f45f76cd/html5/thumbnails/28.jpg)
28
Challenges Learning Health System
Formalisation of indicator definitions
Information models Terminological systems
![Page 29: Challenges for institutional performance measures...Challenges for institutional performance measures Responsible Data Science in health care Nicolette de Keizer Dept Medical Informatics](https://reader034.fdocuments.net/reader034/viewer/2022042311/5ed96c5ff59b0f56f45f76cd/html5/thumbnails/29.jpg)
29
Conclusion Performance measurement of health facilities might be biased by – Data source – Case mix correction model – Definitions (of endpoints) Need for methods to unambiguously capture health data, formalize indicators and make health data transparent for different reuse purposes