Clinical Prediction Rules
-
Upload
jen-hsiang-chuang -
Category
Health & Medicine
-
view
89 -
download
1
Transcript of Clinical Prediction Rules
![Page 1: Clinical Prediction Rules](https://reader030.fdocuments.net/reader030/viewer/2022032502/55b9a16dbb61eb27408b46f0/html5/thumbnails/1.jpg)
Clinical Prediction Rules
Jen-Hsiang Chuang, MD, MS, PhDCenters for Disease Control Taiwan
1
![Page 2: Clinical Prediction Rules](https://reader030.fdocuments.net/reader030/viewer/2022032502/55b9a16dbb61eb27408b46f0/html5/thumbnails/2.jpg)
Clinical Prediction Rules (CPRs)
• Synonym: clinical decision rules
• Definition: decision-making tools for clinicians including 3 or more variables– Provide the probability of an outcome– Suggest a diagnostic or therapeutic course of
action
Laupacis A, et al. Clinical prediction rules. JAMA 1997;277:488-494. 2
![Page 3: Clinical Prediction Rules](https://reader030.fdocuments.net/reader030/viewer/2022032502/55b9a16dbb61eb27408b46f0/html5/thumbnails/3.jpg)
Clinical Prediction Rules Vs. Clinical Practice Guidelines
• Clinical prediction rules– Derived from original research involving many
patients and mathematical analysis
• Clinical practice guidelines– Consensus among experts– GOBSAT (Good Old Boys Sat At Table)
(Miller J, et al. Lancet 2000;355:82-3)– But can include CPRs
3
![Page 4: Clinical Prediction Rules](https://reader030.fdocuments.net/reader030/viewer/2022032502/55b9a16dbb61eb27408b46f0/html5/thumbnails/4.jpg)
Functions of CPRs
• CPRs help clinicians cope with uncertainty and improve efficiency– Cope with uncertainty
• Community-acquired pneumonia (Fine MJ, et al. NEJM 1997;336:243-250)
– Improve efficiency• Ottawa Ankle Rules for the use of
radiography (Stiell IG, et al. Ann Emerg Med 1992;21:384-90)
Stiell IG, et al. Annals Emergency Med 1999;33:437-47. 4
![Page 5: Clinical Prediction Rules](https://reader030.fdocuments.net/reader030/viewer/2022032502/55b9a16dbb61eb27408b46f0/html5/thumbnails/5.jpg)
Prototype of a CPR for Predicting Death
Predictor variables ScoreAge > 75 yr 6Severe pain 10Emergency 5
Total points 0-21
Interpretation of the scoreHigh risk: > 6 points (30% deaths) -> aggressive TxLow risk: 6 points (3% deaths) -> conservative Tx
Wasson JH, et al. Clinical prediction rules. NEJM 1985;313:793-9. 5
![Page 6: Clinical Prediction Rules](https://reader030.fdocuments.net/reader030/viewer/2022032502/55b9a16dbb61eb27408b46f0/html5/thumbnails/6.jpg)
Three Stages in theEvaluation of a CPR
1. Development of a CPR
2. Prospective validation of a CPR
3. Impact analysis of a CPR
McGinn TG, et al. Users’ guide to the medical literature. JAMA 2000;284:79-84. 6
![Page 7: Clinical Prediction Rules](https://reader030.fdocuments.net/reader030/viewer/2022032502/55b9a16dbb61eb27408b46f0/html5/thumbnails/7.jpg)
So What?
• Q: “Give me the reasons why I need to stay here to listen your presentation?”
• A: a medical informatician may play two roles– Reader role– Developer role
7
![Page 8: Clinical Prediction Rules](https://reader030.fdocuments.net/reader030/viewer/2022032502/55b9a16dbb61eb27408b46f0/html5/thumbnails/8.jpg)
Development of a CPR
8
![Page 9: Clinical Prediction Rules](https://reader030.fdocuments.net/reader030/viewer/2022032502/55b9a16dbb61eb27408b46f0/html5/thumbnails/9.jpg)
Checklist of Standards for Development of a CPR
1. Definition of outcome
2. Definition of predictor variables
3. Reliability of predictor variables
4. Selection of subjects
5. Sample size
6. Mathematical techniques
7. Sensibility of CPR
8. Accuracy
Stiell IG, et al. Annals Emergency Med 1999;33:437-47. 9
![Page 10: Clinical Prediction Rules](https://reader030.fdocuments.net/reader030/viewer/2022032502/55b9a16dbb61eb27408b46f0/html5/thumbnails/10.jpg)
1. Definition of Outcome
• Clearly defined and clinically important– Explicit criteria for diagnosis– Biologic better than behavioral outcome
• Blind assessment of outcome– More important for a “soft” outcome– Less important for a “hard” outcome
10
![Page 11: Clinical Prediction Rules](https://reader030.fdocuments.net/reader030/viewer/2022032502/55b9a16dbb61eb27408b46f0/html5/thumbnails/11.jpg)
2. Definition of Predictor Variables
• Clearly defined– Best: collected prospectively, specifically– Less good: collected prospectively as part of
another study– Worst: collected from retrospective review of
records
• Blind assessment of predictor variables
11
![Page 12: Clinical Prediction Rules](https://reader030.fdocuments.net/reader030/viewer/2022032502/55b9a16dbb61eb27408b46f0/html5/thumbnails/12.jpg)
3. Reliability of Predictor Variables
• Only reliable variables be included– Intraobserver reliability– Interobserver reliability
• Measurement of reliability– Dichotomous or nominal data: – Ordinal data: weighted – Continuous data: intraclass correlation
coefficient
http://www.dmi.columbia.edu/homepages/chuangj/kappa/12
![Page 13: Clinical Prediction Rules](https://reader030.fdocuments.net/reader030/viewer/2022032502/55b9a16dbb61eb27408b46f0/html5/thumbnails/13.jpg)
4. Selection of Subjects
• Patient characteristics stated– Inclusion and exclusion criteria– Method of selection– Clinical and demographic characteristics
• Study site described– Type of institution (primary, secondary, tertiary)– Setting (clinic, ER, hospital ward)– Teaching or non-teaching
13
![Page 14: Clinical Prediction Rules](https://reader030.fdocuments.net/reader030/viewer/2022032502/55b9a16dbb61eb27408b46f0/html5/thumbnails/14.jpg)
5. Sample Size
• Overfitting problem– Too few outcome events per predictor variable
• Appropriate sample size– Rule of thumb: at least 10 outcome events per
independent variable– e.g., 3 findings to predict death => at least 30
patients died
14
![Page 15: Clinical Prediction Rules](https://reader030.fdocuments.net/reader030/viewer/2022032502/55b9a16dbb61eb27408b46f0/html5/thumbnails/15.jpg)
6. Mathematical Techniques
• Mathematical methods adequately described and justified– Multivariate analysis
• Logistic regression• Discriminant analysis
– Machine learning• Recursive partitioning (including decision tree
learning)• Neural networks
– Survival analysis (survival data only)• Cox's Proportional Hazard Model
15
![Page 16: Clinical Prediction Rules](https://reader030.fdocuments.net/reader030/viewer/2022032502/55b9a16dbb61eb27408b46f0/html5/thumbnails/16.jpg)
Multivariate Analysis
• General model
• Logistic regression
– Where P is probability of outcome; G is log odds of outcome
• Discriminant analysis– Compute cutoff (C)– Assign patient to class 1 if G < C; otherwise assign
patient to class 2
nnxbxbxbbG 22110
P
PG
1ln
16
![Page 17: Clinical Prediction Rules](https://reader030.fdocuments.net/reader030/viewer/2022032502/55b9a16dbb61eb27408b46f0/html5/thumbnails/17.jpg)
Logistic Regression Vs. Discriminant Analysis
• Logistic regression is much more popular than discriminant analysis (Concato et, al. 1993)– Logistic regression
• Binary outcome• Estimate individual risk and odds ratios
– Discriminant analysis• Categorical outcome• Optimal performance requires many predictor
variables as continuous data• Famous application: diagnosis alcoholism
17
![Page 18: Clinical Prediction Rules](https://reader030.fdocuments.net/reader030/viewer/2022032502/55b9a16dbb61eb27408b46f0/html5/thumbnails/18.jpg)
Recursive Partitioning
• Principle– Build an empirical tree diagram by repetitively
splitting patient population into smaller and smaller categories
Yes No
4 5Employed
2
1Yes
3
NoAge>30
18
![Page 19: Clinical Prediction Rules](https://reader030.fdocuments.net/reader030/viewer/2022032502/55b9a16dbb61eb27408b46f0/html5/thumbnails/19.jpg)
Recursive Partitioning Vs. Multivariate Analysis
• Recursive partitioning provides a simpler classification rule
• Recursive partitioning may identify nonlinear relationships with outcome event
• Recursive partitioning need greater sample size
• Logistic regression can estimate individual risk and odds ratios
Cook EF, et al. J Chron Dis 1984;37:721-31. 19
![Page 20: Clinical Prediction Rules](https://reader030.fdocuments.net/reader030/viewer/2022032502/55b9a16dbb61eb27408b46f0/html5/thumbnails/20.jpg)
Cross SS, et al. Introduction to neural network. Lancet 1995;346:1075-9.
Neural Networks
20
![Page 21: Clinical Prediction Rules](https://reader030.fdocuments.net/reader030/viewer/2022032502/55b9a16dbb61eb27408b46f0/html5/thumbnails/21.jpg)
Clinical Applications of Neural Networks
• Diagnosis – AMI, Appendicitis, back pain, dementia, STD
• Imaging– Radiographs, PET, NMR, perfusion scans
• Analysis of wave forms– ECGs, EEGs
• Outcome prediction– Recovery from surgery, cancer, liver transplantation
• Identification of pathological specimens• Genomics
Baxt WG. Application of ANN to clinical medicine. Lancet 1995;346:1135-8.
21
![Page 22: Clinical Prediction Rules](https://reader030.fdocuments.net/reader030/viewer/2022032502/55b9a16dbb61eb27408b46f0/html5/thumbnails/22.jpg)
Kattan MW, et al. ANN for medical classification decisions. Arch Pathol Lab Med 1995;119:672-7.
Advantages of Neural Networks
•Multiple partitioning•Nonlinear partitioning
22
![Page 23: Clinical Prediction Rules](https://reader030.fdocuments.net/reader030/viewer/2022032502/55b9a16dbb61eb27408b46f0/html5/thumbnails/23.jpg)
Disadvantages of Neural Networks
• Slow to train
• “Black-boxes”
Kattan MW, et al. ANN for medical classification decisions. Arch Pathol Lab Med 1995;119:672-7.23
![Page 24: Clinical Prediction Rules](https://reader030.fdocuments.net/reader030/viewer/2022032502/55b9a16dbb61eb27408b46f0/html5/thumbnails/24.jpg)
7. Sensibility of CPR
• “Sensibility”– clinically reasonable, easy to use, course of action
described– judgment
• Clinically reasonable– Content validity
• Easy to use– Length of time needed to apply– Simplicity of interpretation
• Course of action described24
![Page 25: Clinical Prediction Rules](https://reader030.fdocuments.net/reader030/viewer/2022032502/55b9a16dbb61eb27408b46f0/html5/thumbnails/25.jpg)
8. Accuracy of CPR
• Rationale• Measurement of accuracy
– 2x2 table with sensitivity, specificity, with respective 95% CIs
– Receiver operator characteristic (ROC) curves• Statistical validation
– Cross-validation: Training set vs. test set
Wasson JH, et al. Clinical prediction rules. NEJM 1985;313:793-9. 25
![Page 26: Clinical Prediction Rules](https://reader030.fdocuments.net/reader030/viewer/2022032502/55b9a16dbb61eb27408b46f0/html5/thumbnails/26.jpg)
Classification Performance of a CPR
Predicted Outcome
Actual Outcome
Disease No Disease
Disease 74 244
No Disease 0 247
Sensitivity (95% CI): 1.0 (0.95-1.0)Specificity (95% CI): 0.50 (0.46-0.55)
Stiell IG, et al. Implementation of the Ottawa Ankle Rules. JAMA 1994;271:827-32. 26
![Page 27: Clinical Prediction Rules](https://reader030.fdocuments.net/reader030/viewer/2022032502/55b9a16dbb61eb27408b46f0/html5/thumbnails/27.jpg)
Kuo HS, Chuang JH, Tang GJ, et al. Chin Med J (Taipei) 1999;62:673-681.
27
![Page 28: Clinical Prediction Rules](https://reader030.fdocuments.net/reader030/viewer/2022032502/55b9a16dbb61eb27408b46f0/html5/thumbnails/28.jpg)
Example: Development of a CPR
Ottawa Ankle Rules
Stiell IG, et al. Ann Emerg Med 1992;21:384-90
28
![Page 29: Clinical Prediction Rules](https://reader030.fdocuments.net/reader030/viewer/2022032502/55b9a16dbb61eb27408b46f0/html5/thumbnails/29.jpg)
Ottawa Ankle Rules
29
![Page 30: Clinical Prediction Rules](https://reader030.fdocuments.net/reader030/viewer/2022032502/55b9a16dbb61eb27408b46f0/html5/thumbnails/30.jpg)
The Need for an Ankle Rule
• Blunt ankle trauma– One of the most common injuries in ER– Less than 15% of patients have fractures– Physicians used to order radiography for all
ankle injury patients– 85% negative for fracture– $500 M annually in North America– No widely accepted guideline
Stiell IG, et al. Implementation of the Ottawa Ankle Rules. JAMA 1994;271:827-32. 30
![Page 31: Clinical Prediction Rules](https://reader030.fdocuments.net/reader030/viewer/2022032502/55b9a16dbb61eb27408b46f0/html5/thumbnails/31.jpg)
Study Design• Objective
– Develop CPR with 100% sensitivity• Design
– Prospective survey of ED patients over 5 months• Patient population:
– Setting: Two university hospital EDs in Ottawa– Inclusion: All acute blunt injuries of ankle– Exclusion: < 18 y/o, pregnant, referral, etc
• Data collection– 32 clinical variables collected by 21 trained physicians before
radiography– 100 patients examined by a 2nd physician
31
![Page 32: Clinical Prediction Rules](https://reader030.fdocuments.net/reader030/viewer/2022032502/55b9a16dbb61eb27408b46f0/html5/thumbnails/32.jpg)
Study Design (Cont.)
• Measurements of outcomes
– Radiography interpreted by a radiologist blinded to the contents of data collection sheets
• No fracture or insignificant fracture
• Clinically significant fracture
• Data analysis
– Variables found to be both strongly associated with a significant fracture (P < 0.05) and reliable ( > 0.6) were analyzed by logistic regression and recursive partitioning
32
![Page 33: Clinical Prediction Rules](https://reader030.fdocuments.net/reader030/viewer/2022032502/55b9a16dbb61eb27408b46f0/html5/thumbnails/33.jpg)
Results
• 70 (10.2%) significant malleolar fractures in 689 ankle injury patients
• Univariate analysis: 17 variables were significantly associated with fractures
• 9 non-reliable variables were further eliminated
• Logistic regression: Sen: 1.0, Spe: .29• Recursive partitioning: Sen: 1.0, Spe: .40
33
![Page 34: Clinical Prediction Rules](https://reader030.fdocuments.net/reader030/viewer/2022032502/55b9a16dbb61eb27408b46f0/html5/thumbnails/34.jpg)
689
561128
70#
39#
Yes No31#
67
Yes
12# 494
No19#
21118#
Yes
283
No1#
35 248
0#NoYes
1#441
High Risk 248 Low Risk
A
B
C
D
LEGEND# FractureA Unable to bear weight immediately and in EDB Age 55 or greaterC Bone tenderness B4 or B5D Bone tenderness B8 or B9
Recursive Partitioning of 689 Cases
34
![Page 35: Clinical Prediction Rules](https://reader030.fdocuments.net/reader030/viewer/2022032502/55b9a16dbb61eb27408b46f0/html5/thumbnails/35.jpg)
Prospective Validation of a CPR
35
![Page 36: Clinical Prediction Rules](https://reader030.fdocuments.net/reader030/viewer/2022032502/55b9a16dbb61eb27408b46f0/html5/thumbnails/36.jpg)
Problems of CPRs With Statistical Validation Only
• Many statistically derived rules fail to perform well when tested in a new population– Overfitting or instability in the original derived
model– Differences in prevalence of disease– Differences in severity of cases– Differences in how the CPR is applied
Stiell IG, et al. Annals Emergency Med 1999;33:437-47. 36
![Page 37: Clinical Prediction Rules](https://reader030.fdocuments.net/reader030/viewer/2022032502/55b9a16dbb61eb27408b46f0/html5/thumbnails/37.jpg)
Prospective Validation of a CPR
• Validation– Its repeated application leads to the same
results
• Types of validation– Narrow validation: application of rule in a
similar setting and population– Broad validation: application of rule in multiple
clinical settings with varying prevalence and outcomes of disease
McGinn TG, et al. Users’ guide to the medical literature. JAMA 2000;284:79-84. 37
![Page 38: Clinical Prediction Rules](https://reader030.fdocuments.net/reader030/viewer/2022032502/55b9a16dbb61eb27408b46f0/html5/thumbnails/38.jpg)
Development of a Clinical Prediction Rule
McGinn TG, et al. Users’ guide to the medical literature. JAMA 2000;284:79-84. 38
![Page 39: Clinical Prediction Rules](https://reader030.fdocuments.net/reader030/viewer/2022032502/55b9a16dbb61eb27408b46f0/html5/thumbnails/39.jpg)
Methodological Standards for Validation of a CPR
• Unbiased, wide spectrum patient population
• Blinded assessment of outcomes and predictor variables
• Careful follow-up of predicted normal patients
• Training for correctly applying rules
McGinn TG, et al. Users’ guide to the medical literature. JAMA 2000;284:79-84. 39
![Page 40: Clinical Prediction Rules](https://reader030.fdocuments.net/reader030/viewer/2022032502/55b9a16dbb61eb27408b46f0/html5/thumbnails/40.jpg)
Results of Validation Studies of Ottawa Ankle Rules
Markert RJ, et al. Am J Emerg Med 1998;16:564-7.
Country (Year)
# of Subjects
Sensitivity (%)
Specificity (%)
CA (1993) 1032 100 39
US (1994) 71 100 19
NZ (1994) 350 93 11
US (1994) 631 100 19
US (1995) 422 95 16
40
![Page 41: Clinical Prediction Rules](https://reader030.fdocuments.net/reader030/viewer/2022032502/55b9a16dbb61eb27408b46f0/html5/thumbnails/41.jpg)
Impact Analysis of a CPR
41
![Page 42: Clinical Prediction Rules](https://reader030.fdocuments.net/reader030/viewer/2022032502/55b9a16dbb61eb27408b46f0/html5/thumbnails/42.jpg)
Reasons for No Impact of an Accurate CPR
• Clinician’s intuition may be as good as the CPR
• Calculations involved may be cumbersome
• Practical barriers to acting on the results of CPR– Medical liability risk– Patient demand factor
McGinn TG, et al. Users’ guide to the medical literature. JAMA 2000;284:79-84. 42
![Page 43: Clinical Prediction Rules](https://reader030.fdocuments.net/reader030/viewer/2022032502/55b9a16dbb61eb27408b46f0/html5/thumbnails/43.jpg)
Methodological Standards for Impact Analysis of a CPR
• Study design– Cluster-based randomized control trial– Before-after study
• Effect on use– e.g., ordering of radiography
• Accuracy of rule
• Acceptability of physicians & patients
Stiell IG, et al. Annals Emergency Med 1999;33:437-47. 43
![Page 44: Clinical Prediction Rules](https://reader030.fdocuments.net/reader030/viewer/2022032502/55b9a16dbb61eb27408b46f0/html5/thumbnails/44.jpg)
Impact Analysis of Ottawa Ankle Rules in France
• Randomized 5 EDs to use or not use CPR• 2 in intervention group (906 patients)
– Meeting, pocket cards, posters, and data collection forms
• 3 in usual care group (1005 patients)– data collection forms only
• Results: (unit of analysis was physician)– ordering of radiography: I: 79%; C: 99% (P=.03)– I: 3/112 missed fractures (incomplete data forms: 2,
rule interpretation error by physician: 1)
Auleley GR, et al. JAMA 1997;277:1935-9. 44
![Page 45: Clinical Prediction Rules](https://reader030.fdocuments.net/reader030/viewer/2022032502/55b9a16dbb61eb27408b46f0/html5/thumbnails/45.jpg)
Summary
• Development of an effective prediction rule is a long, rigorous, and expensive process
• Properly developed and validated prediction rules can influence clinical practice
45
![Page 46: Clinical Prediction Rules](https://reader030.fdocuments.net/reader030/viewer/2022032502/55b9a16dbb61eb27408b46f0/html5/thumbnails/46.jpg)
Performance Evaluation
• Discrimination– Ability of a prediction model to separate those
who experience events from those who do not– Area under a ROC curve (c statistic)
• Calibration– Measures how closely predicted outcomes
agree with actual outcomes– Hosmer-Lemeshow goodness-of-fit test
46
![Page 47: Clinical Prediction Rules](https://reader030.fdocuments.net/reader030/viewer/2022032502/55b9a16dbb61eb27408b46f0/html5/thumbnails/47.jpg)
Hosmer-Lemeshow Goodness-of-fit Test
47
![Page 48: Clinical Prediction Rules](https://reader030.fdocuments.net/reader030/viewer/2022032502/55b9a16dbb61eb27408b46f0/html5/thumbnails/48.jpg)
48
![Page 49: Clinical Prediction Rules](https://reader030.fdocuments.net/reader030/viewer/2022032502/55b9a16dbb61eb27408b46f0/html5/thumbnails/49.jpg)
Thank You!
49