Royal College of Surgeons in Ireland Coláiste Ríoga na Máinleá in Éirinn

Division of Population Health Sciences

Royal College of Surgeons in IrelandColáiste Ríoga na Máinleá in Éirinn

Indices of Performances of CPRs

Nicola Motterlini, Borislav D Dimitrov, Tom Fahey

1


Contents

• CPR derivation– Analytical approach

• CPR validation– Indices of discrimination– Indices of calibration

• Performance of a CPR in several validation studies (systematic review with meta-analysis)

2


Figure 2. Development of a Clinical Decision Rule

Level of Evidence

4 3 2 1

Step 2. Validation

Evidence of reproducible accuracy

Narrow Validation Broad Validation

Application of a rule in a Application of rule in Similar clinical setting and multiple clinical settings Population as Step 1 and varying prevalence and outcomes of disease

Step 3. Impact Analysis

Evidence that rule change physician behaviour and improves patient outcomes and/or reduces costs

Step 1. Derivation

Identification of factors with predictive power

Stages of development of a CPR

3


Multivariable approach(e.g., logistic regression model)

Analytical approachto CPR(an example)

Target disorder(presence or absence)

4

Prediction LOW RISK %Total score ≤ 3

MODERATE RISK %4 ≤ Total score ≤ 7

HIGH RISK %Total score ≥ 8

Decision NO TREATMENT FURTHER DIAGNOSTIC TESTING

TREATMENT

“Weights”(e.g. odds

ratio)

1.9 (1.4-2.5)5.4 (4.1-7.1)16.2 (12.2-

21.6)

5.7 (4.3-7.5)

1.8 (1.5-2.1)

2.3 (1.8-3.1)

18.2 (14.2-23.6)

Points

123

2

1

2

3

Total score:3+2+1= 6

Independent risk factors

Age (years) 60-74 75-84 ≥85

Sex male

Diabetes mellitus

Heart failure

Current smoker


Thresholds approach to diagnosis

Diagnosis / test thresholdDiagnosis / test threshold

Test / reassurance thresholdTest / reassurance threshold

Probability of disease

100%HIGHRISK

0%LOWRISK

Treatment

Further diagnostic testing

No treatment

5

10987654321

Score


Figure 2. Development of a Clinical Decision Rule

Level of Evidence

4 3 2 1

Step 2. Validation

Evidence of reproducible accuracy

Narrow Validation Broad Validation

Application of a rule in a Application of rule in Similar clinical setting and multiple clinical settings Population as Step 1 and varying prevalence and outcomes of disease

Step 3. Impact Analysis

Evidence that rule change physician behaviour and improves patient outcomes and/or reduces costs

Step 1. Derivation

Identification of factors with predictive power

Stages of development of a CPR

6


Performance of a CPR can be assessed like a performance of a diagnostic test!

TP, number of true positives, FP, number of false positives, FN, number of false negatives, TN, number of true negatives, NT+, the number of patients with positive test results, NT-, the number of patients with negative test results, ND+, the number of patients with the disease, and ND−, number of patients without the disease

Disease+

Disease-

Total

Test +score ≥ Cut-off

TP FP NT+

Test -score < Cut-off

FN TN NT-

Total ND+ ND- N

7


Indices of performance

1. For discrimination (classification)– Ability to discriminate between diseased

individuals and non-diseased individuals (e.g., sensitivity, specificity, ROC curve, odds ratio, positive and negative predictive values, likelihood ratio etc.)

2. For calibration (comparison)– Agreement between predicted probabilities and

observed probabilities (e.g., risk ratio, Hosmer-Lemeshow goodness-of-fit test etc.)

8


Sensitivity = TP/ND+

Specificity = TN/ND-

Disease+

Disease-

Total

Test +score ≥ Cut-off

TP FP NT+

Test -score < Cut-off

FN TN NT-

Total ND+ ND- N

1. Discrimination – sensitivity and specificity

9


1 2 3 4 50

5

10

15

20

25

30

Disease

risk (%)

Predicted

Observed

Risk ratio ->

Low Moderate High

Score risk

2. Calibration – risk ratio

Predicted

Observed

<1 “under-prediction”=1 “accurate prediction” >1 “over-prediction”

10


Performance of a CPR in several validation studies(systematic review with meta-analysis)

11


Pooled analysis

• When the 95% CI includes 1 there is no significant statistical difference between the observed and predicted by CPR

• The smaller the interval, the more accuracy are the results

12


Index of heterogeneity

I2 value over 50% is a strong indicator for heterogeneity

13


Summary

• CPR derivation– Multivariable approach to develop a CPR

• CPR validation– Indices of discrimination (sensitivity,

specificity)– Indices of calibration (risk ratio)

• Systematic review with meta-analysis of validation studies of CPRs

14

Royal College of Surgeons in Ireland Coláiste Ríoga na Máinleá in Éirinn

Documents

Transcript of Royal College of Surgeons in Ireland Coláiste Ríoga na Máinleá in Éirinn