External Validation of a Prognostic Model to Predict Mortality After Traumatic Brain Injury

Dhaval Shukla*, Akhil Deepika*, GS Umamaheshwar Rao#, DK Subbukrishna@

Departments of Neurosurgery*, Neuroanesthesiology#, and Biostatistics @

NIMHANS, Bangalore

Prediction Models

• Statistical models that combine two or more items of patient data to predict outcome

• Two requirements– clinically valid– methodologically valid

• More reliable than what doctors can foretell• Influence patient management

Hierarchy of Prediction Models

• Univariate Analysis

• Multivariate Analysis

• Logistic Regression Analysis

• Discriminant Analysis

• Web Based Calculator

Clinical Predictors

• GCS

• Motor Response

• Pupillary Reaction

• Ocular Movements

• Blood Pressure

CT Scan Predictors

• Midline Shift

• Cisterns

• Ventricles

• Hematomas

• Petechial Hemorrhages

Biochemical Predictors

• Oxygen

• Hemoglobin

• Glucose

102 Prediction Models of TBI

• Small Sample Size

• Logistic Regression

• 93% High Income Countries

• 11% External Validation

• 19% User-friendly

Perel, et al. BMC Medical Informatics and Decision Making 2006

External Validation of CRASH Prediction Model from IMPACT Dataset

Perel, et al. BMJ 2008

External Validation

King: Tell me about my futureSoothsayer 1: Your all relatives will DIE in front

of your eyesKing punishes him.

Soothsayer 2: You will LIVE longestKing rewards him.

Its only human to cross check if someone predicts bad about you

External Validation

• The performance of a model on a different population

(‘generalizability’ or ‘transportability’)

• CRASH model not validated in middle/ low income country

BMJ 2008;336:425

• Review of clinical and CT Scan data of consecutively admitted TBI patients in ICU over 6 months

DATA COLLECTION

• Univariable Logistic Regression Analyses (LRA)

• Multivariate LRAMODEL

CONSTRUCTI ON

• Discrimination• Calibration

MODEL PERFORMANCE

• Bootstrap MethodVALIDATION

Indicates how closely predicted outcomes match observed outcomes

Resample from the sample data at hand for approximating sampling distribution of a statistic and bias correction

Describes how well a model distinguishes between those who die from those who survive0.90-1 is Excellent

Demographics

• Total no of patients: 150

• Male : Female :: 5.5 : 1

• Age range: 1 to 85

• Mean ICU stay: 8.3±7.2 days

• Mortality: 15.3%

• Time to death: 7.52±4.56 days

Variable CRASH OR (95%CI)

NIMHANS OR (95%CI)

p Value

Age 1.46 (1.39 to 1.54) 0.99 (0.95 to 1.04)

GCS 1.27 (1.24 to 1.31) 2.14 (1.27 to 3.60) 0.004

Pupil Reaction Both One None

11.45 (1.14 to 1.86)3.12 (2.46 to 3.97)

11.30 (0.3 to 43.7)1.23 (0.4 to 32.66)

Extracranial Injury 1.08 (0.91 to 1.28)

CT Scan

Petechial Hemorrhages 1.26 (1.07 to 1.47) 0.81 (0.16 to 4.00)

Obliteration of 3rd Ventricle/ Basal Cisterns

1.99 (1.69 to 2.35) 7.32 (1.27 to 42.14) 0.026

SAH 1.33 (1.14 to 1.55) 0.98 (0.23 to 4.17)

Midline Shift 1.78 (1.44 to 2.21) 0.42 (0.06 to 2.63)

Non Evacuated Hematoma

1.48 (1.24 to 1.76) 0.70 (0.00 to 1.70)

Complications in ICU 0.04 (0.00 to 0.29) 0.001

Model Construction

• CRASH Variables • 3 variables from univariate analysis

– (Pre ICU GCS, Intubation, Complication )

Result• Pre ICU GCS (P_GCS)• Obliteration of 3rd ventricle/ Basal Cistern (OB)• Complication during stay (CD)

DiscriminationVariables Adjusted OR 95% Confidence Interval

Lower Upper

OB 3.565 1.069 11.882

P_GCS 1.819 1.242 2.662

CD 0.053 0.011 0.262

Performance of Model

Area : 0.925, 95% CI : 0.877 – 0.973

Calibration

Calibration: Chi2 : 4.796, Significance: 0.685

Internal Validation

Predi cted group * OUTCOME Crosstabul at i on

13 3 1672. 2% 4. 0% 17. 2%

5 72 7727. 8% 96. 0% 82. 8%

18 75 93100. 0% 100. 0% 100. 0%

Count% wit hin OUTCOMECount% wit hin OUTCOMECount% wit hin OUTCOME

DEAD

SURVI VED

Predict edgroup

Tot al

DEAD SURVI VEDOUTCOME

Tot al

Overall Accuracy 91.4%

Explanation

• CRASH model never validated for developing countries

• Only ICU patients were sampled• Many patients underwent surgery• Pre ICU GCS

Conclusions

• Prediction models based on large population studies may not be valid for a selected group of patients

• Each intensive care should have their own prediction models, which should be revised when services improve

Don't ever prophesy;

for if you prophesy right nobody will remember you,

and if you prophesy wrong, nobody will forgive you

- Josh Billings