Secondary Database Analysis II

Case-Mix Adjustment

What is the Purpose of Health Services Research?

Income Ethnicity EMR CPOE

Utilization

A1c

Immunization

Medical Errors

Last Time – Uses

Secondary Data: Can provide new information on health care delivery

(quality; geographic variation; post-marketing adverse events; cost), on the natural history of disease, and on regulatory matters

Can save data-collection resources: time, money, personnel, participant burden

Can save researcher resources: your time, your money May permit you to build your CV without anyone’s help

or funds – just your time (but co-authors are a good thing)

Can provide preliminary data for a grant proposal May enable research on rare events or difficult

populations

Last Time – Precautions

Take note of the study design; is it suitable?

Consider: What are inclusion & exclusion criteria? Why? How does that affect the sample and generalizability?

This Time – Precautions

What bias may be inherent in the database – perhaps from the population or from the method of measurement?

Selection bias: who got included Migration bias: who was lost/gained and why Other sources of systematic error (bias)?

This Time – Precautions & Solutions

Control for bias – Restrict Match Stratify Adjust with covariates

ValidityTruth in the Universe

Truth in the Study

Findings in the Study

infer infer

ERRORS ERRORS

RESEARCH QUESTION

STUDY PLAN

ACTUAL STUDY

Target Population

Phenomenon of Interest

Intended Sample

Intended Variables

Actual Subjects

Actual Measurements

EXTERNAL VALIDITY

INTERNAL VALIDITY

Risk Adjustment

Nesting/Clustering of data

Bias

Systematic error in measurement or a systematic difference (other than the one of interest) between groups Selection

For cohorts, assembly, migration, contamination, and referral bias

Measurement Confounding

Restriction (may lose generalizability) Matching (limited # factors) Stratification Standardization Multivariate Adjustment Assuming the worst (sensitivity analyses)

If needed, conduct sensitivity analyses on multivariable models

>> Always discuss potential impact of uncorrected bias on your results

Bias: Anticipate and Control

Bias: Systematic Error

Selection Bias Case-Mix/Disease Severity Nesting/clustering

Measurement Bias Recall Bias Investigator Bias

11/05/03

Example: Hospital Mortality Report Cards

Originally unadjusted Hospitals without trauma centers,

doing primarily elective surgery, etc., looked really good

Made hospitals who took care of the sickest of the sick look bad

Quality Assessment

Data Quality: Garbage in, garbage out

Risk Adjustment: To remove the confounding effect of different institutions providing care to patients with dissimilar severity of illness and case complexity

Risk Adjustment

Controls for patient characteristics that are related to the outcomes of interest

Removes the confounding effect, e.g., of different institutions providing care to patients with dissimilar severity of illness and case complexity

Addresses regional variations Inadequate case-mix adjustment can lead

to misclassification of outlier status

Essential Elements of Risk Adjustment

Outcome-specific Contains specification of the principal

diagnosis Contains demographics as proxies for

preexisting physiological reserve Measures # of comorbidities and allows

all the most important comorbidities to assume their own empirically derived coefficients

Additional considerations

– outcome-specific: is one available or do you need to construct & validate one yourself?

– an index incorporates many predictors – do you need to study some separately?

– established comorbidity indices for case-mix adjustment include Charlson, Elixhauser, Selim, developed on various patient pops – suitable for yours?

– other predictors based on your reading of the literature – do you need omitted constructs?

– propensity scores (for treatment choice; good for small datasets, for non-randomized studies of treatment effect)

Risk Adjustment & Outcomes

Primary data collection vs. administrative data

Disease-specific vs. generic Commercial vs. developed for your

study Predictors vary by outcomes being

predicted

Classification of Disease States

ICD-9: too many specific codes (n~10,000) Clinical Classifications for Health Policy

Research (CCHPR): good for chronic illness and longitudinal care [http://www.ahrq.gov/data/hcup/his.htm]

Primary diagnosis: good for studies that focus on a single episode of care

Famous Methods of Risk Adjustment

DRGs: diagnosis related groups Used by Medicare to set hospital

reimbursement APACHE III

Adult ICU PRISM

Pediatric ICU Charlson Score

Adult 1 year survival after hospitalization

Risk Adjustment: Charlson

Advantages: Commonly used case-mix classification

system in the health care industry System with which most clinicians and

reviewers are familiar

Risk Adjustment: Charlson

Disadvantages Principal diagnosis not differentiated Original work did not specify ICD-9 codes

that went into the disease categories Developed on inpatients predicting

mortality; may not be well suited to outpatients at low risk of death

Not good for longitudinal care / chronic illness

Demographic Factors in Risk Adjusters

Age (e.g., age-adjusted Charlson) Proxies of Social Support

Marital status Race Gender SES (occupation, employment status,

education) Proxies of Socioeconomic Status

Health insurance status Home address zip code average income

Race and Gender

Don’t adjust for automatically Ideally adjust for variation in the

patients’ physiological reserve and disease burden but not for variation in care rendered to patients

Propensity Scores

Useful when sample size is small, to conserve power

An alternative to including a lot of covariates Ask: propensity for what? Include as many predictors as possible to get

predicted probability of group membership (Rosenbaum & Rubin)

Published schema may include predictors you want to study separately

Best for non-randomized studies of treatment effect where you want to adjust for the factors that may have influenced the treatment choices

Study Design: Minimize Bias

Decision #1:

Alter events under study?

Decision #2:

Make measurement on more than one occasion?

Experimental Study Apply intervention, observe effect on outcome

yes

Observational Study

no

Cross-Sectional Study Each subject examined on only one occasion

no

yes

Longitudinal Study Each subject followed over a period of time

Case-control Cohort

Risk adjustment is needed…

When subjects are not randomly assigned People do not randomly distribute by

Setting Provider

Risk Adjust Outcomes = f (intrinsic pt factors; treatment

applied; quality of treatment; random chance)

On What Factors Should We Risk Adjust?

Risk of what outcome? Ejection Fraction Readmission Activities of Daily Living (feeding, dressing, etc.)

For what population? Inpatient Outpatient Nursing Home

For what purpose? Clinical Quality Payment

Classes of potential risk factors

Demographics: age, sex, etc. Physiologic status Number and type of medical diagnoses Cognitive and mental status Sensory function Social, economic, environmental factors Functional/overall health status

An Important Distinction

Disease Severity Case-mix or co-morbidity

What is the difference?

Disease Severity v. Case Mix

0

10

20

3040

50

60

70

HTN DM COPD Arthritis

case-mix

disease severity

Patient A

Patient B

Nesting or Clustering of Observations

A threat to validity

Nesting/Clustering of Observations

Traditional methods of analysis assume observations are independent: people who see same MD are not!

By setting Multicenter studies By clinic By clinician

Key issue is understanding SOURCE of variance in the observed data Within group (clinic) Between groups (clinics)

Survival by OR Team

0

10

20

30

40

50

60

0 2 4 6 8 10

cases per month

10 y

r su

rviv

al

OR 1

OR 2

OR 3

Common Terminology

Intra-class Correlation Coefficient (ICC) The extent to which individuals within the

same group are more similar to each other than they are to individuals in different groups

The proportion of the true variation in the outcome that can be attributed to differences between the clusters

ICC Values

20 primary care clinics 30-32 patients with type 2 DM per

clinic For process of care measures (foot

exam, labs ordered etc.) ICC = 0.32 For A1c values, ICC = 0.12

Question: Which is more dependent upon site of care: processes of care or outcomes of care (A1c control)?

Sample Size Implication

Design Effect = 1 + (n-1)ICC(n= number of subjects per cluster)

So if estimated sample size of 300 per group for an intervention study, what sample size would you need for 30 subjects per cluster (10 clinics) and ICC of 0.12?

Assignment

Design a study: Experimental Cross-sectional Prospective cohort Case-Control

What factors you would measure to “risk adjust” and how measure them?

How would you would adjust for nesting/clustering of data?

Questions?

Common Research Design Issues in Health Services Research