Transcript of How to Design and Interpret Observational Outcomes Studies in Cardiovascular Disease Nathan D. Wong,...
- Slide 1
- How to Design and Interpret Observational Outcomes Studies in
Cardiovascular Disease Nathan D. Wong, PhD, FACC Professor and
Director Heart Disease Prevention Program Division of Cardiology,
UC Irvine Adjunct Professor of Epidemiology, UCLA and UC Irvine
President, American Society for Preventive Cardiology
- Slide 2
- Why are papers rejected for publication? (The Top 11 Reasons)
1.The study did not address an important scientific issue 2.The
study was not original 3.The study did not actually test the
authors hypothesis 4.A different type of study should have been
done 5.Practical difficulties led the authors to compromise on the
original study protocol (e.g., recruitment, procedures) Greenhalgh
T, BMJ 1997; 15: 243-6
- Slide 3
- Reasons 6-11 for Paper Rejection 6.The sample size was too
small 7.The study was uncontrolled or inadequately controlled 8.The
statistical analysis was incorrect or inappropriate 9.The authors
drew unjustified conclusions from the data 10.There is a
significant conflict of interest among authors 11.The paper is so
badly written that it is incomprehensible
- Slide 4
- Critical Appraisal 1.Why was the study done, and what clinical
question is being asked? (a brief background, review of the
literature, and aim / hypothesis should be stated) 2.What type of
study was done? (experiment, clinical trial, observational cohort
or cross-sectional study, or survey)
- Slide 5
- Critical Appraisal (cont.) 3. Was the design appropriate for
the research? Clinical trial preferred to test efficacy of
treatments Cross-sectional study preferred for testing validity of
diagnostic/screening tests or risk factor associations Longitudinal
cohort study preferred for prognostic studies Case-control study
best to examine effects of a given agent in relation to occurrence
of an illness, esp. rare illnesses (e.g., cancer)
- Slide 6
- Outline Elements of Designing a Research Protocol Concepts of
Study Design: Observational cross-sectional, case-control, cohort
studies Advantages and Disadvantages of Different Study Designs
which is right for you? Analysis of Observational Studies
- Slide 7
- Nine Key Elements of a Research Study Protocol Background
Hypotheses Clinical Relevance Specific Aims / Objectives
Methodology Power / Sample Size Measures and Outcomes Data
Management Statistical Methodology (UCI School of Medicine
Scientific Review Committee)
- Slide 8
- Background A brief review of the problem to be studied and of
related studies that generated the rationale and the central idea
of the proposed study. Several pertinent references should be
provided.
- Slide 9
- Was the study original? Few studies break entirely new ground
Many studies add to the evidence base of earlier studies which may
have had other or more limitations Meta-analyses depend on
literature containing multiple studies addressing a question in a
similar manner
- Slide 10
- Features Distinguishing New vs. Previous Studies Sample size
Length of follow-up More rigorous methodology Different population
studied different from that of previous studies (ages, gender,
ethnic groups)? Does the new study address a clinical issue of
sufficient importance? Greenhalgh T, BMJ 1997; 315: 305-8
- Slide 11
- Specific Aims / Objectives What the study is intended to study
or demonstrate; includes mention of predictor and outcome (or
endpoint) variables. For example: "The primary aim of the study is
to examine whether treatment A is more effective than treatment B
in reducing levels of C", or "in finding out whether X is
associated with Y", etc. There may both principal and secondary
aims
- Slide 12
- Elements of a Formulated Question Patient or Population: Who is
the question about? (e.g., pts with diabetes mellitus) Intervention
or Exposure: What is being done or what is happening to the
patient/population? (e.g., tight control) Outcome(s): How does the
intervention affect the patient/population (mortality, CHD
incidence) Comparison(s): What could be done instead of the
intervention? (e.g., standard management)
- Slide 13
- Hypotheses The problem/s stated in the Background may generate
a primary hypothesis and possibly one or two secondary hypotheses.
A hypothesis is often stated in the null e.g., "No difference
between treatments A and B" is anticipated, or "No association
between X and Y exists". Alternatively, it can be stated according
to what one expects e.g., A will be more effective than B in
reducing levels or symptoms of C", or X will be associated with
Y".
- Slide 14
- Clinical / Community Relevance In the case of clinical studies,
the potential value in the understanding, diagnosis, or management
of a clinical condition or pathological state should be stated.
Funding agencies often now require a statement of community
relevance e.g., how will the results be translated and disseminated
to the target population or community.
- Slide 15
- Methodology Methodology should validate or not validate the
hypothesis and specific aims using procedures consistent with sound
scientific study design including: the size and nature of the
subjects studied recruitment, screening, and enrollment procedures
inclusion and exclusion criteria treatment schedules, and follow-up
procedures, if applicable. A chart of the studies to be performed
at each visit and the time of each visit and test is needed.
- Slide 16
- Study Population Issues How were the subjects recruited? Is
there potential recruitment bias (e.g., from taking respondents of
advertisements), or is survey done in a random (e.g., random
digit-dialing) or consecutive sample? Who was included? Many trials
exclude those who have co-morbidities, do not speak English, or
take other medicationsmay provide scientifically clean results, but
may not be representative of disease in question.
- Slide 17
- Study Population (cont.) Who was excluded? Study may exclude
those with more severe forms of disease, therefore limiting
generalizibility Were subjects studied in real-life circumstances?
Is the consenting process describing the benefits/risks, access to
study staff, equipment available, etc. be similar to that in an
ordinary practice situation?
- Slide 18
- Power / Sample Size A power/sample size analysis should include
an estimate of minimum effect or difference expected at a given
level of power when the sample size is fixed, or a projection of
the number of subjects needed to achieve a clinically important
difference in what is being examined in the hypotheses and the
specific aims.
- Slide 19
- Measures and Outcomes Includes both independent (predictor) and
dependent (outcome) variables. Outcomes include what the
investigator is trying to predict, e.g., new or recurrent onset of
a disease state, survival, or lowering of cholesterol. The
independent or predictor variables should always include treatment
status (e.g., active vs. placebo) in the case of a clinical trial,
or primary variables of interest (such as age, gender, levels of X
at baseline) for other studies. The measures and outcomes should
expect to answer the proposed question and the importance of the
knowledge expected from the research.
- Slide 20
- Data Management Data Management includes how data is captured
for analysis and the tools that will be utilized while capturing
the data. This includes: Case report forms for clinical trials
Surveys, questionnaires, or interview instruments Computerized
spreadsheets or entry forms Methods for data entry, error checking,
and maintenance of study databases
- Slide 21
- Statistical Methods of Analysis Statistical analysis includes a
description of the statistical tests planned to perform to examine
the results obtained, e.g., Students t-test will be used to compare
levels of A and B between treatment and placebo groups Multiple
logistic regression analysis will be used to examine an independent
treatment effect on the likelihood of recurrent disease.
- Slide 22
- Hierarchy of Evidence (for making decisions about clinical
interventions or proving causation) 1.Systematic reviews and
meta-analyses 2.Randomized controlled trials with definitive and
clinically significant effects 3.Randomized controlled trials with
non- definitive results 4.Cohort studies 5.Case-control studies
6.Cross-sectional surveys 7.Case reports
- Slide 23
- Features Affecting Strength and Generalizability of Study
sample size selection of comparison group (control or placebo)
selection of study sample (is it representative of population the
study results are intended to apply to?) length of time of
follow-up outcome assessed (e.g., hard vs. soft or surrogate
endpoint) Measurement and ability to control for potential
confounders
- Slide 24
- Case Reports and Series Provides anectdotal evidence about a
treatment or adverse reaction Often with significant detail not
available in other study designs May generate hypotheses, help in
designing a clinical trial. Several reports forming a case series
can help establish efficacy of a drug, or thru adverse reports,
cause its demise (example: Cerivastatin fatal cases of
rhabdomyolysis).
- Slide 25
- Observational Studies Cross-sectional, prospective, and case-
control studies seldom can identify two groups of subjects (exposed
vs. unexposed or cases vs. controls) that are similar (e.g., in
demographic or other risk factors). Much of the controlling for
baseline and/or follow-up differences in subject characteristics
occurs in the analysis stage (e.g., multivariable analysis as in
Framingham)
- Slide 26
- Observational Studies (cont.) While statistical procedures may
be done correctly, have we considered all possible confounders?
Some covariates may not have been measured as accurately as
possible, and more often, may not be even known or measured.
- Slide 27
- Observational, cross-sectional Examines association between two
factors (e.g, an exposure and a disease state) assessed at a single
point in time, or when temporal relation is unknown Example:
Prevalence of a known condition, association of risk factors with
prevalent disease. Conclusions: Associations found may suggest
hypotheses to be further tested, but are far from conclusive in
proving causation
- Slide 28
- Cross-Sectional Studies and Surveys Examples: NHANES III, CHIS
(telephone), chart-review studies Surveys should include a
representative, ideally randomly-chosen (rather than a small sample
of approached subjects who actually agree to be surveyed) sample.
Data collected cannot assume any directionality in exposure /
disease. Can statistically adjust for confounders, but difficult to
establish the temporal nature of exposure and disease.
- Slide 29
- Prevalence of CHD by the Metabolic Syndrome and Diabetes in the
NHANES Population Age 50+ CHD Prevalence % of Population = No MS/No
DM 54.2% MS/No DM 28.7% DM/No MS 2.3% DM/MS 14.8% 8.7% 13.9% 7.5%
19.2% Alexander CM et al. Diabetes 2003;52:1210-1214..
- Slide 30
- Prospective (Cohort) Studies Cohort studies begin with
identification of a population, assessment of exposure (e.g., lipid
or BP levels) Follow-up to the occurrence of outcomes (CHD
events)-- temporal sequence (e.g, follow-up time) to events is
known
- Slide 31
- Cohort Studies (cont.) Difficult to ascertain effect of
exposure because of many differences between exposed and unexposed
groups (confounding factors). Statistical adjustment for known risk
factor differences can help, but unknown factors that may differ
between exposed and unexposed groups will never be adjusted
for.
- Slide 32
- Duration of Follow-up Is the planned follow-up reasonable and
practical for the study question and sample size utilized? effect
of a new painkiller on degree of pain relief may only require 48
hours effect of a cholesterol medication on mortality may require 5
years
- Slide 33
- Prospective cohort studies Examples: Framingham Heart Study
Cardiovascular Health Study (CHS) Multiethnic Study of
Atherosclerosis (MESA) Nurses Health Study Advantages: large sample
size ability to follow persons from healthy to diseased states
temporal relation between risk factor measures and development of
disease
- Slide 34
- Prospective Studies (cont.) Disadvantages: expensive due to
large sample size often needed to accrue enough events many years
to development of disease possible attrition causal inference not
definitive as difficult to consider all potential confounders
- Slide 35
- Framingham Heart Study Longest running study of cardiovascular
disease in the world Began in 1948 with original cohort of 5,209
subjects aged 30-62 at baseline Biennial examinations, still
ongoing, most of original cohort deceased Offspring cohort of 5,124
of children of original cohort enrolled in 1971, and more recently
and still being enrolled to better understand genetic components of
CVD risk are up to 3,500 grandchildren of the original cohort.
Routine surveillance of cardiovascular disease events adjudicated
by panel of physicians
- Slide 36
- Framingham Most Significant Milestones 1960 Cigarette smoking
found to increase the risk of heart disease 1961 Cholesterol level,
blood pressure, and electrocardiogram abnormalities found to
increase the risk of heart disease 1967 Physical activity found to
reduce the risk of heart disease and obesity to increase the risk
of heart disease 1970 High blood pressure found to increase the
risk of stroke 1976 Menopause found to increase the risk of heart
disease 1978 Psychosocial factors found to affect heart disease
1988 High levels of HDL cholesterol found to reduce risk of death
1994 Enlarged left ventricle (one of two lower chambers of the
heart) shown to increase the risk of stroke 1996 Progression from
hypertension to heart failure described
- Slide 37
- Low HDL-C Levels Increase CHD Risk Even When Total-C Is Normal
(Framingham) Risk of CHD by HDL-C and Total-C levels; aged 4883 y
Castelli WP et al. JAMA 1986;256:28352838 0 2 4 6 8 10 12 14 <
4040495059 60 < 200 230259 200229 260 HDL-C (mg/dL) Total-C
(mg/dL) 14-y incidence rates (%) for CHD 11.24 11.91 12.50 11.91
6.56 4.67 9.05 5.53 4.85 4.15 3.77 2.78 2.06 3.83 10.7 6.6
- Slide 38
- Cardiovascular Health Study 5,201 Medicare eligible individuals
aged 65-102 at baseline enrolled beginning 1992 at six field
centers. Assessment of newer and older risk factors. Ongoing
follow-up of cardiovascular events and mortality Subclinical
disease measures included: carotid B-mode ultrasound for carotid
IMT at Year 2, Year 7, and Year 11 m-mode echocardiographic
measures of left ventricular mass and dimensions, left atrial
dimension done at baseline (Year 2) (at UC Irvine) and follow-up
(Year 7) examinations. Ankle brachial index (ABI) for measurement
of PAD Pulmonary function (FVC and FEV1)
- Slide 39
- ProcedureBAS E Call B YR 3 Call 3 YR 4 Call 4 Tracking
UpdateXXXXXX Stressful Life EventsXXXXXX Depression ScaleX X X
Quality of LifeX X X Social Support and Network X X X Medications -
PrescriptionX X X OTC Physical Function: ADL/IADL X XXXX Cognitive
Function - MMSE X 3MSE X X Digit Symbol Substitution X X X Benton
Visual Retention PhlebotomyX Anthropometry - WeightX X X Standing
HeightX Waist CircumferenceX Hip CircumferenceX Arm Span
- Slide 40
- Cardiovascular Health Study: Combined intimal-medial thickness
predicts total MI and stroke Cardiovascular Health Study (CHS)
(aged 65+): MI or stroke rate 25% over 7 years in those at highest
quintile of combined IMT (OLeary et al. 1999)
- Slide 41
- Case-control Studies Most frequent type of epidemiologic study,
can be carried out in a shorter time and require a smaller sample
size, so are less expensive Only practical approach for identifying
risk factors for rare diseases (where follow-up of a large sample
for occurrence of the condition would be impractical) Selection of
appropriately matched control group (e.g., hospital vs. healthy
community controls) and consideration of possible confounders
crucial Relies on historical information to obtain exposure status
(and information on confounders)
- Slide 42
- Case-Control Studies (cont.) Cannot determine for sure whether
exposure preceded development of disease Also difficult to identify
all differences between cases and controls that can be
statistically adjusted for
- Slide 43
- Example of case-control study: Folate and B6 intake and risk of
MI (Tavani et al. Eur J Clin Nutr 2004) Cases were 507 patients
with a first episode of nonfatal AMI, and controls were 478
patients admitted to hospital for acute conditions Information was
collected by interviewer- administered questionnaires Compared to
patients in the lowest tertile of intake, the ORs for those in the
highest tertile were 0.56 (95% CI 0.35-0.88) for folate and 0.34
(95% CI 0.19-0.60) for vitamin B6. Author conclusion: A high intake
of folates, vitamin B6 and their combination is inversely
associated with AMI risk
- Slide 44
- Potential sources of bias and error in case control studies
Information on the potential risk factor or confounding variables
may not be available from records or subjects memories Cases may
search for a cause of their disease and be more likely to report an
exposure than controls (recall bias) Uncertainty as to whether
agent caused disease or whether occurrence of the disease caused
the person to be exposed to the agent Difficulty in assembling a
case group representative of all cases, and/or assembling an
appropriate control group
- Slide 45
- Prospective, observational: nested case-control In this design,
one takes incident cases (e.g., incident CVD) and a matched set of
controls to examine the association of a risk factor measured
sometime before development of the outcome of interest Less costly
than a true prospective design where all subjects are included in
analysis; may not provide equivalent estimates
- Slide 46
- Prospective study of CRP and risk of future CVD events among
apparently healthy women (Ridker et al., Circulation 1998) a nested
case control study 122 female pts who suffered a first CVD event
and 244 age and smoking-matched controls free of CVD Logistic
regression estimated relative risks and 95% CIs, adjusted for BMI,
diabetes, HTN, hypercholesterolemia, exercise, family hx, and trt
Those who developed CVD events had higher baseline CRP than
controls; those in the highest quartile of CRP had a 4.8-fold (4.1
adjusted) increased risk of any vascular event. For MI or stroke,
RR=7.3 (5.5 adjusted)
- Slide 47
- hs-CRP Adds to Predictive Value of TC:HDL Ratio in Determining
Risk of First MI Total Cholesterol:HDL Ratio Ridker et al,
Circulation. 1998;97:20072011. hs-CRP Relative Risk
- Slide 48
- Examples where observational studies have taken us down the
wrong path Meta-analysis of observational studies have shown a 50%
lower risk of CHD among estrogen users vs. non-users (which may
have had many unknown differences that were not adjusted for), but
recently randomized trials (HERS, WHI) show no benefit Numerous
prospective studies show a 25-50% lower risk of CHD among those
taking vitamin E and other antoxidants vs. placebo recent
randomized trials (e.g., HOPE, HPS) show no benefit.
- Slide 49
- Randomized Clinical Trial Considered the gold standard in
proving causation e.g., by reducing putative risk factor of
interest Randomization equalizes known and unknown
confounders/covariates so that results can be attributed to
treatment with reasonable confidence Inclusion and exclusion
criteria can often be strict (to maximize success of trial) and may
require screening numerous patients for each patient
randomized
- Slide 50
- Randomized Clinical Trials (2) Expensive, labor intensive,
attrition from loss to follow-up or poor compliance can jeopardize
results, esp. if more than outcome difference between groups
Conditions are highly controlled and may not reflect clinical
practice or the real world Funding source of study and commercial
interests of investigators can raise questions about conclusions of
study
- Slide 51
- Randomized Controlled Trials (3) Randomized controlled trial
eliminates systematic bias (in theory) by allocating treatments
among participants in a random fashion The allocation process
eliminates selection bias in group characteristics (check
comparability of baseline characteristics such as age, gender,
severity of disease and covariate risk factors) (selection
bias)
- Slide 52
- Questions to Ask Regarding Statistical Analysis Was there
sufficient power/sample size? Was the choice of statistical
analysis appropriate? Was the choice (and coding/classification) of
outcome and treatment variables appropriate? Is there an adequate
description of magnitude and precision of effect? Was there
adjustment for potential confounders? Have the results been
correctly interpreted and not overstated?
- Slide 53
- Statistical significance and power Statistical significance is
based on the Type I or Alpha error the probability of rejecting the
null hypothesis when it was true (saying there was a relationship
when there isnt one) usually we accept being wrong
- Odds of CVD Stratified by CRP Levels in U.S. Persons (Malik and
Wong et al., Diabetes Care, 2005) * p