External validity: to what populations do our study results apply ?

External validity: to what populations do our

study results apply?

Epidemiology matters: a new introduction to methodological foundations

Chapter 12

2Epidemiology Matters – Chapter 1

Seven steps

1. Define the population of interest2. Conceptualize and create measures of exposures and health

indicators3. Take a sample of the population4. Estimate measures of association between exposures and health

indicators of interest

5. Rigorously evaluate whether the association observed suggests a causal association

6. Assess the evidence for causes working together7. Assess the extent to which the result matters, is externally valid,

to other populations

Epidemiology Matters – Chapter 12

Generalizability or external validity refers to our capacity to generalize our results beyond our study sample


Question: How can we assess the extent to which results of a study are applicable in populations outside of the underlying population base of particular study?Answer: Think through characteristics of population of interest and determine how robust study findings might be across populations with similar or different characteristics.


1. Validity, four stages

2. Introduction to external validity

3. Prevalence of component causes

4. Causation and study design

5. External versus internal validity

6. Randomized control trials

7. Representative samples

8. Summary


Four types of validity

1. Measurement validity2. Statistical conclusion validity3. Internal validity4. External validity


1. Measurement validity

An association cannot be valid beyond the study sample unless it is valid within the study sample

Accuracy and precision are key measurements Have we measured what we wanted to measure?


2. Statistical conclusion validity

Is the association observed due to chance? We assess this via confidence intervals around

estimates of association to describe role of sampling variability

We aim to rule out the potential that our results arose by chance in the sampling process from an underlying population of interest


3. Internal validity

Assessment of non-comparability between exposed and non-exposed in any study


4. External validity

Explore external validity after assessing and ensuring Measurement validity Statistical conclusion validity Internal validity









8. Summary


External validity

External validity: the applicability of study findings beyond the study sample

In an epidemiologic study we

1. Identify a population of interest

2. Sample from population – random or purposive

3. Conduct study

4. Sample result should reflect underlying association in population of interest

Therefore, identifying population of interest is central to exploring external validity once we have our findings









8. Summary


Prevalence of component causes

To understand external validity we must understand the prevalence and distribution of component causes across populations.


Prevalence of component causes, example

Question: Does exposure to ambient air pollution cause lung cancer? Component cause A: Ambient air pollution and smoking;

therefore, smoking will cause lung cancer only among individuals exposed to ambient air pollution

Component cause B: Genetics


Non-diseased Diseased

Non-exposed Exposed air pollution

Exposed genetic, diseased Exposed genetic,

diseased, exposed air pollution, smoker

Smoker Exposed air pollution, diseased,

smoker


Prevalence of component causesexample lung cancer

Population 1

Exposed to air pollution and genetic

(regardless of disease status and smoking status) 2

Un-exposed to air pollution and exposed to genetic

(regardless of disease status and smoking status)2

Exposed to air pollution and smoker

(regardless of disease status and genetic status) 6

Black = exposed to air pollutionDots = genetically determinedHat = smoker

Un-exposed to air pollution and smoker

(regardless of disease status and genetic status)6


Prevalence of component causesexample lung cancer

Population 2

Exposed to air pollution and genetic

(regardless of disease status and smoking status) 2

Un-exposed to air pollution and exposed to genetic

(regardless of disease status and smoking status)2

Exposed to air pollution and smoker

(regardless of disease status and genetic status) 3

Black = exposed to air pollutionDots = genetically determinedHat = smoker

Un-exposed to air pollution and smoker

(regardless of disease status and genetic status)3


Prevalence of component causesexample lung cancer risk difference

Population 2

Black = exposed air pollutionDots = genetically determinedHat = smoker

Population 1

Exposed diseased = 3 = 30% riskUnexposed diseased = 2 = 20% riskRisk difference: 30% – 20% = 10%

Interpretation: 10 cases of lung cancer are associated with ambient air pollution per 100 exposed

Exposed diseased = 6 = 60% riskUnexposed diseased = 2 = 20% riskRisk difference: 60% – 20% = 40%

Interpretation: 40 cases of lung cancer are associated with ambient air pollution per 100 exposed


Prevalence of component causesexample lung cancer interpretation

Two studies asked the same question Both are internally valid studies because the exposed and unexposed are

comparable on genetic determinism Population 1: the causal effect is a risk difference of 40% Population 2 : the causal effect is a risk difference of 10%

Why do these two causal effects differ?

Prevalence of people exposed to ambient air pollution is the same in both studies Prevalence of genetic determinism is same in both studies The reason that these two causal effects diverge is the different prevalence of

smoking between the two populations


Prevalence of component causesexample lung cancer interpretation

When two causes interact the measure of association for the effect of one cause on the outcome will differ across levels of the second cause

Air pollution example Ambient air pollution and smoking are causal partners within the same

sufficient cause Prevalence of one of them (smoking) influences the causal effect of the other

(ambient air pollution) on the outcome (lung cancer) We would therefore expect the causal effect of ambient air pollution on lung

cancer to differ across population where prevalence of smoking also differs Therefore, there is no one causal effect for all populations; the causal effect is

dependent on prevalence of component causes in each population Therefore, the result from one study will be externally valid to populations in

which the distribution of component causes of exposure is similar to the study sample









8. Summary


Causation and study design

The magnitude of an association will be applicable beyond our study to the extent that the distribution of causal partners of exposure is similar in the population

If we want to identify a cause of disease, should it be a cause absolutely and in all types of populations?


Causation and study design

Cause: a factor that was necessary for that disease to occur in an individual at that time; most causes are insufficient and unnecessary in isolation

Causal effect: epidemiology studies populations; therefore we focus on the effect of causes

We document an association between those who embody cause (exposed) and those who do not (unexposed); this is context specific and dependent on prevalence of component causes

Therefore, understanding a cause in context of causal partners is central to theory, design, and analysis









8. Summary


External vs. internal validity

Internal validity is a prerequisite to external validity

To achieve internal validity we need to design a study with narrow population of interest and minimize non-comparability

The resulting sample may not reflect broader swath of population beyond underlying population of interest

The more narrow a sample becomes - due to strict inclusion and exclusion criteria for internal validity - the less external validity it may have if causal partners of exposure have differing prevalence in study compared with other populations


External vs. internal validity

Balancing external and internal validity is a a trade off To build a scientific argument for causal effect of

exposure on outcome, we select study design and assess internal validity of causal question

After causal effect of exposure is established in narrow population, we expand the causal question to ask

How often? Among whom? Under what conditions?









8. Summary

An example, RCTQuestion: Does weight-loss drug reduce obesity among school-age children?Study details: Recruit children for randomized drug trial with body mass index 25< (BMI) < 40 Exclude children with diabetes Parents must be fully participatory (monitoring children’s drug regime and attend

study clinic once per week) Baseline survey and monthly measurements Children randomized to receive weight loss drug or placebo Follow-up over two yearsStudy results: Mean BMI among drug group declines 31.5 to 26.7 (4.8 points BMI) Mean BMI among placebo group declines from 31.4 to 28.5 (2.9 points BMI) Reduction of 1.9 (95% CI 0.9 – 2.9) more points of BMI in drug group than placebo

group Conclusion: Weight-loss drug reduced obesity among school-age children



An example, RCT

Questions to ask about external validity Are these results externally valid to a broader population all

overweight children in Farrlandia? What about overweight children in other places? What information would we need to know in order to inform

this issue?

An example, RCTAre we confident of a causal effect in the study?

Can only be externally valid if internally valid Good reason to conclude that results obtained are approximation of causal

effect of drug for populationConsider characteristics of population from which participants were drawn

Good adherers to study protocol Diabetes-free Actively participating parents There is evidence that drug is effective in reducing BMI

Consider characteristics of larger population to assess external validity Does action of drug interact with other factors? Do other factors have a different prevalence in general the general population

of overweight children who may be prescribed the drug?









8. Summary


Example, representative sample

Question: Do sales tax on sugar-sweetened beverages reduce obesity among children aged 7 to 13?Study details: Enumerate all school-age children in Farrlandia Take random sample of 1,000 eligible Farrlandians who are between age 7 to 13

and live in Farrlandia Measure BMI before tax goes into effect Measure BMI after tax across a two-year periodStudy results: Mean BMI school-age children prior to the tax = 26.7 (95% C.I. 24.2-29.3) Mean BMI of school-age children was 24.3 (95% C.I. 23.7-24.9) two years after taxConclusion: Tax lowered mean BMI among school-age children



Snowtown is considering a similar tax Are Farrlandian results externally valid to Snowtown? What information would we need to know about Farrlandians

and Snowtownians?



Are we confident of a causal effect in the study? Internal validity: If school lunches changed to healthier offerings

during study period we would not make a causal claim that tax reduced BMI

What are potential causal partners of soda tax? External validity Soda availability is component cause

Is the distribution of causal partners similar across populations? Soda is plentiful in Farrlandia Soda is hard to find in Snowtown









8. Summary


Seven steps

1. Define the population of interest2. Conceptualize and create measures of exposures and health

indicators3. Take a sample of the population4. Estimate measures of association between exposures and health

indicators of interest

5. Rigorously evaluate whether the association observed suggests a causal association

6. Assess the evidence for causes working together7. Assess the extent to which the result matters, is externally valid,

to other populations


epidemiologymatters.org

External validity: to what populations do our study results apply ?

Documents

Transcript of External validity: to what populations do our study results apply ?