Study Design for Quantitative Evaluation ©2015 The Water Institute.

Study Design for Quantitative Evaluation

©2015 The Water Institute

Objectives

• To understand the strengths and limitations of basic quantitative study designs

• To understand main sources of error resulting from poor study design

• To know how to control or minimize these sources of error.


A possible scenario

You get a memo from headquarters asking how can you show, quickly, that your new water supply programs are “sustainably functional”, and “better than average”

How would you design a study to do this?


Let’s look at some ways to do this…..

1. Visit all the systems your program is built

2. Assess what fraction is working well

3. Compare with overall fraction of water supplies in your area that are working


CROSS-SECTIONAL STUDIES


Cross-sectional Studies

2000 2010 2020

Time

Assess your completed systems and access to water


What are you comparing?

An external evaluator from HQ comes along and raises questions:

– How old are your systems?– How old are the systems with which you are

comparing your systems?– How were the communities in which you worked

selected?– Are they similar, or different, from your

comparison group?


Control Groups

The first question (system age) is relatively easy to answer by choosing your comparison group carefully.

The second one (innate differences between intervention and control groups) is much more difficult, and is a fundamental limit to cross-sectional studies,


Cross-Sectional Studies

Strengths– Simple– Fast…snapshot at one point of time

Weaknesses– Association, not causation– Can mislead, if not thought through


A second chance to solve the problem

Fortunately, headquarters is now concerned about the long term answer to these questions, and is seeking advice on how to do this better. They want to know how well your “training of water committees” program works to improve access.

If you had more money and time available, what could you do?


COHORT STUDIES


More detailed studies

Instead of only looking at one time, you could follow a group of your projects, and a group of other projects over time.These are called “cohort” studies, and the groups that you follow over time are called “cohorts”.


Cohort Studies

• Measure the intervention and the outcome at the start

• Measure the outcome the same way at a later date

• Measure other relevant variables as needed to test hypotheses


Cohort Design

2000 2010 2020

Time

Assess training for water committees and

access to water

Assess access to water


Cohort Studies

Strengths• Tells a story over time• Cause and effect are clearer

Weaknesses• Takes more time, money


The need for controls

2000 2010 2020

Time

Assess water committees that you have trained and access to water


2000 2010 2020

Assess water committees that have not received your training and access to water


Intervention

Control


A better example

2000 2010 2020

Time

Assess well drillers that you have trained and access to water


2000 2010 2020

Assess well drillers that have not received your training and access to water


Intervention

Control


Intervention and control groups

What are you comparing with what?– Do you want to compare with other similar programs?– Do you want to compare with communities with no

programs?– Do you want to compare with national average, or local

average?Your intervention and control groups define what you can say from your monitoring/evaluation


Greater insight costs money!

Simple cross-sectional study – One group, one time snapshot

Simple cohort study– One group, followed over time

Controlled cohort study– Intervention and control groups followed over

time


CONFOUNDING & CONFUSION

20

Source: Simonkneebone.com


Some sources of error• Selected populations or samples reflect unintentional bias or

restriction• Control groups reflect unintended bias or are inappropriate• Ecological fallacy. Trends between groups are falsely assigned

to individual differences within the groups• Confounding. Statistical associations between factors, causes

and effects that do not reflect the causal chain

Thinking about these problems when designing your evaluation is much better than waiting until after all the data are collected!

21©2015 The Water Institute

A warning from a health study

The example which follows is from a health studies of WaSH, but issues/approaches are the same for measuring “health of water systems” as well as “the health of people”

Experience shows health effects from WaSH require are difficult to measure, and erroneous conclusions are easily drawn!

Here is a story of two villages in Africa, one with mostly piped water, another using hand-dug wells…


Compare two communities’ water services

Village Piped Water(# of villagers)

Dug Home (# of villagers)

% Piped Water % diarrhea(children <5)

Namabengo 216 70 76% 7%

Mkongo 100 134 43% 32%

Researchers noted that only 7% of Namabengo’s children had diarrhea in one week, while 32% in Mkongo had diarrhea… and Namabengo has more people with better service than Mkongo.

Does the better water supply in Namabengo make the difference?


Not if you look carefully!

Village Piped Water(children with diarrhea

/# of villagers)

Dug Home (children with diarrhea

/# of villagers)

Avg. diarrhea(children <5 years of age

/total population)

Namabengo 15/216 (7%) 5/70 (7%) 20/286 (7%)

Mkongo 37/100 (37%) 39/134 (29%) 76/234 (32%)

Children < 5 years with diarrhea during previous week

Source: Prag JB et al. (1983) Water Master Plan for Iringa, Ruvuma and Mbeya Regions, Tanzania Vol. 13 Ch. 11


Ecological Fallacy

“Ecological fallacies” occur when an association between groups is held to apply to individuals within the groups.

– While a village with more piped water had less disease, the individuals using piped water in each village were not significantly more healthy.


Typhoid Fever & Telephone Poles

An epidemiologist once showed:– as the number of telephone poles increased in the US, typhoid fever

decreased.1

– We are also sure that traffic deaths also increased (from accidents NOT involving telephone poles)

Did the telephone poles reduce the typhoid, or increase the traffic deaths?_____________________________________________1

Kawata, K. (1978) Of Typhoid Fever and Telephone Poles: Deceptive data on the effect of water supply and privies on health in tropical countries Prog. Wat. Tech. Vol 11, Nos 1 – 2, pp 37-43.


A causal model

27

Fewer typhoid cases

Economic Growth

Better Water

More Telephone

Poles

More traffic deaths

More Cars

Causal factorIrrelevant Causal chainNon-causal Association

Better Education

Better Sanitation


Confounding

A “confounding variable” is:– Statistically associated with a cause– Statistically associated with the effect of interest– NOT on the causal chain between the cause and the effect

of interest

Confounding variables matter, because they are often mistaken for causes, or distort findings


Confounding and Control Groups

In one cross-sectional study, sanitation is highly correlated with low diarrhea• People with sanitation had less diarrhea than people without

sanitation

Question: What kinds of people invest in sanitation?


What to do about confounding?

Identify likely confounders, and measure them!– Can then often control for them statistically

Randomize between control and intervention at the outset of a cohort study

– Random means “rigorously random” by statistical methods, it does not mean “haphazard”


PROCESS OF STUDY DESIGN


Basic stages of study design• Define questions to study• Identify your statistical support!• Define intervention (cause) and outcome (effect) to study• Identify other factors that influence the outcome to be

measured…”map” them• Determine whether x-sectional or longitudinal study• Define populations of interest (including controls if needed)

• Define sampling strategy of populations to minimize bias• Define methods of data collection• Define methods of analysis-- The rest is implementation….made MUCH easier by good design!


RECAP


Study designs

Examples have been from “health studies”, but logic and issues are the same to study “healthy water supplies” or “healthy sanitation programs”.

Cross-sectional studies are a snapshot in time Relatively quick and cheap Can’t establish cause and effect, only association Good to generate ideas

Cohort studies occur over time Take more time, and more money Can investigate causation, For outcome assessment, need control groups


Process of study design

An iterative process, as constraints become apparent (e.g. on sampling, timing, etc.)

– Plan the data analysis from the beginning…so that you KNOW you will collect enough of the righ data to answer your question!


Study Design for Quantitative Evaluation ©2015 The Water Institute.

Documents

Transcript of Study Design for Quantitative Evaluation ©2015 The Water Institute.