How to Develop and Implement Effective Research Tools from Ilm Ideas on Slide Share
description
Transcript of How to Develop and Implement Effective Research Tools from Ilm Ideas on Slide Share
What to research and how? Research questions, sampling and all that
Faisal BariAssociate Prof. of Economics, LUMS
Associate Fellow, IDEAS(With contribution from Dr. Farooq
Naseer, IDEAS)
Outline
• What does the TNA tell us• Framing of research issues, questions and
tools. This appears to be simpler than it is….demands reflexivity
• Sampling and related issues…all about power
Ilm-Ideas TNA: Research Tools
0
2
4
6
8
10
12
14
16
18Research tools used most often
1 2 3 4 5
Ilm-Ideas: Tools Required
Conducting a research needs assessment and/or defining research objectives
Identifying priority research questions
Selecting research sites and developing criteria for the selection
Selecting and justifying the sampling strategy and target numbers
Sampling – selecting the research target group
Conducting desk research to identify good practice examples within and outside the country of similar researches undertaken
Developing research indicators
Developing research tools and instruments
Piloting the research instruments
Conducting qualitative and quantitative research tools and instruments
Management of data collection/fieldwork, including the control, supervision and debriefings of field workers/interviewers
Use of data analysis software and systems
Conduct data interpretation and analysis
Report writing
11
13
12
15
13
11
14
14
10
15
14
17
14
13
6
4
5
2
4
6
3
3
7
2
3
0
3
4
Priority Non-Priority
Research Capacity
Conducting research needs assessment and/or defining research objectives
Identifying priority research questions
Selecting research sites and developing criteria for the selection
Selecting and justifying the sampling strategy and target numbers
Sampling – selecting the research target group
Conducting desk research to identify good practice examples
Developing research indicators
Developing qualitative and quantitative research tools and instruments
Piloting the research instruments
Conducting qualitative and quantitative research tools and instruments
Management of data collection/fieldwork, including the control, supervision and debriefings of field workers/interviewers
Use of data analysis software and systems
Conduct data interpretation and analysis
Report writing
0 2 4 6 8 10 12 14 16 18
8
7
7
6
7
10
7
10
12
10
10
3
9
11
8
10
8
8
8
5
8
7
3
5
5
7
5
4
1
0
2
3
2
2
2
0
2
2
2
7
3
1
Organizational Capacity - Research
High Medium Low
Main Challenges in Policy Research• Getting concerned institutions engaged and motivated• Data management • Interpreting data• Report writing• Availability of updated data• Accessing policy documents• Low experience/expertise in conducting policy research • Access and availability of public expenditure documents• Discrepancy in government data/inaccurate govt. data• Dearth of qualitative research experts in the country• Lack of interest within policy circles• Shortage of sector experts• Community based research• Sometimes funding agency and govt. interests don’t match
Framing Research
• Can you tell whether you are drinking Coca Cola?
• For a single person: coke or not• For a single person: coke or other colas• For many people: coke or not• For many people: coke or other colas• Trivial? Think of cure for cancer 10% total cure
versus 50 percent improvement for 50% (but not cure)
Framing Research: Examples
• Private Public Partnerships in Education: Adopt a school programme
• Importance and need: 25 A and quality issues• Variation in legal frameworks: Punjab and
Sindh• Variation in models: PEN, CARE, SEF• Variations across time: do models mature.
What is the exit strategy
Framing Research: Examples
• Remedial education for teachers (will come back to this at the end too)
• DSD reports, PEC results….content knowledge of teachers is a significant issue
• How to remedy that? CPD already in place• Something that is scale-able also• Using DTEs to reach teachers (Maths and
Science)• Use technology to reduce cost
Framing Research: Examples MFN
• Post fact impact evaluation…one way MFN paper
• Introductory paras set the context and question
• Issue of composite effect…rather than isolating contributions. Child friendly (teacher training, materials, parental involvement)…better learning
Framing Research: Examples MFN
• Propensity score matching (not gold standard…but best available here)
• Two stage matching: Schools and then children (need both school and children/family characteristics)
• School level matching: geography, medium, level of school
Framing Research: Examples MFN
• Within school blocks….child matching• Robustness• Children joining…dropped…selection bias• Treatment and control children…good match
on average
Framing Research: Examples MFN
• Mining….Item Response Theory (IRT)• Possibility of leakage (teacher transfers,
student transfers)• No non-cognitive testing….where gains might
be large too• Could we check if the effect was different on
the weakest/strongest students
Framing Research: Examples
• Tahir Andrabi and the recent education recovery paper.
• Distance from Fault Line as the independent variable
• How is that established? And What is its importance
• The results are insightful….the ‘hey, wait a minute’ moment
Sampling Issues:
• Statistics Refresher: Summarizing data• Sampling:– Minimizing error– Representativeness
• Hypotheses testing• Power
Data: Summarizing
• Variation is what we study: variation is King• Statistics helps us summarize data by using two
important features of a dataset:– the average (mean, center)
• what is the average age of participants in this room?• Is it important?....not a technical issue only (The
deer hunter)
– the variance (variability, spread)• by how much does age vary across participants?• Again….is it important…and when (50 or 0/100)
Distribution
Population and Sample
• Measuring the population gives us the truth! (assuming there is no measurement error)– But we usually cannot survey the entire
population– Hence we must draw a sample• How do we choose the sample? • How large should be the sample?
Sample
• Sample must be representative of the population:– Draw a random sample– Jute example, skulls, Indian census
• But still, the sample is not some fixed subset of the population so each sample will be different!
• This is called “sampling error.” How to reduce it? – Draw a larger sample. – But how large? (depends on the hypothesis of
interest and sampling error… want to maximize the “power” to reject incorrect hypotheses)
Simple Random Sampling
• List every individual in the population of interest (population size: N)
• Decide on a sample size based on ‘power’ calculations (sample size: n < N)… to be discussed
• Randomly pick n individuals from the population such that each individual has a positive chance of being picked
• Examples: • Toss a coin• Draw lots out of a basket• Use a computer software
Stratified Random Sampling• Mark separate sub-groups (or strata) in the
population list before drawing a random sample from each
• Stratified Sampling– For adequate representation
of different sub-groups (i.e. strata) in the population
– For a given sample size, reduces the sampling error as compared to the un-stratified simple random sampling
• Trade-off between the cost of doing stratification and the smaller sample size needed
• Fraction sampled could be different across strata; improves across-group comparisons
Two Nice Results
• Before we turn to hypothesis testing and the concept of statistical power, important to recognize that the sample average behaves well in large samples
• Law of Large Numbers– The sample average will approach the true
population average as the sample size increases• Central Limit Theorem– The sample average will tend to be normally
distributed, around the true population average value, as the sample size increases
Normal Distribution𝑀𝑒𝑎𝑛:𝜇=
∑ 𝑥 𝑖
𝑛
𝑆𝑡 .𝑑𝑒𝑣 :𝜎=√∑ (𝑥𝑖−𝜇)2
𝑛
Hypothesis Testing
• Suppose the average pre-training knowledge of M&E in the population is 3/10 points on a standardized test
• How can we empirically test whether this course improves M&E knowledge?
• In statistical terms, this test can be stated as follows:– H0 or the null hypothesis: This hypothesis states what you
would like to disprove i.e. “no effect”.– H1 or the alternative hypothesis: The course improves M&E
knowledge i.e. “positive effect”.
Hypothesis Testing• Ex-post, administer the test on multiple cohorts of course
participants –OR– use statistical theory to decide based on just one cohort
• When is the average test score of course participants in a cohort “significantly” (i.e. statistically) higher than 3?
• That is, allowing for sampling error, when can we be “confident” that we are observing a real improvement in M&E scores?
• Depends on the sampling errorin average test score
Hypothesis Testing
• Suppose, you want to test a promising intervention designed to improve
• (M&E) education. Question: Is the intervention (“treatment”) effective?
Statistical Power
• The power of a test is the probability of correctly rejecting the null hypothesis
• In other words, power is the probability of correctly declaring the treatment as beneficial
• Hence, Statistical Power = 1 – Prob(Type-II error)
Importance of getting power right
Testing a new ‘miracle’ cure for cancer– Power too low; missed a large treatment effect– Power too high; wasted resources in doing a large study to
declare a tiny, clinically irrelevant effect as statistically significant
– Power just right; have a good chance of detecting reasonably sized effects, but not tiny ones
Power: Main Ingredients
For a given significance level, power depends on the following:
1. Sample Size 2. Assumed Effect Size under H1
3. Variance of outcome in the study population4. Proportion of sample in T vs C5. Clustering
Power Sample Size
• Increasing the sample size reduces the sampling error (i.e. sample-to-sample variation) in the sample average
Treatment Effect
Variance
• The “sampling error” in the sample average, sigma^2/n, is directly proportional to the (“natural”) variance in the outcome variable in the population
• There is sometimes very little we can do to reduce the noise
• The underlying variance is what it is• We can try to “absorb” variance: – controlling for other variables
Clustering:
• You want to know how close the upcoming national elections will be
• Method 1: Randomly select 50 people from the entire population
• Method 2: Randomly select 10 families, and ask five members of each family their opinion
• Method 2 will yield relatively imprecise/noisy estimates if the political opinion within families does not tend to vary a lot (high “intra-cluster correlation”)
Sampling Frames for Examples Used
• For PPP• For remedial education for teachers• Why did MFN go the way he did
And last but not least
• Happy hunting
• Thank you