IDS Impact, Innovation and Learning Workshop March 2013: Day 2, Paper Session 3 Laura Camfield
-
Upload
institute-of-development-studies -
Category
Technology
-
view
313 -
download
0
Transcript of IDS Impact, Innovation and Learning Workshop March 2013: Day 2, Paper Session 3 Laura Camfield
Qualitative methods in modern impact evaluation Who’s doing it?
How are they doing it?
How is its quality being assured? And are existing checks sufficient?
Ethical codes, peer review
Archiving
Restudies
2
Endorsement
from DFID (funded
reports by Garbarino and
Holland, Stern)
And from
NONIE, 3ie,
World Bank IEG,
Ford
Foundation, etc.
Growing interest in approaches to
causation that incorporate
qualitative methods
Small n/ case study
Theory Based
Realist
Process tracing
Range of research designs limited
Qualitative research typically post-hoc
rather than ex-ante
Integrated analyses are rare
Hard to access full methodological accounts
– including how the analysis was done - or
raw data
Growing numbers of mixed methods designs, many ongoing 3ie – 700+ studies, 50 qual or mixed
methods (education, health, social protection, rural livelihoods)
But qualitative component often small or superficial, ◦ e.g. sampling through participatory wealth
ranking or using criteria from PPA, ‘field visits’, ‘qualitative surveys’
‘Empowering parents to improve education: evidence from rural Mexico’
◦ Reports data from 30 focus groups with parents from treatment and control schools, but no analysis of differences between groups or how the programme promoted parental participation. For further information we are referred to an internal report
‘Changing Households’ Investments and Aspirations through Social Interactions’
◦ Mentions ‘focus groups and semi-structured interviews with a wide set of beneficiaries and other local actors’ which ‘suggests that aspirations and perspectives towards the future may have been a key part of program impact’ (p9). The methodology and data underpinning these claims are in a report that has not been translated and isn’t available online
‘Assessing the Impacts of Farmer Field Schools on Excessive Fertilizer Use in China’ Detailed description of RCT methodology and sampling -
‘qualitative data will also be collected to see what worked and how it worked'
‘Evaluating the Impact of Technology Development Funds in Emerging Economies: Evidence from Latin America’ Qualitative methods not included in the methodology, but
emerges like a rabbit from a hat ... ‘finally, based on qualitative evidence from this study, financial support should be accompanied by the infrastructure and technological services of research centres and universities’
◦ Data production
‘Intellectual biographies’ of researchers and fieldworkers
Quality of note taking – capturing embodied knowledge?
◦ Data management
Transcription
Translation
◦ Analysis
‘Tertiary’ analysis
◦ Representation
Moving beyond quotes and text boxes
Evidencing claims - ethnographic authority?
9
1. Record of publications, presentations etc which explain and /or draw on the archived data.
2. Research design
a) Brief context and logic of research design (in each data collection period if appropriate).
b) Is research exploratory or question driven? What are the questions?
3. What were the sampling decisions and how do they relate to the research questions?
a) Was the desired sample achieved? How does the sample relate to wider empirical evidence across the population and/or theoretical issues?
b) Are there implicit as well as planned ways in which the sample is structured?
4. An overview of what data is provided as part of the project.
5. A descriptive profile of each participant specifying units of data (interview; diary; by wave).
10
Rigour, Credibility, Relevance ◦ 1. Was there a clear statement of the aims of the research?
◦ 2. Is a qualitative methodology appropriate?
◦ 3. Was the research design appropriate to address the aims of the research?
◦ 4. Was the recruitment strategy appropriate to the aims of the research?
◦ 5. Were the data collected in a way that addressed the research issue?
◦ 6. Has the relationship between researcher and participants been adequately considered?
◦ 7. Have ethical issues been taken into consideration?
◦ 8. Was the data analysis sufficiently rigorous?
◦ 9. Is there a clear statement of findings?
◦ 10. How valuable is the research?
Validity in Quantitative and qualitative research
12
Quantitative research
Qualitative research
Bryman Guba & Lincoln “Naturalistic inquiry”
“trustworthiness and authenticity”
Internal validity Legitimate causality claims Do the observations correspond well to the
theoretical constructs.
Can you support claims that you observed
what you claim?
Credibility Multiple accounts of the same phenomenon
researcher validation
respondent validation
triangulation
External validity Generalisability
(representative samples)
Can the qualitative findings be transferred
across social contexts.
Transferability Research on small groups;
Depth vs. breadth
“Thick” description (Geertz)
Reliability Repeatability, replicablity Could the qualitative research be replicated
(access to qualitative inquiry “texts”)?
Researchers adopt similar social roles
Do researchers agree about what they see
and here? – inter-observer credibility
Dependability Trustworthiness by providing “audit trail” –
extensive records of all aspects of the research,
including raw data
Re-studies
Objectivity Objectivity, methodological
rigour
Objectivity in social science is spurious
Values more embedded in qualitative
research
Confirmability Researcher worked in good faith
Possible solutions (Camfield and Palmer-Jones, 2013)
Ethical codes, peer review
Data archiving
Restudies
13
Social science ethical codes typically focused on care of the subject, but see GoS’s (2007:1) emphasis on ‘respect, rigour and responsibility’ ‘act[ing] with skill and care in all scientific work’
‘communicating results and intentions honestly and accurately, and understanding that your work or its outputs will have an impact on society in its broadest sense’
Not all countries/ institutions/ disciplines have ethical codes or committees
Ethics committees assess research designs, but do not monitor data production or presentation
Peer reviewers rarely see the data – no qualitative data deposit with journals – so much is taken on trust/ reputation
Grey literature and working papers may not receive the same level of scrutiny, but are still influential
‘Audit trail’ obscure in earlier examples
What does it offer? ◦ Methodological insights
◦ Increased use of data for research and teaching
◦ Respect for respondents’ accounts and time
◦ Judging validity of claims
◦ Historical perspective
Secondary analysis is difficult to do well, but within impact evaluation, most qualitative analysis is secondary, or even tertiary
‘Scientific replication’ ◦ Goldthorpe and Lockwood (1962-3) tested the ‘affluent worker’
hypothesis by taking highly-paid car assembly workers at the Vauxhall factory in Luton as a ‘critical case’ to investigate whether everyone was becoming middle-class. They concluded that this wasn’t the case, albeit on the basis of a small quantitative sample and (Savage argues) inadequate engagement with the data. Savage (2005:39) suggests that the conclusion arose because they had fitted their data into a particular typology which closed off alternative interpretations.
Methodological insights ◦ Gillies and Edwards (2011:23) describe how in Townsend and
Marsden’s (1965) study of single mothers, their “physical attractiveness (or lack of it) is commented on [...and] perceived intelligence and character was also subject to evaluation”.
Some examples from developing countries, e.g. ESRC funded restudy of Indian villages studied in the 1950s by Bailey, Mayer and Pocock; van Schendel, 1981 (Bangladesh); Breman et al., 1997 (S. & S.E. Asia); Breman, 2007 (Gujarat, India),
And some controversies, often due to ‘interpretive overreach’ (e.g. Freeman vs. Mead, Lewis vs. Redfield, Tierney vs. Chagnon)
Why restudy? (Dis)confirming original findings
Epistemological/ methodological insights
Longitudinal perspective
Running out of field sites...
Constructivist revisits (types 1 and 2) assume the site being studied at two points in time does not change so differences are due to the different relation of the ethnographer to the site (type I) or theory that the ethnographer brings to the site (type 2) e.g. Weiner's (1976) feminist reconstruction of Malinowski’s
(1922) Argonauts of the Western Pacific highlighted the importance of mortuary ceremonies in cementing women’s control over ancestral identity alongside Malinowski’s celebration of the ‘Kula ring’
Positionality and assumptions also important in evaluation
Realist revisits (types 3 and 4) study historical change. Type 3 revisits focus on internal processes in accounting for differences between authors’, while type 4 emphasise external forces Hutchinson’s (1996) revisit to Nuerland (now Southern
Sudan), which was studied by Evans-Pritchard (1940), explicitly to explore the impact of decolonization, war, Christianity, and transnational capital on the lives of the Nuer
Hard to reliably attribute and understand effects of particular interventions in complex, changing environments
Would Bemba society (N. Zambia) break down when men migrated to southern African mines because the slash and burn agricultural system (citimene) needed men to cut down the trees? ◦ Richards – yes (conclusions enthusiastically adopted by colonial
administrators/ chiefs for their own reasons)
◦ Moore and Vaughan (50 years later) – no (real threat was Zambian government’s agrarian reforms which promoted the labour-intensive cash crop maize)
But Moore and Vaughan’s analysis may also have been ‘one-sided, governed by specific feminist and Foucauldian assumptions’ (Burawoy, 2003:667) and effects will be seen in future revisits…
What lessons can be learned for evaluation? ◦ Watch out for interest groups!
◦ Don’t forget about the women
◦ Impacts of interventions are often unintended, multi-dimensional, and affect every member of the household, not just the target maize income and women’s labour controlled by men,
women diverted from subsistence farming, family becomes food insecure and women not entitled to take a share of the income to buy food, children’s health suffers...
Quality of research is an ethical issue
Increasing role for qualitative research in impact
evaluation requires increased reflection on current
standards
Epistemological and methodological problems
cannot easily be detected or influenced through
conventional processes (standards, committees....)
But qualitative researchers have own strategies for
ensuring rigour and reliability
Data archiving and restudies contribute to these