Issues and Challenges around Synthesising Qualitative Research

• Most methods of primary qualitative data analysis can be applied to qualitative evidence synthesis

• ‘New’ evidence synthesis approaches have emerged –eg meta-ethnography – but use recognisable qualitative analysis principles

• Many methods have similar sounding names – rush to publish ‘my method’ and call it something slightly different – meta-narrative, meta-ethnography, meta-summary, meta-synthesis, meta-aggregation – etc etc.

• Some reviewers mix or combine methods – eg thematic and realist into a single review

What have we learned so far?

• Many authors appear to stick with familiar methods rather than select the most appropriate method to address the question and type of evidence

• Software evolving and improving: EPPI reviewer / Atlas Ti

• Searching methods and approaches have evolved for specific methods of qualitative synthesis (purposive versus exhaustive, tipping point)

• Increasing numbers of (very good and inevitably some very bad) QES published!

• Cochrane finally getting to grips with qualitative evidence synthesis!

What have we learned so far cont?

Looking to the future: where are we now and where do we want to go next?

• Increasing number of reviews including qualitative evidence

• Prescription versus pragmatism• Protocols• RevMan

• And, revisiting primary qualitative research toolbox to help advance methodological challenges

• Huge increases in qualitative syntheses published

• There is a big gap between what people claim to use as a synthesis approach and what is actually done in practice

• Do not need more methods – but better application and further evaluation of existing ones

New and updated guidance and new acceptance of the value of qualitative evidence

Greater specification and further adaptation of methods for qualitative evidence synthesis

Thematic synthesis –various types with different starting points - inductive and deductive

• 3 stage thematic synthesis - Line by line inductive coding- eg Thomas and Harden

• A priori - eg 5 stage Ritchie and Spencer Framework Synthesis

• Conceptual frameworks and models to guide analysis and synthesis

• Additional adaptations to conceptual frameworks to ‘best-fit’ specific reviews – eg conceptual development by the back door (pragmatism)

Gallacher, et al. (2013). Qualitative systematic reviews of treatment burden in stroke, heart failure and diabetes-

Methodological challenges and solutions. BMC medical research methodology, 13(1), 10.

• Framework analysis… was appropriate due to our a priori knowledge in this area.

• …always a risk with framework analysis that data has been ‘shoe horned’ into the framework, with the possibility that some data may be missed. .. although this work was deductive to some extent, we were careful to augment the framework during analysis, being careful to ensure that our findings were derived directly from the data, and importantly, made a careful note of any data that fell outside of our framework.

• We failed to find any such data [that fell outside of framework], which suggests that the use of NPT as the underpinning theory for our analysis proved to be appropriate in this case.

‘Best Fit Framework Synthesis’

Carroll, C., Booth, A., Leaviss, J., & Rick, J. (2013). “Best fit” framework synthesis:

refining the method. BMC medical research methodology, 13(1), 1-16.

BMC Med Res Methodol. 2014 Jun 21;14(1):80. Meta-ethnography 25 years on: challenges and insights for synthesising a large

number of qualitative studies. Toye F et al.

• Builds on Noblit & Hare to explore challenges of including a large number of qualitative studies into a QES. Challenges hinge upon epistemological and practical issues alongside expectations of what determines high quality research.

• Central to method - collaborative interpretation of concepts and decision to exclude original material where team could not decipher a concept. Uses excerpts from research team's reflexive statements to illustrate development of methods.

Lee, et al. Qualitative synthesis in practice: some pragmatics of meta-ethnography

Qualitative Research, February 24, 2014 doi:10.1177/1468794114524221

• Reviews existing meta-ethnographies with focus on methods reported by authors.

• Considers implications of these methods and reason for presence (and absence) of practices in reporting on meta-ethnographies. Draws upon team experiences of conducting meta-ethnographies with focus on two key practices: ‘reading’ and ‘conceptual innovation’.

• Concludes by discussing how meta-ethnographic process requires active reading, recognition of multiplicity, realistic approach to conceptual innovation and, importantly, collaborative work.

• Therapist dependent interventions (where the intervention is a combination of the therapist effect and the therapy or procedure and the effectiveness is potentially dependent on both);

• Complex healthcare interventions (where the intervention is a combination of several actions, e.g. multidisciplinary health care in stroke units); multilevel public health interventions (e.g., a healthy living initiative that aims to impact behaviour at the community, school, and individual levels);

• Professional or patient education interventions (e.g., introduction of clinical guidelines).

• Complex interventions may contain a mix of effective, ineffective, and even harmful actions which may interact synergistically or dysynergistically or be interdependent.

Greater emphasis on QES for complex interventions

MICCI - Methodological Investigation of Cochrane Reviews of Complex Interventions

Applying qualitative methods to explore complexity

Development of little used methods• Qualitative Comparative Analysis (QCA); (actually a mixed

method approach) is a promising method for providing evidence in situations where interventions interact with contexts, enabling causal pathways to be discerned from how sets of conditions combine with particular outcomes.

• Byrne D. Evaluating complex social interventions in a complex world 2013

• Blackman et al (2013) Using Qualitative Comparative Analysis to understand complex policy problems

• Candy, B., King, M., Jones, L., & Oliver, S. (2013). Using qualitative evidence on patients' views to help understand variation in effectiveness of complex interventions: a qualitative comparative analysis.

QCA for synthesis

Development of little used methods

• Use of content analysis to conduct knowledge-building and theory-generating qualitative systematic reviews Finfgeld-Connett 2013

• Systematically reviewing and synthesizing evidence from conversation analytic and related discursive research to inform healthcare communication practice and policy: an illustrated guide Parry and Land 2013

• Systematic text condensation: A strategy for qualitative analysis Malterud 2012

The procedure consists of the following steps: 1) total impression – from chaos to themes; 2) identifying and sorting meaning units – from themes to codes; 3) condensation – from code to meaning; 4) synthesizing – from condensation to descriptions and concepts.

Greater use of Conceptual Frameworks, Theoretical and Logic models

• Programme logic is the way in which a ‘programme’ fits

together, usually in a simple sequence of inputs, activities,

outputs, and outcomes.

• Programme theory goes a step further and attempts to

build an explanatory account of how the

intervention/programme/service works, with whom, and

under what circumstances.

Baxter, S K., et al. "Using logic model methods in systematic review synthesis: describing complex pathways in referral management

interventions.“ BMC Medical Research Methodology 14.1 (2014): 62.

Use of logic models

Turley et al 2013

Slum upgrading

review:

methodological

challenges that

arise

in systematic

reviews of

complex

interventions

Health belief model: public information to prevent skin cancer

Looking to future: where are we now and where do we want to go next?

• Increasing the number of reviews including qualitative evidence

• Prescription versus pragmatism• Protocols• RevMan

• And, revisiting primary qualitative research toolbox to help advance methodological challenges

Increasing number of reviews including qualitative evidence

Multiple methods within a single HTA review

Prescription versus pragmatism

Purposive Sampling and Excluding Studies

• More purposeful way of sampling papers - as opposed to the comprehensive search suggested by the Collaboration for reviews that are focused on the effects of interventions.

• Purposive sampling not comprehensive… Interest of authors not in seeking a single ‘correct’ answer, but rather in examining the complexity of different conceptualizations.

• Reviewers synthesizing qualitative research may see little value in limiting the critical appraisal of studies to detecting potential methodological flaws in them, because methodologically flawless studies are no guarantee for an in-depth or rich contribution to a QES

Protocols

• Most reviewers have “pragmatic orientation toward standards and protocols. They make them work to the extent that they serve their particular goals, but once these goals diverge too much from the interests advanced by those stakeholders, then little space remains for interaction. In such cases, the standard may need to be renegotiated”.

• Need to be “accompanied by a movement in the direction of considering flexible, iterative protocols as an acceptable standard”

RevMan

• Currently, the template used for the review process only supports a linear approach to synthesis. Over recent years, RevMan developers have been increasingly more receptive….to negotiate with members of the CQIMG.

• Substantial efforts from both sides to try to adapt the software to support the inclusion of QES findings.

• “Complete consonance between what qualitative researchers would like to see and what The Cochrane Collaboration is able to deliver is unlikely to happen in the next few years, due to limited manpower and resources”.

Revisiting the primary qualitative research toolbox

Harnessing “dual heritage”• Faced with dual heritage, [conventional SR methods + primary

qualitative research] exponents of QES may select judiciously from competing techniques, adapt from richness of both traditions or maintain open dialogue around viable alternatives.

• “Placing [quantitative/qualitative] approaches in opposition does a great disservice by detracting from the contribution to be made by each, including what each can contribute to the other” (Wolcott, 2001).

• Rapprochement of two heritages heralded by recent case study that recognises unique contribution from each source::

• “such reviews are, to some extent, methodologically sui generis and cannot be governed solely by concepts imported either from SRs of quantitative evidence (e.g. comprehensiveness) or from primary qualitative research (e.g. saturation)” (Lorenc et al, 2012).

Finfgeld-Connett D. Use of content analysis to conduct knowledge-building and theory-generating qualitative

systematic reviews. Qualitative Research June 2014 14: 341-352

• Content analysis is a flexible data analysis method; but application for qualitative systematic reviews has not been fully explicated.

• Qualitative systematic reviewers are urged to adapt content analysis methods to accommodate data that are, by nature, highly organized and contextualized.

• Reviewers encouraged to use reflective memoing and diagramming to ensure valid integration, interpretation, and synthesis of findings across studies.

• Reviewers advised to clearly and fully explain their data analysis methods in research reports.

New publication standards

Enhancing transparency in reporting the synthesis of qualitative research: ENTREQ BMC

Medical Research Methodology, Vol. 12, No. 1. (27 November 2012), 181,

doi:10.1186/1471-2288-12-181 by Allison Tong, Kate Flemming, Elizabeth McInnes, Sandy Oliver

, Jonathan Craig

Continuing challenges and issues

• Many methods, small number consistently used• Need more high quality published examples and

methodological testing

• Evidence synthesis cheaper than primary research – funders increasingly fund systematic reviews – need more teams with quan/qual skills

• Need greater specification of methods for implementation reviews/complex interventions