In search of Conceptual Richness and Contextual Thickness: Implications of Realist Synthesis for...

In search of Conceptual Richness and Contextual Thickness: Implications of Realist Synthesis for Information Retrieval

Dr Andrew Booth, Reader in Evidence Based Information Practice, School of Health & Related Research

The Place of Information Retrieval

• “Realist synthesis is a systematic, theory-driven approach for making sense of diverse evidence about complex interventions applied in different settings. To achieve this, a realist review brings together multiple sources of evidence to develop a programme theory of the intervention in question” Goodman et al, 2014.

21/04/23 © The University of Sheffield /School of Health & Related Research

Multiple Sauces?


Conceptual richness - “a degree of theoretical and conceptual development that explains how an intervention is expected to work”

Contextual thickness requires:1. Sufficient detail to enable reader to establish what exactly is going on, in the intervention and the wider context.2. Sufficient detail to enable reader to infer whether findings can be transferred to other people, places, situations, or environments .

Types of Data

• Randomised Controlled Trials

• Stand-Alone Qualitative Research Studies

• Qualitative Research Studies Nested in or Alongside Trials

• Mixed Methods Studies

• Process Evaluation Studies

• Case Studies

• Survey and Questionnaire Data

• Discussion Sections in Trial Reports

• Policy Documents

• Theoretical Papers

Etcetera……………………….


Theory & ContextTable 1 - Systematic review methodologies requiring identification of theory and/or context

Methodologies requiring identification of theory

Methodologies requiring identification of context

Best fit framework synthesis

Framework synthesis

Realist synthesis Realist synthesis

Systematic review of complex interventions

Systematic review of complex interventions


Two Potential Problems

• Empirical studies and theoretical studies are frequently separated and difficult to “join up”

• Individual studies typically lack the detail of context required to understand “what is working”. Clusters of related studies are required. Not all of these will be published in the journal literature.

• Both the above problems suggest that comprehensive searches of the journal literature will not be adequate in retrieveing relevant studies.


A Potential Solution:• “Supplementary” search techniques

pursuing “sibling” relationships (i.e. where different papers relate to the same study) and “kinship” relationships (i.e. where papers are related theoretically)

• We trade fingertip “combing of the area” (comprehensive searching) for “looking for clues” (purposive sampling).


Interrogating Suspects!• Jagosh et al (2012) identified “23 partnerships,

collectively composed of 276 documents, including peer-reviewed and non-peer-reviewed publications and websites”.

• Authors did not use systematic approach to identify “clusters”. Contact with authors (CwA) was used and if successful may identify most, if not all, “sibling” papers directly associated with a named cluster.

• CwA would not reveal additional papers invoking theory or related projects with a common provenance “kinship”).



Looking for Clues requires:• Starting with a KEY (i.e. an Access Point)

• An Access Point may be:• Name of a Principal Author (or Co-Author)

• Institutional Name

• Name/Identifier of a Project

• Theory Label (e.g. “Theory of Planned Behavior”)

….Etcetera

This presentation will:• Review search methods used in published realist

syntheses.

• Identify common themes & information retrieval challenges.

• Offer range of search techniques for realist synthesis.

• Update thinking on CLUSTER method

• Rehearse strengths and limitations of citation searching

• Discuss role of systematic searching for theory.

• Explore potential to enhance existing RAMESES reporting standards

Search methods• ‘Digging through’ the literature and drawing on experience to identify key

terms, concepts and mid-range theories that provide some explanation about the subject of interest….Concept mining and theory formulation was achieved through a mixture of face-to-face and virtual brain storming”

• “Literature needs to be scrutinised to identify studies related to the targeted ‘programme theories’ . This focus does not mean that the approach to search and appraise literature is any less rigorous or systematic… The search is purposive…for each of the theory areas the group embarked on producing a list of relevant and related search terms. The final list of terms…was used to guide the searches, …conducted by two team members in consultation with their institution’s health science librarians” (Rycroft-Malone et al, 2012)


Common themes & information retrieval challenges in RS

• Iterative Process and How to Document? (Rigour vs Reality)

• Defining Relevance (and operationalising it!)

• Tension between Context (which “narrows”) and Theory (which “broadens”) [e.g. Community engagement – UK Context but Developing World contributed Theory]

• Purposive Sampling (including Intensity Sampling!)


Role of Searching/ Sampling in RS


Constructing a Sampling Frame

Homogenous Sampling

Intensity Sampling

Jagosh et al, 2014

Challenges & Issues in Searching for Theories• Theory not typically reported in Abstracts

• Theoretical base differs by discipline cp. HSR vs Public Health vs Nursing vs Psychology vs Sociology

• Reporting of Theory differs by discipline

• Level of Theory may vary e.g. Individual versus Society (Psychology vs Sociology)

• When is a “theory” a Theory? – labelling (model, framework, concepts ) and naming (“Health Belief Model”)

Search techniques for realist synthesis

• Searching for Theory

• Searching for Sibling/Kinship Studies

• Citation Searching

• CLUSTER searching


Process of Realist Synthesis from:

McCormack et al (2007)

Cluster searching for “Siblings”

Element Procedural Steps

Citations Backwards reference chaining

Lead Authors Author searching; Backwards reference

chaining

Unpublished

materials

Web searches; repositories

Scholar searches Citations

Theories Backwards reference chaining

Early Examples Cited works (Forward reference chaining)

Related Projects Co-citations

CLUSTER method

Strengths and limitations of citation searching

• Strengths: compensate for indexing inadequacies/terminology variation; longitudinal development of argument

• Weaknesses: multiple reasons why people cite; choice of reference may be arbitrary or reflect accessibility; many alternative equally valid citations may exist; self citation and citation “schools” or “communities”

• Weaknesses: Citation searching may be used strategically – to ignore opposing views or to falsely emphasise originality


CLUSTER method 2.0• Studies with large numbers of citations may

make systematic reading prohibitive:• Use “Search Within Citing Articles”

e.g. for a study type: “qualitative”, “findings”, “interviews”; or “process evaluation” or “process data”

e.g. for a theory: “theor*” or “concept*” or “model*” or “framework*”

• Use “Related Articles” features e.g. Google Scholar, PubMed etcetera


Role of systematic searching for theory?


BeHEMoThBe - Behaviour of Interest: Way

population or patient interacts with

health context e.g. access for a

service, compliance, attitude to policy.

H - Health Context: i.e. the service,

policy, programme or intervention

E – Exclusions: To exclude non-

theoretical /technical models (depends

on volume).

MoTh - Models or Theories –

operationalized as a generic “model*

or theor* or concept* or framework*”

strategy together with named models

or theories if required.

BeHEMoTh (Example)Be - Behaviour of Interest:

Compliance/acceptability/preferenc

e/decision making

H - Health Context: Vitamin

Supplement Use

E – Exclusions: NOT 'regression

model' OR 'integrative

model/integrative care model' OR

'economic model' OR "Markov

model" OR "animal model"

MoTh - Models or Theories –“model*

OR theor* OR concept* OR

framework*” (and/or) Named

Models or Theories if required.

Four Phase Process1. “Trawling” using BeHEMoTH structure

(dropping concepts as appropriate)2. “Depth-charging” using Behaviour and

Health context with most common theories3. “Fishing” using named item searches for

list of theories generated from Phase One (above) [excluding those already covered in Phase Two].

4. “Using a sprat” – citation searching (combined with topic)

“Unidentifiable” Theory Articles

Existing RAMESES reporting standards?

• Searching should be guided by the objectives and focus of the synthesis, and revised iteratively in the light of emerging data. Data…may lie in a broad range of sources that may cross traditional disciplinary, program and sector boundaries. The search phase is thus likely to involve searching for different sorts of data, or studies from different domains, with which to test different aspects of any provisional theory.

• Search methods using forward and backward citation tracking may be particularly valuable in finding the documents necessary to develop and

then test provisional theories. 21/04/23 © The University of Sheffield /School of Health & Related Research


• Realist syntheses do not exclude sources solely on the basis of their study design; hence, 'methodological filters' …may add little to the search and could potentially miss relevant papers.

• Searching is likely to be iterative because, as the synthesis progresses, new or refined elements of theory may be required to explain particular findings, or to examine specific aspects of particular processes.



• As new elements of theory are included, searches for evidence to support, refute or refine those elements may be required…the process used for any such additional searches should be clearly documented. A single pre-defined search is unlikely to be sufficient and may suggest insufficient reflection on emerging findings.

• Sufficient detail should be given to enable the reader to judge whether searching was likely to have located sources needed for theory building and/or testing.


References - 1• Booth, A (2013) Acknowledging a

Dual Heritage for Qualitative Evidence Synthesis: Harnessing the Qualitative Research and Systematic Review Research Traditions. PhD thesis, University of Sheffield.

• Booth A, Harris J, Croot E, Springett J, Campbell F, Wilkins E. Towards a methodology for cluster searching to provide conceptual and contextual "richness" for systematic reviews of complex interventions: case study (CLUSTER). BMC Med Res Methodol. 2013 Sep 28;13:118

• Jagosh, J., Pluye, P., Macaulay, A. C., Salsberg, J., Henderson, J., Sirett, E., ... & Green, L. W. (2011). Assessing the outcomes of participatory research: protocol for identifying, selecting, appraising and synthesizing the literature for realist review. Implement Sci, 6(24).

• Jagosh, J., Macaulay, A. C., Pluye, P., Salsberg, J., Bush, P. L., Henderson, J., ... & Greenhalgh, T. (2012). Uncovering the benefits of participatory research: implications of a realist review for health research and practice. Milbank Quarterly, 90(2), 311-346.


References - 2• Jagosh, J., Pluye, P., Wong, G.,

Cargo, M., Salsberg, J., Bush, P. L., ... & Macaulay, A. C. (2014). Critical reflections on realist review: insights from customizing the methodology to the needs of participatory research assessment. Research Synthesis Methods. 5(2):131-141.

• McCormack, B., Wright, J., Dewar, B., Harvey, G., & Ballantine, K. (2007). A realist synthesis of evidence relating to practice development: findings from the literature analysis. Practice Development in Health Care, 6(1), 25-55.

• Rycroft-Malone, J., McCormack, B., Hutchinson, A. M., DeCorby, K., Bucknall, T. K., Kent, B., ... & Wilson, V. (2012). Realist synthesis: illustrating the method for implementation research. Implementation Science, 7(1), 33.

• Wong G, Greenhalgh T, Westhorp G, Buckingham J, Pawson R. RAMESES publication standards: realist syntheses. BMC Med. 2013 Jan 29;11:21. doi: 10.1186/1741-7015-11-21.


In search of Conceptual Richness and Contextual Thickness: Implications of Realist Synthesis for...

Documents

Transcript of In search of Conceptual Richness and Contextual Thickness: Implications of Realist Synthesis for...