Use of mixed methods in Use of mixed methods in the evaluation of suicide the evaluation of suicide prevention strategies and prevention strategies and interventionsinterventions

Stephen Platt

Qualitative research and suicideQualitative research and suicide

Seminar, Cardiff University, 2 July Seminar, Cardiff University, 2 July 20072007

Structure of presentationStructure of presentation

Mixed methods research Suicide in Scotland: trends and context Choose Life strategy and action plan Evaluation plan Selected main findings Selected recommendations

Mixed methods researchMixed methods research

Adoption of a research strategy involving more than one type of research method

May be mix of qualitative and quantitative methods or mix of quantitative methods or mix of qualitative methods

Increasing use of mixed methods strategies, especially combining qualitative and quantitative approaches

Many reasons why this is occurring

Why the increasing popularity Why the increasing popularity of mixed methods research? of mixed methods research? (1)(1)

Opportunity for skills enhancement Broadening methodological repertoire

mitigates vs. “trained incapacities” (Reiss)

Encourages thinking ‘outside the box’ Cross-national research (e.g. EU)

provides increased opportunities for mixed methods research

Why the increasing popularity Why the increasing popularity of mixed methods research? of mixed methods research? (2)(2)

Fits with political currency accorded to ‘practical enquiry’ that speaks to policy/ makers and informs practice “Whole industry” (Brannen) of mixed methods

research created around evidence-based policy and in policy evaluation

Increasing emphasis upon dissemination Researchers need to communicate in “double

speak” (Brannen): technical/specialised language of research and popular language that can easily communicate findings/messages to ‘users’

Words as important as numbers in writing up research

Misconceptions of ‘the Misconceptions of ‘the other’ other’

Quantitative attitudes qualitative: Too context specific Selection of data to fit preconceptions Unrepresentative samples/examples Unwarranted claims

Qualitative attitudes quantitative: Overly simplistic Decontextualised Reductionist in terms of generalisations Failing to capture subjective meaning

Rationales underlying choice Rationales underlying choice of method(s) of method(s) (1)(1)

Paradigms/philosophical assumptions Qualitative and quantitative research seen

as intrinsically different (particularly with regard to philosophical traditions)

But surveys not necessarily conducted on basis of positivist assumptions and qualitative researchers using participant observation often work in realist tradition

Micro-level emphasises subjective interpretations. Macro-level concerned with larger patterns/ trends and seeks structural explanations. But all aim to understand individuals in society. Methods need to be congruent with this quest.

Rationales underlying choice Rationales underlying choice of method(s) of method(s) (2)(2)

Pragmatics Commonplace to argue that methods should

be appropriate (and subordinate) to research Q

But usually many research Qs. Some may be underpinned by realist assumptions, others by interpretevist assumptions.

And practicalities of research process may change original intention anyway, with outcomes of research taking precedence

Mixed methods believed/claimed to produce “better” outcomes than reliance on single method (pragmatic orientation linked to emphasis upon policy/practice application)

Rationales underlying choice Rationales underlying choice of method(s) of method(s) (3)(3)

Politics Political rationales for using mixed methods Example: addressing condition of women in

society requires use of large-scale quantitative data (structural) as well as in-depth qualitative data (personal experience/perspective) in order to understand/expose gendered inequalities

Crucial issue is the purpose to which methods are put rather than the methods per se

Combining methods in Combining methods in research process: context of research process: context of justificationjustification

Stage of data analysis and interpretation Data derived from different methods cannot

be added together to produce unitary reality Possible outcomes when methods are

combined Corroboration (‘triangulation’) (‘same’

findings from different methods) Elaboration (one method exemplifies how

findings apply in particular cases) Complementarity (findings differ but together

– synergistically – they generate insights) Contradiction (findings from different methods

conflict)

Mixed methods designs: Mixed methods designs: key dimensionskey dimensions

Logic of enquiry: inductive (aimed at discovery) or deductive (aimed at hypothesis testing) No one-to-one correspondence between one

particular logic of enquiry and one type of method

(If mixed method approach is warranted) the ordering of methods needs to be considered: sequential or simultaneous?

How dominant is a particular method going to be (consider scarce resources)?

Possible permutations of Possible permutations of mixed methods designsmixed methods designs

Examples of simultaneous designs QUAL + quan QUAN + qual QUAL + QUAL QUAL + qual

Examples of sequential designs Qual QUAN QUAL quan Quan QUAL QUAN qual Qual QUAL QUAL QUAL QUAL QUAL

Suicide rates across the world Suicide rates across the world (2002)(2002)

Age-standardised suicide Age-standardised suicide rates, by country, 1991/93-rates, by country, 1991/93-2002/04, males2002/04, males

0

5

10

15

20

25

30

35

1991/ 93

1992/ 94

1993/ 95

1994/ 96

1995/ 97

1996/ 96

1997/ 99

1998/ 2000

1999/ 2001

2000/ 02

2001/ 03

2002/ 04

Sta

ndar

dis

ed s

uic

ide

rate

England

Wales

Scotland

N Ireland

Age-standardised suicide rates, Age-standardised suicide rates, by country, 1991/93-2002/04, by country, 1991/93-2002/04, femalesfemales

0

3

6

9

12

15

1991/ 93

1992/ 94

1993/ 95

1994/ 96

1995/ 97

1996/ 96

1997/ 99

1998/ 2000

1999/ 2001

2000/ 02

2001/ 03

2002/ 04

Sta

ndar

dis

ed s

uic

ide

rate

England

Wales

Scotland

N Ireland

Intentional self harm & Intentional self harm & undetermined deaths, Scotland, 15+ undetermined deaths, Scotland, 15+ years, 1970-2005years, 1970-2005

0

5

10

15

20

25

30

35

40

1970 1974 1978 1982 1986 1990 1994 1998 2002

Year

Suic

ide

rate

per

100,0

00

MaleFemale

Scotland’s national suicide Scotland’s national suicide prevention strategy & action plan: prevention strategy & action plan: Choose LifeChoose Life

Launched in December 2002 Major element of Scottish Executive’s

work on health improvement and mental health

Plan being implemented in phases Budget for phase 1 (April 2003-March

2006) was £12m Additional £8.4m allocated for first two

years of phase 2 (2006-08) Overall aim: to reduce suicide rate in

Scotland by 20% in 2013 (cf 2002)

Choose LifeChoose Life: national and national and local infrastructurelocal infrastructure

Designated National Implementation Support Team (NIST) co-ordinates and supports national development and implementation

NIST’s core functions include: awareness raising/campaigning; working with the media; development/dissemination of information and knowledge; and guiding and supporting local implementation

In each local authority Choose Life action plans have been developed by the Community Planning Partnership (CPP)

National and local budgetary allocation

Evaluation: main objectivesEvaluation: main objectives

Assess whether sustainable infrastructure is being developed nationally and locally to support achievement of Choose Life strategy

Measure and review progress towards implementation of Choose Life milestones

Examine whether and how Choose Life is stimulating effective forms of practice

Provide detailed recommendations to guide the next phase of the action plan

Theory-based evaluationTheory-based evaluation

Theory-driven approaches are intended to address needs of programme implementers

Aims to articulate and test the explicit and implicit theories that shape the design and planned implementation of a programme

Attempts to determine whether a programme has been delivered

as intended what aspects of the programme work, for

whom and in what circumstances/contexts/settings

Theory of change: “a systematic and cumulative study of the links between activities, outcomes and contexts…” (Weiss)

Theory-based evaluation: Theory-based evaluation: the role of contextthe role of context

Context as a powerful ingredient in programme evolution and success – not a variable to be ‘controlled for’

Social programmes can’t be tested/rolled out in laboratories, so getting to grips with the interface between context and intervention is of fundamental importance

Qualitative approaches can do this to a degree but rarely in a way that address concerns about programme impact

Theory-based evaluation: Theory-based evaluation: the processthe process

Through a collective/collaborative process, the evaluator encourages programme stakeholders to articulate at the earliest possible stage : The programme’s rationale Its intended outcomes The activities that will be implemented to

achieve these Contextual factors and their influence

The ensuing ‘theory of change’ can be used to improve programme planning and of make evaluation decisions

Main methodsMain methods

Two electronic surveys of local co-ordinators

Two rounds of interviews with key informants at national level (including NIST)

Detailed and in-depth exploration of theories of change in 8 selected local areas (using interviews, workshops, observation, documentary analysis)

Two workshops with local coordinators and national informants

Co-ordinator surveysCo-ordinator surveys Covering range of areas:

(Progress towards) Local vision for change Progress in the development of the local infrastructure Resource allocation and generation Examples of innovative and effective practice Monitoring and evaluation Sustainability and mainstreaming Collection of data on suicide and deliberate self-harm Reflections on national support and on local progress

Open and closed questions Rating scales to measure satisfaction with

national action towards achievement of milestones and to review local implementation progress

Interviews with key Interviews with key informantsinformants

Key elements of semi-structured interviews: Considering progress towards the achievement of

the national milestones set out in Choose Life Understanding the criteria applied at the national

level to assess local implementation Considering the quality of collaboration between the

NIST and major national agencies for the achievement of Choose Life objectives.

Interviews with members of NIST and sample of key national informants (e.g. ChildLine, Samaritans, SAMH, NUJ)

Additional component: exploration of the NIST ‘story’ via individual interviews and joint workshop

Local area case studies Local area case studies (1)(1)

Case studies were main vehicles for exploring implementation process identifying overall theoretical framework within

which local suicide prevention teams worked Representative sample selected by:

Geographical type (rural/remote, urban and mixed) [1º criterion]

Local suicide rate, focus on priority groups, interaction between national and local levels, and approaches to coordination [2º criteria]

Two fieldwork visits 6 months apart

Local area case studies Local area case studies (2)(2)

Interviews with case study informants Purposive sampling: 4 key informants involved

in decision-making process (e.g. members Choose Life partnership responsible for priority setting) and 2 representatives involved in Choose Life funded activities

16 project activities selected for more detailed examination (2 per case study site) Intended to provide representative selection

across Choose Life priority groups and objectives

Selection negotiated with coordinators Interviews conducted with project

representative, usually project lead/manager

Local area case studies Local area case studies (3)(3)

Observational activities and collection of documentation Observation of Choose Life events (usually

partnership meetings but also training days, evaluation days and practitioner fora)

Key documents included: minutes of Choose Life partnership meetings, locality reports on progress, reports of previous needs assessment

Evaluation team aimed to understand theories of change at overall programme level as well as at project level

Local area case studies Local area case studies (4)(4)

Changes to approach Workshops replaced individual interviews with

stakeholders at second fieldwork visit Provided opportunity for joint testing of, and

reflection on, local area theories of change Local participants encouraged to assess

progress towards Choose Life objectives and milestones, and how this could be demonstrated

Problems created as result of mixed levels and responsibilities of participants (e.g. strategic versus operational)

National workshops National workshops (1)(1)

Two workshops one year apart brought together evaluation team, NIST, local coordinators and several key stakeholders

1st workshop aimed to: develop understanding of the different

models of national and local actions and activities being put in place

explore the evolving relationship between the approaches of the centre and of local areas

identify ways in which progress could be measured.

National workshops National workshops (2)(2)

2nd workshop aimed to: Review progress and learning in relation to

objectives of Choose Life Test out key themes emerging from

evaluation Identify future priorities for development,

support required, measures of progress and outcome, and implications for information collection and research

Data analysis Data analysis (1)(1)

Continuous (iterative) process throughout study Evaluation database designed and used to store

(and retrieve) data on all 32 local areas Findings from each element of data collection

(case studies, workshops, national interviews, surveys) written up in detailed reports which were then used for comparative analysis

Data analysed according to predefined themes (e.g. sustainability, partnerships) …

… and themes also developed from inductive analysis conducted at each phase of evaluation

Data analysis Data analysis (2)(2)

Analytic framework developed to guide team through the evaluation This was expanded and changed according

to themes emerging from data Analysis primarily drew on ‘charting’

method of systematically handling complex datasets by drawing out dimensions relating to each theme across all ‘cases’

Evaluation: main findingsEvaluation: main findings

Sustainable infrastructures for implementation

Allocation and use of resources Innovative practice and use of

evidence Sustainability Decision making processes and

learning

Sustainable infrastructures Sustainable infrastructures for implementationfor implementation

Demonstrable progress made by NIST CPP successes … but not as evident in

less ‘mature’ partnerships and in engaging with clinical services

Various models of local coordination developed Preference for a dedicated (full-time)

coordination post … … but evaluation unable to demonstrate the

superiority of this model

Allocation and use of Allocation and use of resourcesresources

CPPs have attracted considerable additional investment at local level and in-kind contribution …

… but areas have not been equally successful in raising additional funding …

… and there has been unnecessary duplication of effort at local level

Choose Life has stimulated a considerable amount of activity relating to self-harm …

… but local areas have different understandings of ‘high risk’ suicidal behaviour and have adopted different responses to address the problem

Innovative practice and Innovative practice and use of evidenceuse of evidence

Many examples of locally defined innovative practice …

… and multiple sources of information and evidence used to inform local planning and activity …

… but research rarely used systematically

SustainabilitySustainability

NIST identified several achievements in building a sustainable infrastructure for suicide prevention

At local level, most success achieved in mainstreaming training activities

Decision making processes Decision making processes and learningand learning

Local stakeholder consultation: key approach to set implementation priorities

Short timescale to develop first action plan was a major challenge

National support for learning has been delivered through diverse routes

NIST has highlighted a strong commitment to evaluation …

… but a national framework for evaluation remains to be completed

In local areas different levels of priority and attention have been attached to evaluation.

Recommendations: Recommendations: mainstreaming at national mainstreaming at national levellevel

Incorporate Choose Life objectives and priorities into other policy streams/initiatives

Involve clinical services in population-based suicide prevention activities

Involve national voluntary sector organisations in awareness raising and campaigning

Engage in purposive innovation to test out, evaluate, learn and implement

Recommendations: Recommendations: mainstreaming at local levelmainstreaming at local level

Using intelligence from a range of sources, as tools in planning for sustainability

Building in mechanisms to track and review progress towards objectives across policy areas

More focused targeting of action is required

Recommendations: self-Recommendations: self-harmharm

More consideration to be given in phase 2 to the integration of self-harm into Choose Life

The strategy should continue to encompass high risk self-harm …

… but the less ‘serious’ component of self-harm cannot be ignored

Recommendations: CPPRecommendations: CPP

The CPP remains the most appropriate vehicle for developing strategy and overseeing delivery in relation to Choose Life at the local level

But its limitations should be recognised Need to examine the necessary partnerships

that have yet to be put in place Priority should be given to establishing/building

on effective links with clinical & drug/alcohol services

NIST should continue to work closely with CPPs to ensure that Choose Life budgets are fully spent on suicide prevention activities

Recommendations: central Recommendations: central coordination bodycoordination body

Some type of central coordination body will continue to be required in the immediate future

Key tasks: provide national oversight, assess and support performance and ensure accountability at local level, promote learning and effective knowledge transfer, and co-ordinate action.

There should be a review of how the central coordinating function is delivered and where it is situated

Choose Life evaluation (first phase):Choose Life evaluation (first phase):research teamresearch team

University of EdinburghStephen Platt (RUHBC)Emma Halliday (RUHBC)Margaret Maxwell (General Practice)Scottish Development Centre for Mental HealthJoanne McLeanAllyson McCollamAmy WoodhouseLondon School of EconomicsDave McDaid (Health & Social Care)Glasgow UniversityMhairi Mackenzie (Public Health & Health Policy)Avril Blamey (Public Health & Health Policy)