1

Mixed Methods Research: State of the Art

(What Has Developed In Mixed Methods)

by John W. Creswell, Ph.D.

Department of Educational Psychology, University of Nebraska-LincolnCo-editor, Journal of Mixed Methods Research, and Co-Director, Office of Qualitative and Mixed Methods Research

© Please do not duplicate or use these slides without the express permission of the author.

2

The Story of Mixed Methods in the Last 20 Years

Highlight four developments

Why I can tell the story First generation mixed methods – reflect on my role in story Balanced – background in quan and qual 5 books on research methods Applied research methodologist Focused on education, social and human sciences Journal editor

What I am trying not to do “Fix” the history Discourage discourse Speak for all disciplines Limit the discussion to a US perspective Cover all developments

3

Four Developments (last 20 years)

Increasing interest in and advocacy for mixed methods Evolving understanding of what is mixed methods research Emerging philosophical perspectives as a foundation for mixed

methods Developing designs and innovative techniques

4

Development #1: Increasing Interest in and Advocacy for Mixed Methods

Research “movement” Federal funding interest Discipline interest International interest Books Journals Conferences Critics

Key Developments

5

“The emergence of mixed methods as a third methodological movementin the social and behavioralsciences began during the 1980’s.” (p. 697)

-Tashakkori & Teddlie (2003)

Mixed Methods asa “Movement”

6

New NIH Awards Using the Term "Mixed Methods"

0

5

10

15

20

25

30

35

40

45

1998 2000 2002 2004 2006 2008

Year

Nu

mb

er

of

New

Aw

ard

s

NewNIH Awards Using Mixed Methods

7

Development of NIH guidelines (1999)

- Mentioned severalapproaches for combiningqualitative and quantitative research- Advanced considerations for deciding what model to use (literatureavailable, prior studies, realistic design, expertise)- Suggested to describe each method thoroughly

“Combining qualitative and quantitative methods has gained broad appeal in public health research.” (1999)

8

NIH Summer Institute:The Design and Conduct of Qualitative and Mixed-Method Research in Social Work and Other Health Professions

August 4 -8, 2004  

Table of Contents

•Course Objectives •Dates and Location •Background •Concept •Institute Co-Chairs •Faculty/Presenters •Speaker Presentations •Eligibility Requirements •Number of Participants •Costs •Application Process and Receipt Date •Additional Information

Course Objectives To provide a thorough grounding in the design and conduct of qualitative and mixed method research to social work researchers and other health professionals interested in developing competence in the planning, design, and execution of these techniques and become more successful in the NIH extramural research program.

Social Work Summer Institute on Mixed-MethodsResearch

9

National Science Foundation Workshop, 2003, Includes Mixed Methods Research

-Provided generalguidelines for developingqualitative research projects-Several papers addressed mixed methods research

10

14+ books on mixed methods research have been written since 1988

Books onMixed MethodsResearch

11

Two New Books FromSage Publications

12

Editors: John W. Creswell and Abbas TashakkoriManaging Editor: Vicki L.Plano Clark

Email: vpc@unlserve.unl.edu

JournalsDevoted toMixed Methods:FieldworkQuality andQuantityMixed MethodApproaches

A NewJournal

13

Conferences

Mixed Methods Conference, Cambridge, England, July 21-24, 2008

Proposed mixed methods conference, Sydney, Australia, 2009

Discipline conferences with mixed methods papers

14

Critics (Creswell, 2007) Is there a post-positivist leaning to mixed methods?

Howe, K. R. (2004). A critique of experimentalism. Qualitative Inquiry, 10, 42-61.

Giddings, L. S. (2006). Mixed-methods research: Positivism dressed in drag? Journal of Research in Nursing, 11(3), 195-2003.

Holmes, C. A. (2007). Mixed(up) methods, methodology and interpretive frameworks. Contributed paper for the Mixed Methods Conference, Cambridge, University, July 10-12, 2006.

What are the limits for mixed methods research? Sale, J. E. M., Lohfeld, L. H., Brazil, K. (2002). Revisiting the quantitative-

qualitative debate: Implications for mixed-methods research. Quality and Quantity, 36, 43-53.

Leahey, E. (2007). Convergence and confidentiality? Limits to the implementation of mixed methodology. Social Science Research, 36, 149-158.

Is there a dominant discourse in mixed methods research?• Freshwater, D. (2007). Reading mixed methods research: Contexts for

criticism. Journal of Mixed Methods Research, 1(2), 134-146.

15

Development #2: Evolving Understanding of What is Mixed Methods Research

The emergence of diverse perspectives - Two strands to “mixing” to “methodology” to “mixed methods within designs”

Incorporation of perspectives into current definitions Questions arise about what is “mixing?” Recent assessment of the value of mixed methods research (or

mixing)

Key Developments

16

Framework for Viewing Perspectives on Mixed Methods

Quantitative Data Qualitative Data

MethodologyMethod

ParadigmPerspective Research Design

Procedures

Mixed Methods

17

Mixed methods is… a Method, a Design, a Methodology

Collecting data

Analyzing and interpreting data

Identifying a purpose and stating questions

Reporting and evaluating the study

Identifying a research problem

Reviewing the literature

18

Mixed Methods within Designs

Ethnography Case Study Research Narrative research Experimental research

Qual ----- Quan Qual----- Quan Qual-----Quan Qual---- Quan

19

What is this Method Called?

Multi-method Triangulation Integrated Combined Quantitative and qualitative methods Multi-methodology Mixed methodology Mixed-method Mixed research Mixed methods

20

A Definition

Mixed methods research is a methodology for conducting research that involves collecting, analyzing, and integrating (or mixing) quantitative and qualitative research (and data) in a single study or a longitudinal program of inquiry. The purpose of this form of research is that both qualitative and quantitative research, in combination, provide a better understanding of a research problem or issue than either research approach alone.

Aspects of this definition: Collecting both quantitative and qualitative data Mixing the data (from merging in which both quan and qual loses identity to keeping them distinct but connected) Single vs multiple-phase project

Raises issues: how, what, where, why, value (of mixing)

21

Mixing (How)

Results

Connect data:

Results

Converge data:

Embed the data:

Quan data

Qual data

Qual Quan

Qual Quan

22

Mixing

What Data (Methods) Qualitative and quantitative research Philosophical assumptions

Where Throughout the process of research Data collection Data analysis Interpretation

23

Mixing – Why? (Bryman, Qualitative Research, 2006)

Validity – to corroborate quan and qual data Offset – offset weaknesses of quan and qual and draw on strengths Completeness – more comprehensive account that qual or quan alone Process – quan provides outcomes; qual, the processes Different question – quan and qual answer different questions Explanation – qual can explain quan results or vice-versa Unexpected results – surprising results from one, other explains Instr development – qual employed to design instr, then instr tested Sampling – one approach facilitates sampling from other approach Credibility – both approaches enhance integrity of findings Context – qual provides context; quan provides general. Illustration – qual data helps develop “depth” for quan data Utility – more useful to practitioners Confirm – quan tests qual generated hypotheses Diversity of views – relationship and meaning; researcher/participant views Enhancement – augmenting or building on one form of data with the other

24

Value of “Mixing”

“What does mixed methods research provide that either quantitative or qualitative research alone does not provide?”

Indicators of “yield” (O’Cathain, Murphy, & Nicholl, JMMR, 2007) Integration as a measure of “yield” Publications as a measure of “yield”

My speculations: Results are useful Stronger impact on policymakers Cost-effective Better solutions through interdisciplinary input More evidence for a study

25

Development #3: Emerging Philosophical Perspectives as a Foundation for Mixed Methods Research

Key developments:

There are many stances taken by researchers. “Purist” stance - Paradigms are incompatible (fixed,

unmoveable), paradigms relate to methods, therefore mixing is impossible.

“Dialectic” stance - Multiple paradigms are possible and can be used, but honor and make explicit the paradigms.

“Single paradigm” stance - There is a single paradigm for mixed methods research: pragmatism, transformative.

“Design” stance - Multiple paradigms can be used – relate them to your design.

“Community” stance – source of belief systems come out of shared meanings and understanding

26

Philosophical Assumptions Behind Mixed Methods Research

Worldview or philosophy (e.g., attitudes and beliefs about knowledge, such as constructivism, post-positivism)

Theoretical lens (e.g., feminist, racial)

Methodological approach (e.g., experimental, survey, ethnography, mixed methods)

Methods of data collection (e.g. interviews, focus groups)

Adapted from Crotty M. (1998).

27

PostpositivismDeterminationReductionismEmpirical observation and measurementTheory verification

ConstructivismUnderstandingMultiple participant meaningsSocial and historical constructionTheory generation

Advocacy/ParticipatoryPoliticalEmpowerment issue-orientedCollaborativeChange-oriented

PragmatismConsequences of actionsProblem-centeredPluralisticReal-world practice oriented

Four Worldviews

28

Smith & Heshusius, (1986)

“Incompatibility thesis”

“The contention of thispaper is that the claim ofcompatibility, let alone one of synthesis, cannot besustained.” (p. 4)

Purist Stance

29

“Dialectic Stance”

•Paradigms – philosophical assumptions,knowledge, methodology, values• Paradigms are different – but are social constructions – not sacrosanct• Differences should be respected,intentionally used together• Used together, tensions achieve dialectical discovery of enhanced,reframed, or new understandings.

Greene (2007)

30

“Single” Paradigm Stance

Pragmatism (Tashakkori & Teddlie, 2003) 13 writers endorse Based on Dewey, Rorty, Pierce, others (Americans) Focus on research question What “works,” consequences Multiple methods – large toolkit

Transformative (Mertens, 2003) Transform lives of underrepresented, marginalized groups Incorporate this focus into all phases of research

31

“Design” Stance (Creswell & Plano Clark, 2007)

Multiple paradigms can be used Different paradigms can provide the foundation for different

phases of research Often post-positivist paradigm provides the foundation for

quantitative aspects of research Often constructivist/advocacy paradigm influences

qualitative aspects of research Often pragmatism paradigm influences both qualitative and

quantitative aspects of research This does not suggest that certain paradigms

are linked to certain methods (paradigm-method fit)

32

“Community” Stance (Morgan, 2007)

Espouses the pragmatic approach Shifts conversation to the importance of

shared meanings and joint actions between researchers.

“To what extent are two people (or two research fields) satisfied that they understand each other, and to what extent can they demonstrate the success of shared meaning by working together on common projects.” (Morgan, 2007, p. 67)

33

Development #4: Developing Designs and Innovative Techniques

Evaluation designs Notation Typologies, and a parsimonious set Diagrams of procedures Complex evaluation models Reconceptualizing designs The practice of research Emergence of innovations in procedures

Unusual blends Methodological issues Data analysis techniques Presentation techniques

Key Developments

34

• Triangulation (convergence,corroboration)• Complementarity(clarify results fromone method to theother)• Development (one method informsthe other method)• Initiation (paradox,contradiction)• Expansion (extendsbreadth of inquiry)

Early Designs(Greene, Caracelli,& Graham, 1989)

35

Preliminary Design Considerations (Morse, 1991)

Approach Type Purpose Limitations Resolutions

QUAL + quan Simultaneous Enrich description of sample

Qualitative sample

Utilize normative data for comparison of results

QUAL Sequential Test emerging H, determine distribution of phenomenon in population

Qualitative sample

Draw adequate random sample from same population

QUAN + qual Simultaneous To describe part of phenomena that cannot be quantified

Quantitative sample

Select appropriate theoretical sample from random sample

QUAN Sequential To examine unexpected results

Quantitative sample

Select appropriate theoretical sample from random sample

quan

qual

36

QUANData & Results

QUANData & Results InterpretationInterpretation

QUALData & Results

QUALData & Results

QUANPre-test Data & Results

QUANPre-test Data & Results

QUANPost-test

Data & Results

QUANPost-test

Data & Results

Intervention

qualProcess

qualProcess

InterpretationInterpretation

Triangulation Design

Embedded Design

Concurrent Mixed Methods Designs

Parsimonious Designs (Creswell & Plano Clark, 2007)

37

Sequential Designs Mixed Methods Designs

QUANData & Results

QUANData & Results

InterpretationInterpretation

qualData & Results

qualData & Results

Following up

QUALData & Results

QUALData & Results

quanData & Results

quanData & Results InterpretationInterpretation

Building to

Before-intervention

qual

Before-intervention

qual

QUANIntervention

Trial

QUANIntervention

Trial

After-

intervention qual

After-

intervention qual InterpretationInterpretation

Exploratory Design

Explanatory Design

Sequential Embedded Design

38

Helpful Tips for Drawing this Visual of the Design:

1. Give a title to the visual model.2. Choose either horizontal or vertical layout for the model.3. Draw boxes for quantitative and qualitative stages of data

collection, data analysis and interpretation of the study results.4. Use capitalized (QUAN) or small letters (quan) to designate priority

of quantitative and qualitative data collection and analysis.5. Use single-headed arrows to show the flow of procedures in the

design. 6. Specify procedures for each quantitative and qualitative data

collection and analysis stage.7. Specify expected products or outcomes of each quantitative and

qualitative data collection and analysis procedure.8. Use concise language.9. Make your visual diagram simple.10. Size your visual diagram to one page.

Source: Ivankova, N. et al. (2006)

39

Example ofA Diagram

Complex Mixed MethodsEvaluation Designs (Nastasi, Hitchcock,Sarkar, Burkholder,Varjas & Jayasena, 2007)

Study of mental healthinterventions for youthin Sri Lanka

Concurrent,Sequential,Recursive Qual/QuanPhases(Nastasi et al.,2007)

42

Reconceptualizing Designs

Three ways to look at mixed methods designs:

Typological approach (Bryman, 1988; Creswell & Plano Clark, 2007)

Systems theory (Maxwell & Loomis, 2003)

Synergistic approach - Hall & Howard (in press), “A Synergistic Approach: Conducting Mixed Methods Research with Typological and Systemic Design Considerations”

Combined effect greater than separate parts Position of equal value of qual and quan Ideology of difference – protect paradigm differences Researcher as collaborator and relationship with design – skilled in

both quan and qual or part of interdisciplinary team

43

The Practice of Conducting A Mixed Methods Study

Cordon & Hirst (in press) “Implementing a mixed methods approach to explore the financial implications of death of a life partner” – Social Policy Researchers, UK

Why they chose mixed methods Reasons for their choice of design Sampling decisions Concept map Discussions among team members

44

Technique - Designing a Quantitative Survey Based on Qualitative Findings

QUAL data analysis

Quotes

Codes

Themes

Quan data analysisinstrument development

Items on a survey

Variables on a survey

Scales on a survey

45

CountAdj.Count** Row PctColumn Pct

PatientsN=2

PhysiciansN=4

Medical AssistantsN=4

Themes

FamiliarityWith the Form

136.541.95.8

174.2527.43.0

194.7530.75.3

4915.5100.0

Reactions to the Form

2311.522.210.2

10025.048.317.4

6115.329.516.9

18451.75100.0

Use for Managing Depression

6733.538.629.8

17744.2551.030.7

369.010.410.0

28086.75100.0

Changes to the Form

11557.537.551.1

19649.032.034.0

18746.830.551.7

498153.3100.0

Situational Use of the Form

73.58.93.1

8621.554.414.9

5814.536.716.1

15139.5100.0

225112.5100.0

576144.0100.0

36190.3100.0

Technique- Matrix with Quantitative and Qualitative Data

46

Technique –Raising and Addressing Methodological Issues in Designs

Creswell, Plano Clark, Garrett (in press) Contradictory findings Data merging Sample selection Sample size Introducing bias Time

47

Technique – Unusual Blends of Data Collection

Qualitative Post-modern philosophy

and Longitudinal data and Qualitative data and Qualitative themes

matched

Qualitative discourse data

Quantitative Quantitative data

Longitudinal data Secondary survey data With items on survey to

produce new quan variables

Survey data

48

Other Developments:

International emphases Discipline differences Research team practices The emergence of bilingual nomenclature

(will employ this thinking in the workshop this afternoon as we “title” projects)

49

Four Developments (last 20 years)

Increasing interest in and advocacy for mixed methods Evolving understanding of what is mixed methods research Emerging philosophical perspectives as a foundation for mixed

methods Developing designs and innovative techniques

•Bryman, A. (1988). Quantity and quality in social research. London: Routledge.•Bryman, A. (2006). Integrating quantitative and qualitative research: How is it done? Qualitative Research 6(1), 97-113.•Cordon, A., & Hirst, M. (in press). Implementing a mixed methods approach to explore the financial implications of death of a life partner. Journal of Mixed Methods Research. •Creswell, J. W., & Plano Clark, V. L. (2007). Designing and conducting mixed methods research. Thousand Oaks, CA: Sage.•Creswell, J. W., Plano Clark, V. L., & Garrett, A. L. (in press). Methodological issues in conducting mixed methods research designs. In M. Bergman (Ed.), Advances in mixed methods research. London: Sage.•Crotty M. (1988) The foundations of social research: Meaning and perspective in the research process. London: Sage.•Freshwater, D. (2007). Reading mixed methods research: Contexts for criticism. Journal of Mixed Methods Research, 1(2), 134-146.•Giddings, L. S. (2006). Mixed-methods research: Positivism dressed in drag? Journal of Research in Nursing, 11(3), 195-2003.•Greene, J. C. (2007). Mixed methods in social inquiry. San Francisco: Jossey-Bass.•Greene, J. C., Caracelli, V. J., & Graham, W. F. (1989). Toward a conceptual framework for mixed-method evaluation designs. Educational Evaluation and Policy Analysis, 11(3), 255–274.•Hall, B., & Howard, K. (in press). A synergistic approach: Conducting mixed methods research with typological and systemic design considerations. Journal of Mixed Methods Research.•Holmes, C. A. (2007). Mixed(up) methods, methodology and interpretive frameworks. Contributed paper for the Mixed Methods Conference, Cambridge, University, July 10-12, 2006.•Howe, K. R. (2004). A critique of experimentalism. Qualitative Inquiry, 10, 42-61.•Ivankova, N. V., Creswell, J. W., & Stick, S. (2006). Using mixed methods sequential explanatory design: From theory to practice. Field Methods, 18(1), 3–20.•Leahey, E. (2007). Convergence and confidentiality? Limits to the implementation of mixed methodology . Social Science Research, 36, 149-158•Maxwell, J., & Loomis, D. (2003). Mixed methods design: An alternative approach. In A. Tashakkori & C. Teddlie (Eds.), Handbook of mixed methods in social & behavioral research (pp. 241-271). Thousand Oaks, CA: Sage.

References

•Mertens, D. M. (2003). Mixed methods and the politics of human research: The transformative-emancipatory perspective. In A. Tashakkori & C. Teddlie (Eds.), Handbook of mixed methods in social and behavioral research (pp. 135–164). Thousand Oaks, CA: Sage.•Morgan, D. L. (2007). Paradigms lost and pragmatism regained: Methodological implications of combining qualitative and quantitative methods. Journal of Mixed Methods Research, 1(1), 48-76.•Morse, J. M. (1991). Approaches to qualitative-quantitative methodological triangulation. Nursing Research, 40, 120–123.•Nastasi, B. K., Hitchcock, J., Sarkar, S., Burkholder, G., Varjas, K., & Jayasena, A. (2007). Mixed methods in intervention research: Theory to adaptation. Journal of Mixed Methods Research, 1(2), 164-182.•National Institutes of Health (2004). NIH Summer Institute: The design and conduct of qualitative and mixed-method research in social work and other health professions, August 4-8, 2004.•National Institutes of Health. Office of Behavioral and Social Sciences Research (1999). Qualitative methods in health research: Opportunities and considerations in application and review. Washington D.C.: NIH.•National Science Foundation (2003). Workshop on scientific foundations of qualitative research. Washington D.C.: NSF.•O’Cathain, A., Murphy, E., & Nicholl, J. (2007). Integration and publications as indicators of “yield” from mixed methods studies. Journal of Mixed Methods Research 1(2), 147-153.•Plano Clark, V. L., & Creswell, J. W. (2008). The mixed methods reader. Thousand Oaks, CA: Sage.•Sale, J. E. M., Lohfeld, L. H., Brazil, K. (2002). Revisiting the quantitative-qualitative debate: Implications for mixed-methods research. Quality and Quantity, 36, 43-53.•Smith, J. K., & Heshusius, L. (1986). Closing down the conversation: The end of the quantitative-qualitative debate among educational inquirers. Educational Researcher, 15(1), 4-12.•Tashakkori, A., & Teddlie, C. (Eds.). (2003a). Handbook of mixed methods in social and behavioral research. Thousand Oaks, CA: Sage.

References (Cont’d)

52

Mixed Methods Research: State of the Art

(What Has Developed In Mixed Methods)

by John W. Creswell, Ph.D.

Department of Educational Psychology, University of Nebraska-LincolnCo-editor, Journal of Mixed Methods Research, and Co-Director, Office of Qualitative and Mixed Methods Research

© Please do not duplicate or use these slides without the express permission of the author.