Dr. G. Johnson, www.ResearchDemystified.org 1

Qualitative Data Analysis

Chapter 11, Research Methods for Public Administrators Dr. Gail Johnson

Revised & expanded for PADM 522, Designing Applied Research

Dr. Mario Rivera

Dr. G. Johnson, with additional material by Dr. M. Rivera 2

Analyzing Qualitative Data

A process of making sense of non-numeric data. Data from: Narrative documents (speeches, newspapers, diaries, reports, etc) Open-ended interviews Open-ended questions on a survey Case study (as the principal method or as embedded in a larger

complex of qualitative data and forms of analysis). Often, qualitative evaluation research incorporates cross-case

analysis: For instance in evaluating two sites of the same program, or in assessing a program site against another without the intervention. An example might be a school with an extended learning program to others without it.

Dr. G. Johnson & Dr. M. Rivera 3

Qualitative Data AnalysisData From Focus groups transcripts; interview transcripts and notes Unstructured observations Document review Videos, etc.

The goal is to identify common themes Requires a solid plan, attention to detail, good organization and

sufficient time Whether the analysis is done by computer or by hand, it is often

necessary to develop a coding scheme so the data can be systematically organized and analyzed

Computer software can help locate and organize data according to the coding scheme created

Dr. G. Johnson, Dr. M. Rivera 4

Qualitative Data Analysis But the analysis—that is, making sense of the data, discovering

the story the data reveals—is done by the researcher(s) The greatest concern is bias/paradigm blinders: hard to

recognize things you do not expect, or to avoid finding things you are inclined to find.

Maintaining the Thread Review the data, make notes as you go along Read again once all the data collection is completed

Organize the data—according to research questions, by date, by geographic location—what makes sense given the situation

Begin to look for recurrent themes. In one cross-case evaluation of a family-based drug education program, for instance, the instructor (Rivera) found clear commonalities and divergences in program-development in two Navajo communities in New Mexico.

Dr. G. Johnson, Dr. M. Rivera 5

Basic Activities: Preliminary Analysis Identify common words, ideas, themes Develop spreadsheet or write on index cards Identify “quotable quotes”—that is, quotes that highlight

the key issues: general views, divergent views, a range views. In evaluating an afterschool science program for middle-school students, Dr. Rivera conducted short interviews with students and with their parents, on various occasions. These exchanges produced quotes pertaining to favorite and least favorite parts of the program, parents’ views on program impact on students, etc. In another program evaluation, short quotations from participants supplemented survey results.

Dr. G. Johnson, Dr. M. Rivera 6

Qualitative Data Analysis: Coding Process When applicable (focus groups, content analysis of documents),

code the data Identify common word, issues, themes and go through the

material and label them according to that coding scheme You may go through the material several times Sometimes a major theme emerges at the end and you will need

to go back through to see if it was present in the earlier data Software is available that may help. Major software are

ATLAS.ti and NUD*IST, others like NVivo are specialized for various uses. There is growing literature on the use of computer- assisted qualitative data analysis software (CAQDAS).

Researchers might also use spreadsheets, index cards, color coding of documents, or other devices or combinations of these.

Dr. G. Johnson, Dr. Mario Rivera 7

Ensuring Quality in Qualitative Analysis

Work to ensure inter-rater reliability Like content analysis, the researchers should review the same

material and apply the coding scheme Then review and determine if there are difference Discuss, revise the coding scheme if necessary and retrain the

coders Repeat this test until there is agreement in the coding

Compare results, work out differences, then code all the material Inter-rater reliability was a major concern in the assessment of

teacher dossiers for advancement in tier, in Dr. Rivera’s evaluation of Three-tier Teacher Licensure in New Mexico in 2003-2004. Several trial (but real) assessments of dossiers by 3-person rater teams, of teacher volunteering to be thus evaluated, were tried during 2004-05, a transitional year for the new system.

Qualitative Data Analysis: General Process Interpreting the data

Making sense of the data: Review salient themes Are there any seeming relationships between the themes or

characteristics? What are the major points that emerge from the analysis?

What about minor points? Sometimes an uncommon theme stands out, or is noted by one or two

investigators The researchers need to consider that perspective; always indicate

when it is a minority view and be careful that it does not trump majority views

When evaluating programs, principal investigators and program managers are often engaged in conjoint analysis by or with the evaluator. Eliciting a variety of views is important.

These are stakeholders, and stakeholders, when really engaged, make us smarter. Cf. Chen on “stakeholder validity” vs. scientific validity.*

*Chen, H. (2005). Practical program evaluation: Assessing and improving planning, implementation, and effectiveness. Thousand Oaks: Sage Publications.

One stakeholder engagement technique: the Affinity Diagram A group process for analyzing qualitative data:

Research team reviews the material In silence, write down each idea, key work theme on a sticky

note In silence, post on wall In silence, sort into similar categories Once the notes are organized, discuss; look for consensus

themes Interpret the data with stakeholders

Reality check: do others involved in program or project agree? Share preliminary drafts with stakeholders or a small group of

the participants to explore the issues you have discovered Share final draft with experts and cold readers

Affinity Diagram Identify common themes The Affinity Diagram tool is a quick way to set up a coding

framework It is also a great way to identify the major themes and topics in

the final article or report It has the advantage of capturing all ideas in a room Everyone’s ideas have equal value when posted

Focus on major themes: “Some said X, while others said Y . . .” Avoid generalizations: if not everyone was asked to comment

about a particular practice, you do not know if the five who commented represented the views of others or were just the only ones that thought this practice was important.

Dr. G. Johnson, Dr. Mario Rivera 11

Leading Discussion and Writing Up Results—Could function as a facilitated focus group

Provide a range of perspectives. Highlight interesting perspectives even if only said by one

or two people. A few offered unique views: “….” These might be important because “…..”

Do not try to report numbers or percents unless everyone was counted in exactly the same way For example, 10 people out of the 20 who you interviewed

commented that they liked the training program. But unless you asked all 20 people, you do not know what the

other 10 people thought.

Limitations of Qualitative Data Analysis It can be like a Rorschach test –an inkblot that some see as a

butterfly and others see as maple leaf Reasonable people can read the same material and have very

different interpretations. It is important to explore those differences to gain a deeper understanding rather than to win the argument

It is much more difficult to generalize from qualitative analysis than otherwise. Questions of validity, reliability, replicability, etc., inevitably arise with qualitative data. These may be overweighed by the richness of insight gleaned from the data (“thick description”), but these challenges need to be taken into account and addressed with some directness in research reports.

Best when combined with quantitative analysis in mixed-methods research, which allows for triangulation of data and methods alike.

Dr. G. Johnson, www.ResearchDemystified.org 13

Creative Commons

This PowerPoint is meant to be used and shared with attribution

Please provide feedback If you make changes, please share freely and send

me a copy of changes: Johnsong62@gmail.com

Visit www.creativecommons.org for more information