1

Qualitative Methods in Institutional Research

WorkshopCAIR Annual MeetingsNovember 12, 2003

Judith Richlin-Klonsky, Ph.D.UCLA Student Affairs Information &

Research Officejrichlin@saonet.ucla.eduhttp://www.sairo.ucal.edu

2

The goals of the workshop

o To outline the issues that need to be considered when undertaking a qualitative research project.

o To describe experiences with specific qualitative research techniques.

o To respond to questions about using qualitative methods in institutional research.

3

Our agenda

Introductions Overview of qualitative methods in

institutional research (JR-K)

Quick & Easy Focus Groups (WW)

Blitzkrieg Ethnography (BB)

“Fishbowl” Discussions (JR-K)

Q&A/Discussion

4

What do we mean by “qualitative research?”

Examples of qualitative data-collection techniques:

o Individual interviewso Group interviewso Focus groupso Ethnographic observationo Participant-observationo Archival and/or documentary researcho Journaling

(see handout for descriptions)

5

What do we mean by “qualitative research?” (2)

Not just a set of techniques.

Not just “non-quantitative” research methods.

6

What do we mean by “qualitative research?” (3)

Research design and methods that help us fulfill specific goals as researchers:

By gathering data through participants’ own words and/or (inter-)actions, we seek to learn about the phenomenon being studied from their point of view.

By spending time with experts (those directly involved), we seek to learn in depth about the phenomenon being studied.

7

What do we mean by “qualitative research?” (4)

Unlike most quantitative institutional research, qualitative research typically doesn’t:

o Measure. (Ex: “What is the rate of graduation?”)o Produce a quantified product (Ex: “86% of our

students are satisfied with their learning experience.”)

o Seek to establish a causal relationship. (Ex: “Summer attendance shortens time to degree.”)

8

“But it’s not scientific…”

“Whether qualitative techniques are considered scientific description or explanation, or whether they offer merely a prelude to scientific inquiry, depends on the philosophical stance taken with respect to the nature of social science. Indeed…some have argued that qualitative data provide the only empirical foundation on which social science can be built” (Walker, p. 3).

9

Research design: When is qualitative research most useful?

When the research topic is:o Sensitive. o Complicated. o Unmeasurable. o Concerned with interaction and/or

process.

10

Research design: When is qualitative research most useful? (2)

When the research objective is:o To learn about research subjects who are

few in number.o To brainstorm (generate as many ideas as

possible, not reduce to a single number).o To identify important issues to be explored

more broadly through quantitative methods.o To interpret, illuminate, or illustrate.

11

Examples of data-collection issues

Individual & group interviews, focus groups

Recruitment

Representativeness

Ethnographic observation Site selection & access

Participant-observation Recruitment & training of participants to be effective observers

Archival &/or documentary research

Access

Journaling Data management

12

Data collection issues

All:

How to maintain subject privacy and demonstrate to IRB that you are prepared to maintain anonymity or confidentiality?

13

Data analysis

Analysis is aimed at• Answering the research question.• Identifying themes, categories, or

types.

14

Data analysis (2)

o “Analysis of qualitative material is more explicitly interpretive, creative, and personal than in quantitative research, which is not to say that it should not be equally systematic and careful” (Walker, p. 4).

15

How do we know when we have “valid results?”

SaturationWhen not hearing new types of experiences,

observations, roles, etc.

Triangulation within qualitative methodsData gathered at different times or places, at

different levelsBetween multiple observers of the same

phenomenon.

Triangulation between methodsQuantitative and qualitative data

16

Reporting issues

“The report on a qualitative study will be completely different than that on a quantitative study. The structure for the report will have emerged from that used in the analysis: the findings will be presented in terms of impressions gained, as hypotheses rather than as firm conclusions.

[continued]

17

Reporting issues (2)

…There will be no tables… …and no references to numbers or percentages; at most the researcher will use such phrases as ‘most respondents’ or ‘around half’ or…‘a minority view was….’ The researcher will be mainly concerned to identify and describe the range of behavior and options rather than to indicate…how many hold each view. In all cases the description of beliefs, attitudes, and motivations should be supported by evidence in the form of verbatim quotations form the interviews and discussions” (Morton-Williams, in Walker, p. 41).

18

Reporting issues (3)

How to encourage the audience to give appropriate attention and weight to the data?

o Not too little: avoid being dismissed because of “small N.”

o Not too much: avoid emotional effects of direct participant voices.

19

Selected resources: qualitative methodso Sharon B. Merriam, Qualitative

Research and Case Study Applications in Education, John Wiley & Sons, 1998.

o Sharan B. Merriam & Associates, Qualitative Research in Practice: Examples for Discussion and Analysis, Jossey-Bass, 2002.

o Jean J. Schensul and Margaret D. LeCompte, Editors, Ethnographer’s Toolkit, Altamira Press, 1999.

20

Selected resources: qualitative methods (2)

o Robert Walker, Editor, Applied Qualitative Research, Gower Publishing Company, 1985.

o Elizabeth Whitt, “’Making the Familiar Strange’: Discovering Culture,” in Cultural Perspectives in Student Affairs Work, George D. Kuh, Editor, American College Personnel Association, 1993.

21

Transition

The following slides describe use of student “fishbowl” discussions.

22

UCLA Student “Fishbowl” Discussions

Judith Richlin-Klonsky, Ph.D.UCLA Student Affairs Information & Research Officejrichlin@saonet.ucla.eduhttp://www.sairo.ucla.edu

23

The “Fishbowl” Discussions: 1999-2003

Format Background Permutations

24

Five “Fishbowl” permutations

1999 2000 2001 2002 2003

First yrs First yrs Soph. Junior Seniors

Senior Senior Seniors Grad students

--

25

Recruitment and planning(see appendixes of reports at http://www.sairo.ucla.edu, “Recent reports”)

Campus leaders only as general participants. Invitations to be considered for participation:

“ Cold calls” based on lists generated from student records, using parameters related to gender,

race/ethnicity, class standing, source direct/transfer. Referrals from Student Affairs directors. Ad in Daily Bruin

Offered incentive as “thanks” for participation Made clear it would be videotaped Topic areas identified in dialogue with Student Affairs leadership.

26

Selection

o Participation not automatic, not promised.o Each candidate contacted personally for a

short conversation.o Selection based on perceived contribution

to group “mix,” ability and interest to share experience, energy.

o Frequent follow-up contact prior to event.

27

Room set up

Varied by year, depending on anticipated size of audience and location: classroom; stage; in law school classroom “well”; tiered lecture hall; meeting room.

Goal is to arrange students so that they can easily conduct conversation, but that audience can see and hear them

Facilitator and assistant sit with students.

28

Types of subjects addressed

Academic experience Co-curricular activities Personal and academic development Perceptions of campus climate Access to/use of student services Residential life (especially crowding) Use of technology

29

Afterwards

Refreshments Mingling

30

Reviewing and summarizing the data

Review notes taken during discussions. View duplicated copy of videotape. Make notes from videotape, transcribing

comments that concisely capture main points being made.

Use notes to summarize student comments in major areas covered in discussion.

31

Summaries

Full reports and executive summaries posted on the SAIRO web site at http://www.sairo.ucla.edu (“Recent Reports”).

32

Examples

Campus Diversity“I came from a very heterogeneous background, mostly minorities… My thinking was when I left high school, oh cool, I’m going to UCLA and there’s going to be all these other people, but this time it won’t be cliqued, meaning that all the pilipinos will be together and all the [shrug] will be together. But when I went to college, it was more of the same… and I was surprised especially with all of the diversity issues coming up and especially with the abolition of affirmative action… I kinda find it weird that each of their ethnic group clubs have outreach for their ethnic group and it is cliqued even more.” (ud, 21:24:32)

33

Examples

Campus diversity

“It’s such a diverse campus, but people [of the same background] still congregate together.” [ld]

“When I walk around campus, the only ones who are smiling at you are the ones that look like you.” [ld]

34

Examples

Transition to UCLA

“I’ve been told for the last two years [in hs], ‘College is going to be such a different experience for you.’ And then I got here and it was more different than I ever thought. You can’t describe it. You can’t explain it. It’s just a transition. And it’s just a transition. And it’s a different part of life that you have to go through…. Just be prepared to change your outlook on academics and everything.” [ld]

35

Examples

Learning to persevere

A member of the senior group used the metaphor of UCLA as “an emotional roller coaster” and the image resonated with other participants as well. Part of the bumpy ride they described was the discouragement they felt after arriving as someone who had been very “impressive” in high school and then feeling “thrown in a pile” with other high-achieving students. Another student agreed that at UCLA you have to “cope with ups and downs, have to deal with discouragement regularly.”

36

Examples

Learning to persevere

Other seniors described in various ways their personal developments in light of the challenges they faced in a large, complex, and competitive setting:

UCLA does grind you into a tough person. In applying for jobs, you have that toughness going for you – created by adversity.

UCLA has taught me to persevere.

37

Examples

Student culture: time to degree

“Never once during this fifth year have I regretted that decision. It was the best thing. It helped me mature and I think I would really encourage first year students…to consider a fifth year…” (21:14:57)

38

Reports

Full report and executive summary distributed to all those invited to the event: Student Affairs AVCs and directors, Chancellor and his executive team, deans.

Posted on web site.

39

Uses

Some in audience moved and/or energized by hearing from students, especially in early years and in 2003 “good-bye” discussion.

Concrete changes made, e.g., changes in hours of operation for some Student Affairs offices

Contribution to planning for various areas (e.g., technology development, external affairs).

Format also used in session for student loan counselors and planned as a forum for staff experiences.