Program Evaluation: The Structured Interview

swords-to-plowshares.org VETERAN EMPLOYMENT

COLLABORATION

Program Evaluation:The Structured Interview

War causes wounds and suffering that last beyond the battlefield. Swords to Plowshares’ mission is to heal the wounds of war, to restore dignity, hope, and self-sufficiency to all veterans in need, and to prevent and end homelessness and poverty among veterans.

Introduction

GOALS: PROVIDE

• Introductory to intermediate level content• Easy-to-use guide • Lots of practical tips• Stretch your thinking• Opportunity for your input• Time for Q & A

TOPICS

• Structured interviews in context of survey tools• Best practices• Protocol design• Question types• Tips for interviewers• Dealing with data


Introduction

SURVEY TOOLS

• Open-ended interviews (rare)• Semi-structured interviews• Structured interviews• Self-completed questionnaire forms (on line or on paper)


Interviews vs. Questionnaires

INTERVIEWS QUESTIONNAIRES More labor intensive Less labor intensive

Higher skill level Lower skill level

Can follow-up for clarity Not possible to follow up

Richer data Thinner data

Questions & activities Questions only

Less anonymous (courtesy bias) Anonymous

Using Structured InterviewsTake advantage of opportunity to discuss/exchangeEmployment placementExit interviews for staff and clientsExploring potential partnershipsConsider addressing issues related to outputs (activities) as well as outcomes (resulting improvements to quality of life)

Using Structured Interviews

→ Take advantage of opportunity to discuss/exchange

→ Employment placement→ Exit interviews for staff and clients→ Exploring potential partnerships→ Consider addressing issues related to outputs (activities) as well as outcomes (resulting improvements to quality of life)

Best Practices – Research Planning


GATHER INPUT FROM STAKEHOLDERS o Funder requirementso In-house development departmento Those responsible for data collection and management

CONSENSUS ON PROJECT GOALSo Reason for researcho Research questionso Application of findings

Best Practices - Research


METHODOLOGYo Match the tool to the research goals and contexto Recognize strengths and limits of toolo Pilot test all phases of protocol (recruitment, data collection, data management, analysis)

o Sample sizeWORKING BILINGUALLY

o Appropriate for regiono Back translation to test instrumento Allow for translation of datao Native speakers preferred

Best Practices - Respondents


HSS human subject research guidelines (well-being, respect, protect)

Culturally responsible approach (more than just linguistics)

Accessible content

Don’t assume literacy

Structured Interview


PROTOCOL DESIGN - GENERIC

Similar rules of thumb for questionnaires. Brief explanation of the purpose. Start easy…

• Break the ice• Move toward more complex or personal content• Explore the positives of X before turning to

examine the negatives “Areas of Inquiry”

• Begin open-ended… then hone in with pre-meditated prompts.

Wrap-up – it can be useful to get direct re: research goal.



PROTOCOL DESIGN - INTAKE

The respondent is likely to be seeking you out, rather than having been recruited for the interview.

The nature of this interaction, the intake is to collect/share lots of personal information. You might start off with demographics and service history, etc.

Caseworkers are comfortable working from unscripted forms.

Poll Questions


HOW WELL DO YOU THINK THE WEBINAR IS GOING?

1. Extremely well.

2. Very well.

3. Somewhat well.

4. It's okay.

5. Wait, there's something wrong with this question.



QUESTION DESIGN Similar rules of thumb for questionnaires.

NO nice-to-know questions.

Clear, concise wording (don’t make respondent struggle to understand).

NO double-barreled questions.

Examine draft for bias, leading or judgmental verbiage.

Give permission to respondent to disagree or be candid.

Prepare appropriate prompts and follow-up questions in advance.

Keep fixed response choices simple, consider using visual aid.

Use fixed response questions as springboards for conversation.

Question Type:


OPEN-ENDED WITH PRE-MEDITATED PROMPTS

Example: What can you tell me about today’s workshop? Was it useful?

Useful prompts when the respondent needs encouragement: Did you get any new ideas? Can you describe that/one for me? Did you pick up any information that you think will be helpful? What information?

Did you get your questions answered? What questions did/do you have? Were there any highlights? Did anything jump out at you? Were you able to _________________________________?

Question Type:


OPEN-ENDED WITH PRE-MEDITATED PROMPTS

Example: What can you tell me about today’s workshop? Was it useful?

Possible follow-ups or probes: Was that new information for you? How do you think you might be able to use/follow up… Is there anything that could have made it more useful…

Question Type:


FIXED RESPONSE

Multiple Choice & Ratings Scales Keep simple Consider providing a printout of the scale graphic Use as a springboard for discussion: Why did you score X the way you did?

What are 3 words that describe the program you attended today?

You can let this go (great on questionnaire forms too) or follow up on one of the words they offer.

Question Type:


USING IMAGES

Reflect on images to encourage discussion Be careful of getting too concrete, respondents can fixate on the example.

Card Sorts Print a set of images or phrases on cards which respondents can sort into categories.

Preferences, likelihood of behaviors or choices, level of interest or familiarity, etc.

Tips for the Interviewer


USEFUL ALL-PURPOSE FOLLOW-UPS

Remember, context and tone of voice make all the difference. Can you tell me more about that… about X? What do you mean when you say X? Can you think of an example? Was that what you had expected? Why do you think it was memorable? What made you think that? Why do you think that was/happened? How do you feel about that?



TECHNIQUE Consider yourself as a neutral but friendly extension of the survey instrument. Your professional, friendly manner helps the respondent feel confident.

After delivering your concise question, be willing to wait for a response, but be ready with prompts if needed.

Remember what you’ve learned about active listening and put it to use.

When probing for clarity, use respondents’ words back to them.

Once you get started, avoid narrating the process.



TECHNIQUE

Avoid “leading” the respondent by suggesting ideas or vocabulary. They can feel influenced to agree with your suggestion and/or struggle to make it fit their experience.

Natural, conversational prompts are helpful, e.g., “Mhm.” Avoid comments which include suggestions of judgement, e.g., “That’s a great observation!” You can say, “I’m interested to hear more about that.”

Sometimes starting a sentence and purposely leaving it open is a good way to prompt a respondent. E.g.: So, you would say that it….? So, does that mean you would…?

Recording


Audio recording

Handwritten notes

Entering into database at keyboard (intake)

Data Management & Analysis


Custom Databases

SurveyMonkey

Microsoft Excel

Coding open-ended data = categorizing

Poll Questions


HOW WOULD YOU RATE THE USEFULNESS OF THIS WEBINAR?

1. Very useful.2. Somewhat useful.3. Not very useful.4. About as useful as this question is without opportunity

to follow-up about the rating I give.5. More useful than answer No. 4 suggests.

Thank Your For Your Attendance

Copyright © 2017 by Swords to PlowsharesAll rights reserved, including the right of reproduction in whole or in part in any form.

Swords to PlowsharesThe California Veterans Employment and Training Collaborative 1060 Howard StreetSan Francisco, CA 94103Amy [email protected]

Wendy MeluchPrincipal & Founder [email protected](415) 897.4051

Program Evaluation: The Structured Interview

Government & Nonprofit

Transcript of Program Evaluation: The Structured Interview