Evaluation Design Designing & Refining Questions Ben Silliman Youth Development Specialist NC 4-H.
-
Upload
cody-graves -
Category
Documents
-
view
218 -
download
0
Transcript of Evaluation Design Designing & Refining Questions Ben Silliman Youth Development Specialist NC 4-H.
![Page 1: Evaluation Design Designing & Refining Questions Ben Silliman Youth Development Specialist NC 4-H.](https://reader035.fdocuments.net/reader035/viewer/2022081602/5513cccf5503466f748b4ad5/html5/thumbnails/1.jpg)
Evaluation DesignDesigning & Refining Questions
Ben SillimanYouth Development Specialist
NC 4-H
![Page 2: Evaluation Design Designing & Refining Questions Ben Silliman Youth Development Specialist NC 4-H.](https://reader035.fdocuments.net/reader035/viewer/2022081602/5513cccf5503466f748b4ad5/html5/thumbnails/2.jpg)
Who Knows What?
• Youth (over age 3) can report their– Feelings, interests, experiences, goals, opinions/
attitudes on knowledge and skills– (reliable observers may give opinions on these)
• General Rules for who answers what:– You can’t know what you don’t know (thus
retrospective or observer ratings usually better)– No one knows your feelings, experiences, and
attitudes/aspirations better than you(let respondents speak for themselves)
![Page 3: Evaluation Design Designing & Refining Questions Ben Silliman Youth Development Specialist NC 4-H.](https://reader035.fdocuments.net/reader035/viewer/2022081602/5513cccf5503466f748b4ad5/html5/thumbnails/3.jpg)
Who Knows What?
• Youth (over age 3) can complete valid, developmentally appropriate tests of– Knowledge, Skill, Practice(reliable observers may also give opinions on
these)
• How to know what others know– Demonstration, description, or defense
(authentic assessment) gives best evidence…esp. if judged by trained, reliable observers using the same criteria given to youth
![Page 4: Evaluation Design Designing & Refining Questions Ben Silliman Youth Development Specialist NC 4-H.](https://reader035.fdocuments.net/reader035/viewer/2022081602/5513cccf5503466f748b4ad5/html5/thumbnails/4.jpg)
Thinking About Questions
Adjust for Differences in Self-Administered vs. Interviewer-Administered Questionnaires
• Self: make it simple (Eighth Grade), straightforward, and logical. Q1-> Q2 (complex skip patterns, crowded pages, complex instructions delay and discourage respondents)
• Interviewer: accommodate skip patterns, allow for diverse lines of questioning, interaction about ideas, objects, events (note interviewer skills)
![Page 5: Evaluation Design Designing & Refining Questions Ben Silliman Youth Development Specialist NC 4-H.](https://reader035.fdocuments.net/reader035/viewer/2022081602/5513cccf5503466f748b4ad5/html5/thumbnails/5.jpg)
Thinking About Questions
Order questions for smooth flow– Be forthcoming about purpose and
sponsor– Group topic or type of questions– Move from general to specific questions– Ask the easy questions first-build rapport– Ask more complex or thought-provoking
questions near the middle—use rapport– Ask sensitive questions near the end-ditto
![Page 6: Evaluation Design Designing & Refining Questions Ben Silliman Youth Development Specialist NC 4-H.](https://reader035.fdocuments.net/reader035/viewer/2022081602/5513cccf5503466f748b4ad5/html5/thumbnails/6.jpg)
Thinking About Questions
Phrase questions carefully to avoid bias
• “Leading” the respondent– What do you like about… vs
– What, if anything, do you like…(not assuming respondents like
something)
![Page 7: Evaluation Design Designing & Refining Questions Ben Silliman Youth Development Specialist NC 4-H.](https://reader035.fdocuments.net/reader035/viewer/2022081602/5513cccf5503466f748b4ad5/html5/thumbnails/7.jpg)
Thinking About Questions
Keep audience focused• Avoid multiple-option items
– Did public speaking help you gain confidence and skill?
• Avoid complex phrases(Answer Never-Seldom-Often-Always)– This 4-H program did not have an
appropriate level of discipline
![Page 8: Evaluation Design Designing & Refining Questions Ben Silliman Youth Development Specialist NC 4-H.](https://reader035.fdocuments.net/reader035/viewer/2022081602/5513cccf5503466f748b4ad5/html5/thumbnails/8.jpg)
Thinking About Questions
Adjusting questionnaire length, remembering that– A relevant, purposeful study can
motivate cooperation and thought– A clear and smooth-flowing routine can
focus and energize respondents– -A respectful and gracious approach can
affirm respondents, making them feel valued
![Page 9: Evaluation Design Designing & Refining Questions Ben Silliman Youth Development Specialist NC 4-H.](https://reader035.fdocuments.net/reader035/viewer/2022081602/5513cccf5503466f748b4ad5/html5/thumbnails/9.jpg)
Which Questions to Use
• Open-ended: no set responseIs there anything else you would like to
tell about your 4-H camping experience?– Communicates listening– Allows evaluator to see how
respondents prioritize and organize information
![Page 10: Evaluation Design Designing & Refining Questions Ben Silliman Youth Development Specialist NC 4-H.](https://reader035.fdocuments.net/reader035/viewer/2022081602/5513cccf5503466f748b4ad5/html5/thumbnails/10.jpg)
Which Questions to Use
• Close-ended: set response choicesFor the 4-H project areas listed, rank (1st,
2nd, 3rd) each according to your interest:___ Arts & Communication___ Citizenship & Leadership___ Family & Consumer Science___ Health & Nutrition___ Science & Technology
![Page 11: Evaluation Design Designing & Refining Questions Ben Silliman Youth Development Specialist NC 4-H.](https://reader035.fdocuments.net/reader035/viewer/2022081602/5513cccf5503466f748b4ad5/html5/thumbnails/11.jpg)
Styles of Close-ended Qs
Likert (esp. for feelings, attitudes)How important are cash prizes in your
decision to participate in 4-H events?Not at all VeryImportant
Important1 2 3 4
5
![Page 12: Evaluation Design Designing & Refining Questions Ben Silliman Youth Development Specialist NC 4-H.](https://reader035.fdocuments.net/reader035/viewer/2022081602/5513cccf5503466f748b4ad5/html5/thumbnails/12.jpg)
Styles of Close-ended Qs
Multiple Choice (best or all relevant options)
What is the best thing to do when you plan not to use a gun for a while? (choose one)
a) Cover it with a clothb) Place it on top of a cabinetc) Remove ammunitiond) Call law enforcement
![Page 13: Evaluation Design Designing & Refining Questions Ben Silliman Youth Development Specialist NC 4-H.](https://reader035.fdocuments.net/reader035/viewer/2022081602/5513cccf5503466f748b4ad5/html5/thumbnails/13.jpg)
Styles of Close-ended Qs
Ordinal (ranking of options)Rank these topics in order of importance
(1st, 2nd, 3rd, etc) for volunteer training___ Managing a 4-H club___ Teaching using experiential method___ Planning service activities___ Conducting parties___ Mastering program evaluation
![Page 14: Evaluation Design Designing & Refining Questions Ben Silliman Youth Development Specialist NC 4-H.](https://reader035.fdocuments.net/reader035/viewer/2022081602/5513cccf5503466f748b4ad5/html5/thumbnails/14.jpg)
Styles of Close-ended Qs
Categorical (identifying by group)Tell us about yourself. I am a a) Girl b) Boy
Numerical (identifying by number)Tell us about yourself. My age is ___ years