Empowering youth to be evaluators: Involving Young People in Evaluating Informal Education Programs...
-
Upload
mn-association-for-environmental-education -
Category
Documents
-
view
1.225 -
download
0
Transcript of Empowering youth to be evaluators: Involving Young People in Evaluating Informal Education Programs...
3/23/10
1
Empowering Youth To Be Evaluators: Involving young people in evaluating
informal education programs
Amy Grack Nelson, Evaluation & Research Associate Science Museum of Minnesota
Overview
Overview of participatory evaluation
Participatory evaluation examples
Sampling of interactive techniques
What is participatory evaluation?
3/23/10
2
It’s All About Utility
Utility - one of the four essential features of all evaluations
(Joint Committee on Standards for Educational Evaluation, 1994)
A way to help ensure use is to increase the primary intended users’ level of participation in the evaluation. (Cousins & Earl, 1995; Patton, 2008)
Participatory Evaluation
“Applied social research that involves trained evaluation personnel and practice-based decision makers working in partnership.” (Cousins & Earl, 1995, pg. 8)
Core purpose increasing use
Characteristics of Participatory Evaluation Balanced control of evaluation process Involvement of primary users Extensive participation throughout the evaluation
(Cousins & Earl, 1995; Cousins & Whitmore, 1998)
Interactive Evaluation Practice Continuum
(King & Stevahn, 2002)
3/23/10
3
Benefits of Participatory Evaluation
Increases use of evaluation results by: Enhancing relevance of the evaluation Increasing understanding of the data Increasing ownership of the findings
(Cousins & Whitmore, 1998; King & Stevahn, 2002; Patton, 2008)
Evaluation capacity building Develop analytic and evaluative skills Stakeholders develop a more “critical eye”
(Cousins & Earl, 1992, 1995)
Science Museum of Minnesota’s Kitty Andersen Youth Science Center
Participatory Evaluation Examples
Kitty Andersen Youth Science Center’s Park Crew
Facilitate earth science and environmental education activities in the Big Back Yard and on outreaches
3/23/10
4
Example 1: Summative Evaluation
Youth will… Learn about water related earth surface processes Develop teaching skills Learn about related science, technology, engineering, and
math (STEM) careers
Evaluation design Observations and interviews of youth staff at the
beginning and end of summer Evaluation workshop to engage youth in results
Reviewing the Work
Keep/Change Discussion
3/23/10
5
Incorporating Evaluation Data
How often youth talked about why something is considered a pollutant (n=27)
Keep/Change Discussion
Engages users with data to think about successes and areas of improvement
Can be used to generate recommendations
Takes considerable amount of time
Benefits Limitations
3/23/10
6
Youth Benefits
Meaningful involvement in evaluation can help youth develop higher order thinking skills, specifically analytic and evaluative skills. (London et al., 2003)
Youth became more reflective of their work.
Youth comments reflected increased knowledge of the activities and confidence in sharing that knowledge with visitors.
Youth had a stronger sense of ownership and control.
Adult Staff Benefits
Provided important feedback about the crew’s work.
Gained deeper understanding of the participants’ experience and could proactively identify and respond to their needs.
Increased understanding of evaluation and ability to interpret data and generate recommendations.
Developed capacity to include participatory evaluation in future work.
3/23/10
7
Outcomes of the Process
Participants experience a sense of empowerment and pride when they have an influence on the way programs are run and see their ideas acted upon.
(Checkoway et al., 2003; Horsch et al., 2002; London et al., 2003)
Youth used suggestions to develop their own training.
They created a visitor survey and collected data.
Youth shared their ideas with a museum operations staff member.
Example 2: Formative Evaluation of Outreaches
Evaluation Process
Identify daily objectives
Craft survey questions Pilot surveys
Discuss pilot data and revise
surveys
Administer surveys
Enter and code data
Analyze and discuss data
Generate recommendations
Improve outreach activities
3/23/10
8
Organizational Requirements for PE
Evaluation must be valued
Sufficient time and resources
Commitment to organizational learning as a means to improvement
Motivated individuals
Interest and ability to learn evaluative skills
(Cousins & Earl, 1992)
Evaluator Requirements for PE
Sufficient technical and facilitator skills
Accessible for participatory activities and support
Necessary resources and time
Serve an instructional role
Motivation and commitment to participate
A tolerance for imperfection
Flexibility (Burke, 1998; Cousins & Earl, 1992; King, 1998)
3/23/10
9
A sampling of interactive methods to gather and discuss evaluation data
Interactive Techniques
What youth do when they get stuck on a project (n=8)
3/23/10
10
Interactive Graphs
Quick data collection Everyone can see the
process and results Can be used as a
starting point for deeper conversations
Can see data by various characteristics
People may be influenced by others
People may be hesitant to place a rating where no one else has
Benefits Limitations
(King, 2009)
Carousel Sheets
Alternative to traditional brainstorming
Lots of information in a short timeframe
Quick way to see patterns Promotes high involvement Involves users in analysis
Participants may influence each others’ responses
Tend to get first responses and gut reaction; not deep and thoughtful
Responses may be too brief
May need to reanalyze some of the data
Benefits Limitations
(King, 2009)
3/23/10
11
Amy Grack Nelson: [email protected]
To see a sampling of evaluation reports visit: www.smm.org/researchandeval
Questions?