Evaluating the Tikkun Middot Project Tobin Belzer PhD October 28, 2013
Embed Size (px)
Transcript of Evaluating the Tikkun Middot Project Tobin Belzer PhD October 28, 2013
Repair IJSL Providers - Clinic
Evaluating the Tikkun Middot ProjectTobin Belzer PhDOctober 28, 2013Introduce myself. Ive been engaged by IJS to conduct an evaluation of the Tikkun Middot project.
1This session will provide:
A brief introduction to program evaluationAn overview of the evaluation plan for the Tikkun Middot ProjectAn opportunity to think about what you would like to learnDuring our time together today, Im going to:1. Give a brief introduction to evaluation.How many of you have ever been part of an evaluation? (positive? Negative? Learn something?2. Orient you about the evaluation Im conducting for the IJS, and what you can expect in terms of your participation.3. Help you start to think about what youd like to learn about the Project as you implement it at your organization
What is program evaluation?
Evaluation is NOT about proving the success or failure of a program. Really, its a process of inquiry designed to answer concrete questions. An Empirical study. Data are gathered from real people in real time, systematically. Not ad hoc/anecdotal. Not theoretical research, asking big questions. Its applied: meant to inform decisions. Its not punitive. The practice of evaluation is relatively new:In the 1960s: government was trying to tackle large social issues.Field developed over the yearsIn the 1990s, corporate sector began to look at accountability, efficiency, About cost effectiveness.Its fairly recent in terms of its role in philanthropic center. Since the 2000s, its been used in the Philanthropic sector: questions about measuring impact. Program evaluation to help philanthropy begin to answer those questions. Dollars contributed to create attitudes, behavior, knowledge, skills.Outcomes-based evaluations are increasingly desired and even required by some funders as verification that their support is actually serving their constituents.
Tell a programs storyImprove implementationTest the theory underlying a programDemonstrate accountabilitySupport fundraising effortsInform the field
Evaluation can be used to:
Evaluations can be carried out for a range of reasons.The data collected through evaluation can provide compelling information to help you describe what your program is doing and achieving. Its a way to tell the story of your program and assess whether a program has achieved its intended goals. Evaluation can help understand, verify or increase the impact of your program on participantsEven the best-run programs are not perfect. Every program can improve; By Identifying program strengths and weaknesses, the information collected in an evaluation can provide guidance about changing and improving a program. Evaluation can also inform how a program can grow or be replicated and/or make informed decisions Evaluation can be used to test the theory underlying your program. The systematic data collected about a programs impact and implementation can help to understand whether (and under what conditions) the assumptions underlying a program are accurate, or whether they need to be modified. For example: A large Los Angeles synagogue has a thriving young adult program. Hundreds of people come regularly. The synagogue boards assumption was: attracting young adults to events would lead them to become active synagogue members. A study showed that this was not necessarily case: most active young adults, so far, stayed active in the young adult group. Evaluation helps you demonstrate accountability, Verify that you're doing what you think you're doing. Typically plans about how to implement a program end up changing substantially as those plans are put into place. Evaluations can verify if the program is really running as originally planned and help you show responsible stewardship of funding dollars and show that you are a learning organization that is gathering information, changing and improving.Support fundraising efforts and program growth. Having a clear understanding of your programwhat you did well, and precisely how you accomplished your outcomescan help you raise additional funds to continue your work and expand or replicate your efforts. Nonprofits that have evaluated and refined their programs can share credible results with the broader nonprofit community and inform the field at large. A community that can share results can be more effective.I want to share an example of an large scale evaluation (national, over the course of 2 years), that used evaluation for many of these purposes. The Evaluation of Hillels Senior Jewish Educator and Campus Entrepreneurial Initiative. That program hires educators, who work with student intern,s to build relationships with students who are otherwise unconnected to Jewish life on campus. Ultimately, the program attempts to build Jewish life and to increase students commitment to Jewish learning.Questions:Does this program expand hillels reach? (Is it doing what they think its doing?)Is there a key activity that contributes more to students growth? (What parts of the program are working especially well?)In what ways do educators and interns affect Jewish students lives? (Whats the programs impact on participants?)Who is it reaching? Is it meeting the program goals?
Types of evaluationImplementation (Formative)
The impetus for the evaluation (why an evaluation is being called for at a specific time and place): drives the type of the evaluationProgram evaluation can include any or a variety of at least 35 different types of evaluation, depending on your needs. Evaluations can offer needs assessments, or examine effectiveness, efficiency. They can be formative, summative, goal-based, process-oriented, outcome focused. You get the point. Im going tell you a little about the 2 most common that might be used in your situation: Implementation and Outcomes/Impact based evaluations(Evaluations can be both/more than one type)
5Who and how many participated? What happened?How was your program received? What should be modified?Questions driving an IMPLEMENTATION focused evaluation could include:Implementation Evaluation: Designed to examine activities that are undertaken to achieve program goals. Its used to document the evolution of a program. Simply put: its about how the program is working. It's the nuts and bolts. Which could include participant satisfaction with programs. Its about both logistics and delivery of program content. Sometimes used to determine which program components are more or less critical. Who and how many participated? (Are you reaching the intended audience? What happened? (Is the program unfolding as planned?)How was your program received? (What do people think about what youre doing?)Which program components are most critical? (What should be modified?)
6Is the program having the desired effect? What else is the program accomplishing? Questions driving an OUTCOME focused evaluation could include:Outcomes Evaluation: Helps you learn about what happens for the participants as a result of their participation.
To do this effectively, it helps to have a clear statement of intended outcomes so you can measure against something that is pre-defined. For example, maybe you determined that an intended outcome is to increase a sense of community among participants, or raise their awareness about local Jewish communal opportunities more broadly. Ideally you would define these goals ahead of time, and then using these goals to help shape the types of programs you design. Then you would use evaluation to tell you whether you are achieving your desired outcomes: why, for whom and under what circumstances the goals are met Outcomes are frequently expressed as changes in knowledge, skill, attitudes, behavior, motivation, decisions, , and conditions. In some cases, Exploratory outcomes study is done when the program provide is unsure of the effect.Outcomes evaluation is different from impact evaluation, even though the two terms are often used interchangeably: Impact evaluations take outcomes evaluation a step further. Impact evaluations focus more specifically on which variables contribute to the impact of the intervention by holding other variables constant. We often dont do impact evaluation in the Jewish world because we dont have control groups. Birthright has a control group of folks who applied but didnt get to participate.
8Tikkun Middot ProjectOutcomes Focused EvaluationThis impact evaluation is designed to explore individuals perceptions of the social, psychological, and intellectual impacts of their participation. Those fall into the categories of attitudes, knowledge, behaviors, (and are of course overlapping)This research will explore the extent to which participation:Leads to greater capacity to practice middot and increases the centrality of values in their livesthe extent to which participation:Deepens participants emotional awareness and increases consciousness of reactivity and defensivenessEnriches the quality of interpersonal relationshipsExpands networks that support personal growth Harnesses the transformative potential of Jewish communal participationAnd ultimately Transforms communal culture Some participants may already be practicing mindfulness meditation the evaluation will seek to understand participants perceptions of the impact on their current practice.
METHODS: 1) Interviews2) Surveys3) Participant observation4) Analysis of primary documentsPOPULATIONS: 1) Leaders2) Study Group Members3) Task Force MembersThis impact evaluation is designed to obtain accurate and reliable information from individuals in three populations, namely: 1) Leaders; 2) Study Group Members; and 3) Task Force Participants (with the understanding that these categories are overlapping). Multiple methods including qualitative interviews, participant observation, surveys, and content analysis of relevant documents will be used.Ill be conducting participant observation during each Cohorts three-day training retreat and at select leadership training webinars. (Along for the ride) Ill undertake content analysis of Community Tikkun Middot Project Implementation Plans to understand how you envision and articulate their strategies to enact organizational change. Im going to interview Leaders from each Cohort immediately following the 10-month course of study.
Your self-reported changes in knowledge, awareness, confidence, attitudes and behaviors will be assessed using a retrospective post-then-pre design, in which both before and after information will be collected together. This method avoids response shift bias that can commonly result from pretest overestimation or underestimation.
A semi-longitudinal evaluative approach will be utilized to explore the longer-term impact of participation on Cohort I Leaders and organizations. Ill conduct Follow-up qualitative interviews with Leaders a year after their completion of the 10-month course of study to understand your perceptions of the projects enduring impact on your character, the quality of your relationships, and your community involvement.
Ill also administer a survey of Cohort I Study Group Members and Task Force Participants from each community 6 months following the end of the Community Tikkun Middot Projects to examine the broader impact of the project on each community.
Interview Participant (2 times)
Evaluator?Your role will be as an interview participant, and also as a liaison, providing me obtain the names and contact information of study group members and task force participants. Also, Ill ask you to do an initial email introduction, letting them know to expect to hear from me, and stressing the importance of their participation in the survey. You also have the opportunity to explore the initial outcomes and implementation yourself, with my support. If you so choose, you can survey the study group members and task force participants yourself, using a survey that I will help you construct.
What do you want to know?Im going to have you split up in groups and ask you to brainstorm using this worksheet.
But first, Im going to refresh you memory about the goals you articulated in your proposals:Take a minute, now look at your partners ideas for changes/refinement?
Are they specific enough?How will you know if these goals have been met? (what information from participants would indicate your goals are being met?)
This exercise is to help me understand your interests and priorities, which I will use to create a brief survey that you can use as you like. Dont get tangled up in the wording, just articulate what you want to know.Demographic Information What background information would be helpful to know about study group participants?ImplementationWhat feedback do you want about the course? About the educator?OutcomesWhat types of attitudinal and behavioral outcomes do you hope to see?
Group 1Temple Tikvah New Hyde Park, NY Cong B'nai Israel Northampton MA Group 2Chochmat- A lot Bnai Keshet Montclair, NJ Group 3NYU-hillelJCCJTSGroup 4Kehillah HSTucson Hebrew AcademyGroup 5Cong Beth Israel, Charlottesville, VATemple Beth Sholom Topeka, KansasCongregation Bnai Jacob, Phoenixville