Introduction to monitoring and evaluation Ungaluk Program 2015
-
Upload
martin-comeau -
Category
Leadership & Management
-
view
411 -
download
2
Transcript of Introduction to monitoring and evaluation Ungaluk Program 2015
Monitoring =
Keeping track of the implementation and performance of a project.
It requires collecting information on the project activities. E.g. collecting information on performance of participants
in a project throughout the project.
It is important to track if the project is meeting its objectives, and if any changes are needed.
The information collected during the monitoring phase is used for the final evaluation of a project.
Evaluation =
It is where you:
monitor what you are doing
measure what you have done
find out what was effective and what was not
Uses information collected during monitoring (during the project) AND/OR after the project
Can answer questions that need to be answered for good project management.
For example, you might ask: Are we reaching who we wanted to reach or are
we missing the people who most need our project? Do we need more project staff? Do project staff need more training? Is the project resulting in the changes we thought
it would? What aspects of the project should be changed,
expanded, continued, or discontinued?
There are three phases in a project lifecycle:
Phase 1 - Needs assessment
Phase 2 - Project planning
Phase 3 – Implementation
You can do an evaluation at any phase. For Ungaluk, most projects will be evaluated at Phase 3 since they are being funded to be implemented.
Monitoring and evaluation are important for identifying:
Challenges during the project
Impacts and results of the activities
What works and what does not work
Lessons learned and how to improve the project
“Doing” crime prevention is an ongoing process that requires a long-term vision so that there is a long-term impact. We can only know if we have an impact by doing M&E.
If we care about preventing crime and victimization in our communities, it only makes sense to care about what works in reducing crime and victimization. The only way to know for sure what works and what doesn’t work is to invest in evaluation.
Three general types for implemented projects (phase 3):
Process evaluation
Outcome evaluation
Impact evaluation
Looks at the development and implementation of a project.
It shows if you’ve hit measurable targets and implemented strategies as planned.
It’s typically done at the end of the project and it looks at the project from start to finish, assessing cause-and-effect relationships between the project components and outcomes.
This type of evaluation can be very useful in determining whether a project should be continued, expanded upon, or eliminated.
For example, your process evaluation might confirm that 200 people have completed your skills-training program.
Outcome evaluation measures the change that has occurred as a result of a project.
For example, an outcome evaluation would tell you how many of the 200 people that completed your skills-training program actually showed an increased confidence, changed behaviours, found jobs because of the new skills, etc.
Looks at the long-term, deeper changes that have resulted from that project.
This type of evaluation could suggest that there were changes to the lives of the participants that did the skills-training, and this change continued over time and maybe transferred across generations.
1. Identify goals (anticipated outcomes)
2. Describe the project (list project activities)
3. Identify what you want to know (evaluation questions) Determine what information you will need to collect:
To see how your project is doing day to day (on-going monitoring)
To see if you are on track to achieve your results, if you are on time and if you are using resources as planned mid-way through your project (mid-term evaluation), so that you may make changes as needed
To see if the overall changes you were trying to achieve actually happened by the end of the project (final evaluation) and identify what you learned.
4. Identify data sources and data collection tools (See **These are some examples of sources of information – how to collect the information)
5. Collect the information
6. Organize the information
7. Analyze the data
8. Report the results, identify next steps
II. Plan You will be provided with monitoring and evaluation tools, and examples of sources of information on how to collect the information. The evaluation plan includes the information that will
be collected throughout the project and the indicators used to evaluate this information. The project manager will complete a final evaluation report by analyzing the information provided in the plan to present the results of the project.
Please fill out the following information detailing your monitoring and evaluation plan which will be used for the final report. Please use some of the indicators below.
Project managers can use their ‘Work Plan’ in the application form (#10 + #11 in the application form) to fill out Column 1 : “What will be monitored and evaluated? “
Column 1
What will be monitored and evaluated? [What do you want to know about your project?]
List the objectives and specific activities
Column 2
How will it be monitored and evaluated? [How will you know that the project is achieving its objectives and outcomes?]
See the list of example indicators below *
GO TO TABLE p.5-7:
*These are some examples of indicators for all projects and for specific priority areas and objectives:
Column 3
How will the information be collected? List the: 1. Source of information (See below for examples**) 2. Tools used 3. How often the information will be collected
GO TO LIST p.8: **These are some examples of sources of information – how to collect the information:
Column 4
What are the expected results? [What will happen as a result of your project?]
When do you expect to see the results?
What will be monitored and evaluated? [What do you want
to know about your
project?]
List the objectives and specific activities
How will it be monitored and evaluated? [How will you know that the project is achieving its
objectives and
outcomes?]
See the list of example indicators below *
How will the information be collected? List the: 1. Source of information (See below for examples**) 2. Tools used
3. How often the
information will be
collected
What are the expected results? [What will happen as a result of your project?]
When do you expect to
see the results?
Objective 1: To reduce the incidence of dating violence among high school youth.
Objective 2: To increase public awareness and understanding of dating violence as a serious issue.
Indicators:
Number of stakeholders
who attend planning
meetings
Number of
volunteer/in-kind
hours spent on
collaborative efforts
1. Project records
2. Minutes taken at
meetings AND
Time sheets
3. Monthly and
Ongoing
Results:
Increased involvement
of community
stakeholders in
collaborative efforts to
reduce dating violence
When get results:
End of the project
1. We need to know the project's goals, priority area and clientele it is trying to reach.
2. We need to develop a work plan, that shows the relationships between the project’s activities, outputs or results :
Planned Activities
E.g. Recruiting participants, hiring staff, organizing workshops, providing services etc.
Tasks
How, by whom, with what etc.
TimelineExpected results or outputs
E.g. Services or products developed by the activities, outcome for the participants etc.
Identify your project goal and who your clientele. Identify the objectives that will lead to your goal. Determine what the components of your project will be -
that is, your broad strategies or service areas. Describe the project inputs. Who and what will be required
to operate your project? For each component, describe your activities. Who will do
what, and when? Identify the outputs of your activities. How many
participants do you expect? What (and sometimes, how many) tools, materials, or events will be produced?
Identify the outcomes/results linked to these activities. Remember that outcomes represent changes you hope to see result from your activities; they are not just the delivery of the activities themselves. You will want to mention the short-term and intermediate
outcomes of your activities, making sure that these in turn link to your overall goal(s).
3. Define the purpose of your evaluation and the questions you want it to answer
4. Define indicators that will show your project is achieving its expected results/outcomes
5. Identify sources of information for these indicators
6. Determine how you will gather the information
3. Identifying evaluation questions
Determine the goal of the evaluation (not of the project) – This will give you an idea of the questions you will want the evaluation to answer.
Find out what the funder, staff, participants, partners, and others want to know.
Here are some ideas: Was the project implemented as planned?
Did the priority clientele access the project?
Did the project achieve its purpose (expected results)?
Were there unexpected results of the project (positive or negative)?
4. Identifying indicators
What is an "indicator"?
A variable (or information) that measures one aspect of a project.
It indicates whether a project has met a particular goal/objective.
There should be at least 1 indicator for each outcome/expected result.
There are two kinds of indicators:
A process indicator provides evidence that a project activity has taken place as planned.
An outcome indicator provides evidence that a project activity has caused a change or difference in behaviour, attitude, etc.
So, an indicator must be something we expect to change from the time the project begins (known as the baseline) until a later point when the project activities have taken place and are likely to have had an impact.
Indicators should:
provide accurate and reliable evidence
be easy to gather
provide useful information for making management decisions
What will be monitored and evaluated? [What do you want
to know about your
project?]
List the objectives and specific activities
How will it be monitored and evaluated? [How will you know that the project is achieving its
objectives and
outcomes?]
See the list of example indicators below *
How will the information be collected? List the: 1. Source of information (See below for examples**) 2. Tools used
3. How often the
information will be
collected
What are the expected results? [What will happen as a result of your project?]
When do you expect to
see the results?
Objective 1: To reduce the incidence of dating violence among high school youth.
Objective 2: To increase public awareness and understanding of dating violence as a serious issue.
Number of stakeholders
who have "signed on" to
the project
1. Letters of
commitment
AND
Project Plan
AND
Literature
2. Comparison of
project plan to best
practices identified in
the literature
3. End of project
Results:
Enhanced commitment of
key stakeholders to a
comprehensive, evidence-
based plan for addressing
dating violence in the
community
When get results:
End of the project
What will be monitored and evaluated? [What do you want
to know about your
project?]
List the objectives and specific activities
How will it be monitored and evaluated? [How will you know that the project is achieving its
objectives and
outcomes?]
See the list of example indicators below *
How will the information be collected? List the: 1. Source of information (See below for examples**) 2. Tools used
3. How often the
information will be
collected
What are the expected results? [What will happen as a result of your project?]
When do you expect to
see the results?
Objective 2: To increase public awareness and understanding of dating violence as a serious issue.
Level of knowledge
about factors leading to
dating violence among
youth
1. Participants
2. Test of participant
awareness of the
factors leading to
dating violence
3. Before project
sessions start and
again when they are
finished
Results:
Increased participant
awareness about factors
that contribute to teen
dating violence
When get results:
Mid-term of the project
5. Identifying information sources
Once you have identified your indicators, you will need to think about who will provide the information you need. It's best to use a number of sources of information.
Some examples are: Participants – intake form, interviews, focus groups,
observation Public – surveys, questionnaires, community-level
statistics Other agencies – focus groups, key informant interviews,
surveys Project staff – focus groups, key informant interviews,
project records/notes Media – review of media reports
6. Choosing data collection methods
How can you get the information?
Project records/document review
Interviews/focus groups
Surveys/questionnaires
Participant observation
Population level data/statistics
Qualitative data:
It is in the form of words, but may also include any information that is not numerical in form, such as photographs, videos and sound recordings.
Describes a specific context, event, people or relationship in a broad way, by trying to understand the underlying reasons for behaviour, thoughts and feelings.
Examples: observation, one-on-one interviews, focus groups and case studies.
Quantitative data
It is in the form of numbers.
Used to count events or the number of people who represent a particular background.
Examples: surveys, questionnaires and statistical data (such as census information).
In all cases, it's important to have informed consent for any information collected.
If information collection includes photographs or videos of participants, always obtain participants' permission to use the photos/videos in whatever way is anticipated.
When collecting information from or taking pictures of children and youth (under 18 years of age), first obtain permission from their parents.
Some options include:
Continuously
After each event/activity
At regular intervals
Before and after projects
You should try to gather information:
before the project/activity begins (or soon after it begins) and
after it is complete
http://www.makivik.org/monitoring-evaluation-tools/
Select:
“Evaluating Crime Prevention through Social Development Projects: Handbook for Community Groups”
1. Evaluation Planning Tools
2. Evaluating Crime Prevention through Social Development Projects: Handbook for Community Groups
3. Australian Government – Evaluating community crime prevention projects
4. Australian Government – Monitoring and evaluating your project tip sheet
In this resource, the following is provided:
Information on planning a project and the project phases
Information on developing an evaluation plan
Examples of a project plan and an evaluation plan
In this resource, the following is provided:
More details than the previous resource on - Planning a project and the project phases Developing a logic model as a project plan Developing an evaluation plan
It also provides information on - Data collection methods Analyzing data/information How to report the data/information Challenges and solutions to evaluation Other sources of information
In this resource, the following is provided:
Step-by-step guide to making an evaluation plan and evaluating a project
Developing an action plan on what to do with the findings from the evaluation
Managing the evaluation