Thinking Strategically:

Applying a Strategic Framework to

Advocacy Activities

Prepared by: Stephen Rabent, Yan Qu, Faisal Hassan, and Shannon Kennedy

Table of Contents

Executive Summary 1

Section One: Developing a Strategic Advocacy Framework 2

Importance and Use of Monitoring and Evaluation 3

The Challenges of Monitoring and Evaluating Advocacy 4

Overview of a Strategic Advocacy Framework

Creating a Theory of Change 5

Components of the Framework 7

The Framework in Action 15

Review 18

Recommendations and Action Items 19

Section Two: Social Impact Evaluation 20

Why Measure Social Impact 21

Introduction to Methodology 23

Frameworks

Longitudinal Survey 25

Key Informant Interviews 28

Focus Group Discussions 31

Recommendations 34

Appendix i

1

Executive Summary

Introduction:

The following report provides a tool, and a way of thinking, for Chintan to engage in the

monitoring and evaluation of its advocacy efforts. We layout a Strategic Advocacy Framework

for Chintan that integrates its programs, activities, and goals, allowing you to see the

interconnection that exists between them. Within the model, monitoring and evaluation

techniques play a key role in allowing you to track your efforts. Perhaps most importantly, it

provides a body of knowledge and evidence for you to draw upon as you make decisions

regarding your advocacy strategies, activities, and goals. We also present a section on the use of

social impact evaluation and techniques to conduct them. However, we would like to make clear

that Chintan is not yet ready to engage in formal social impact evaluation. Once the Strategic

Advocacy Framework is implemented, Chintan will begin to develop the knowledge necessary to

engage in these activities.

Key Recommendations and Action Items:

1) Plan your Advocacy Work Within the Larger Organizational Context

Based on our interactions and discussions with Chintan, your organization would benefit

from periodically taking a step-back from your everyday, essential programmatic and

advocacy work and review how it fits within your broader mission. This exercise can be

beneficial for keeping your organization focused on its mission and reinforcing how its

activities lead to the changes you seek.

Action Item: Chintan should create a theory of change diagram that models how its

various activities, interim outcomes, goals, and mission interact.

2) Become a “Learning Organization”

If Chintan wants to grow and expand its reach while remaining a lean, efficient

organization, you must focus on incorporating the learnings from your activities into

future efforts. Simply identifying the components of the framework and filling in the

details of your work is not enough. Constant review of information, observations, and

lessons learned must be integrated into the fabric of Chintan’s everyday activities.

Action Item: Schedule time during monthly meetings for formal discussion of the

Strategic Advocacy Framework and commit to one quarterly meeting on the topic.

3) Engage in Conversations on Social Impact’s Role at Chintan

Chintan has expressed the desire to engage in efforts to measure its social impact.

However, you are currently not in the position to immediately begin to do so. These

techniques can be time and resource intensive, and require data Chintan does not have.

Action Item: Hold organization-wide discussions on how impact evaluation is, or is not,

an important tool for Chintan to utilize. Keep in mind that impact evaluations can be

done internally, or by external consultant teams.

2

Section One: Developing a Strategic Advocacy Framework

Introduction

Section One of this report provides a Strategic Advocacy Framework that incorporates

monitoring and evaluation tools. Presented within a “theory of change” concept, it provides a

broad overview of the elements of a successful monitoring and evaluation program as well as the

challenges inherent to monitoring and evaluating advocacy efforts. This section specifically

focuses on: (1) How Chintan’s current programmatic work and advocacy efforts can be

incorporated into a broader, organization-wide strategy; (2) The creation of a formal monitoring

and evaluation tool; And (3) how the results can be used to inform policy strategies and move the

organization closer to meeting its overall goals.

Overview

A “Theory of Change” provides the context in which your programs, campaigns, and goals

operate. It models how activities are expected to result in the desired changes to policy and

behavior. Using a strategic approach to your work allows you to see the connections between

your efforts, track your progress, and measure the achievement of your ultimate mission as an

organization. As you read through this report, it is important to note that while it is presented as

a smooth process, the steps involved most often occur simultaneously or even out of order.

Within this Strategic Advocacy Framework, monitoring and evaluation tools provide Chintan

with the information necessary to guide your decisions and measure your progress. The advocacy

framework we propose draws upon best practices from numerous organizations and foundations.

It contains three main components that are necessary to identify and track in order to make full

use of a theory of change:

Goal Creation is the centerpiece of any program or policy campaign. Identifying realistic

and meaningful outcomes sets the context for future organizational efforts.

Interim Outcomes are the intermediate steps necessary to achieve your overall goals and

are useful in tracking progress towards achieving them. Chintan already engages in a

small amount of measuring, however these efforts should be expanded as well as inform

your decisions in relation to the achievement of Chintan’s overall mission.

Activities are the actions you take to measure progress toward reaching goals and

outcomes. They track the programmatic work (inputs) that Chintan does on an everyday

basis, as well as your advocacy efforts. Currently Chintan does engage in activity

tracking (output tracking) and has a system to collect and aggregate this information.

Tying Chintan’s activities to the achievement of interim outcomes is the next in

evaluating progress towards your mission.

3

The Importance and Uses of Monitoring and Evaluation

There is a growing external demand on organizations, particularly for non-governmental

organizations (NGO’s), to monitor and evaluate their programs and activities. Additionally, these

techniques can be extremely beneficial to Chintan internally. This technique can be beneficial

through:1

1. Tracking campaign efforts while allowing for comparison across activities and

programs.

2. Communicating to partners and stakeholders the resources required to achieve your

desired outcomes.

3. Creating an understanding of what activities have been most effective, what

circumstances and actions may have contributed to their success, and adapting your

approaches to focus on these efforts.

4. Communicating to stakeholders what you have learned about your activities

through more detailed quarterly and annual reports.

5. Tracking adjustments to your strategy, actions, and desired outcomes over time.

6. Assessing progress versus only focusing on the achievement of policy impacts.

The importance of a robust monitoring and evaluation structure is thus two-fold, informing both

external and internal actors. This information can lead to greater accountability towards donors

who help to fund Chintan’s work through its incorporation into quarterly and annual reports.

Furthermore, it will allow you to better understand whether you have made progress towards

accomplishing long-term policy or social change goals. Monitoring and evaluation as an

assessment tool helps to ensure that Chintan’s resources are being used effectively while having

the desired impact.

However, monitoring and evaluation techniques do not exist in a vacuum and alone they cannot

reach their full potential in informing Chintan’s decision making. We recommend that to

maximize its effect on the organization, these techniques are incorporated into a broader vision

of how Chintan creates policy change and meets its mission. Moreover, developing this broad

framework is often the starting point in designing a monitoring and evaluation scheme.2 It

provides the context and impetus behind decisions (the “10,000 meter high” view of the

organization’s actions) on how the monitoring and evaluation system is implemented (the “on

the ground” details of the organizations operations).This system provides key insight into how

Chintan approaches its work and the interconnections that exist between its programmatic

efforts, advocacy strategies, and ultimate policy goals.

1 “Advocacy Evaluation Mini-Toolkit: Tips and Tools for Busy Organizations,” 3. 2 “A Guide to Measuring Advocacy and Policy,” 11.

4

The Challenges of Monitoring and Evaluating Advocacy

Advocacy efforts are often difficult to measure and there is not one traditional format for doing

so. While the principals of designing monitoring and evaluation tools and the purpose of them

remain nearly identical between advocacy and service evaluation, certain challenges are unique

to their application and must be confronted before the process is undertaken.3

Challenge One: Constantly Shifting Time Frames, Strategies, and Milestones

Achieving policy goals can take years of work. Oftentimes, progress is made in fits and starts,

creating periods of high activity followed by lulls. Moreover, strategies to achieve outcomes can

shift substantially over these long times horizons in response to changes in the social and

political context surrounding an issue area. As such, milestones and interim targets for the policy

campaign will shift as well.4 It is critical that monitoring and evaluation efforts also adjust to

ensure that the data and metrics collected are still useful for informing the policy advocacy

process.

Challenge Two: Contribution versus Attribution

The role of external events and concurrent factors that occur throughout these long time horizons

makes the attribution of one specific activity towards the achievement of an outcome very

difficult. The political and social environment that Chintan exists within is incredibly

complicated and crowded with actors. As a result, linking efforts to direct effects can be difficult

due to the presence of a vast amount of confounding variables. In most cases, efforts can only be

argued to have contributed to bringing about an interim outcome or goal, rather than directly

attributable to an effect.

Challenge Three: Extreme Time and Resource Constraints

Chintan is a highly successful and continuously busy organization with few excess staff and

resources to devote to monitoring and evaluation. As such, thorough internal monitoring and

evaluation techniques or contracting for external evaluation services for each program and policy

are most likely beyond the everyday capabilities or the organization. Instead, simplified

techniques for a select portfolio of activities and data must be developed to ensure that the

outcomes of the process can inform your work.

Despite these challenges, engaging in monitoring and evaluation is central to your continued and

future success. We have developed a strategic advocacy framework with these challenges and

your organizational capacity in mind, allowing you to implement and advance your work in an

effective manner.

3 Adapted from “Monitoring and Evaluating Advocacy: Companion to the Advocacy Toolkit,” and “A Guide to Measuring

Advocacy and Policy.” 4 “A Guide to Measuring Policy and Advocacy,” 8.

5

Overview of the Proposed Framework

We recommend that Chintan adopt a Strategic Advocacy Framework that incorporates

monitoring and evaluation mechanisms that will aid in the strategic development and

management of programs intended to achieve policy goals. The monitoring and evaluation tool

will help you understand if your activities are having an impact on reaching your interim

outcomes and ultimate goals. These lessons learned and information can be used by Chintan to

better manage its efforts as well as build a case for incorporating your desired programs into the

meaningful policy.

Creating a “Theory of Change”

A “Theory of Change” is a conceptual model utilized in the mapping of a vision for achieving

your policy goal. It links together the activities, interim outcomes, and goals that support the

achievement of the broader mission of Chintan. It is an articulation of what changes Chintan

wants to see and how your work will contribute to these changes.5 This conceptual model can be

most easily expressed in a visual diagram, mapping how your organization translates its activities

into goal achievement.6 We have provided an example of this model applied to the E-Waste

campaign in the Appendix.

Developing a theory of change is intended to be a collaborative process, where key players

within the organization can come to an agreement about the desired impact of their work and

collectively create a clear picture of what success looks like. An important component of this

process is incorporating both interim outcomes and goals into mapping the theory of change.

This process can be instrumental in building an understanding of goals, setting expectations, and

evaluating success.7

Discussions around the How, What, and Why of Chintan’s policy and advocacy work can be

useful in framing discussions about a theory of change. The development of activities, interim

outcomes, and goals can clarify How Chintan’s work contributes to What policy and advocacy

goals you have. Chintan’s mission and the ultimate impact it seeks to achieve help to frame Why

it engages in the preceding activities. Discussing these questions are essential for internally

designing an evaluation and monitoring program:8

1. What activities and interim outcomes should be measured? What information is most

important for you to know?

2. How often, and for how long, should you collect this data?

3. How rigorous should your process be and how much of your resources should you

devote to collecting and analyzing our data? Will you be collecting quantitative or

qualitative data?

4. How will your results be used?

5 “Developing Monitoring and Evaluation Framework for Budget Work Projects: A How-To Guide For Practitioners,” 13. 6 “A Guide to Measuring Advocacy and Policy,” 11. 7 “A Guide to Measuring Advocacy and Policy,” 15-16. 8 “A Guide to Measuring Advocacy and Policy,” 16.

6

In addition to answering these questions about future action, you can also place your current

programs and policy advocacy campaigns within a theory of change framework to better

understand the linkages between activities and outcomes.

In order to work within this framework, you must refine your thinking on how Chintan creates

and manages its projects and advocacy strategies. Instead of an ad-hoc approach to your work,

focusing on responding to events and new information, Chintan can begin to be more strategic

with its activities and aim to shape policy. Rather than focusing on questions of ‘what are we

doing’, the organization should begin to ask ‘what have we and can we achieve’. In this model,

you will gain a clear sense of the purpose and expected results of your programs or policy

strategies. Then you can modify activities and approaches where necessary to ensure desired

results.9

A theory of change allows you to view your programs and advocacy efforts from a high level

and reveals the causal relationships that exist among your activities. It is a management tool,

steering programs and efforts towards the achievement of Chintan’s central mission.10 Once

established, Chintan can delve more deeply into its many components, of which monitoring and

evaluation plays a crucial role. Exhibits One and Two in the Appendix provide an example of

how Chintan may model a theory of change.

9 “Developing Monitoring and Evaluation Framework for Budget Work Projects: A How-To Guide For Practitioners,” 20-21. 10 “Developing Monitoring and Evaluation Framework for Budget Work Projects: A How-To Guide For Practitioners,” 28.

7

Components of the Monitoring and Evaluation Framework:

Within the theory of change, three central elements are present that connect your programmatic

and advocacy efforts to the mission Chintan is attempting to achieve: 1. Goals, 2. Interim

Outcomes, and 3. Activities.

1. Goals

It is important to develop a clear goal for a project or advocacy campaign. This identified main

objective can be accomplished through a set of policy changes that often take years of work and

strategic planning. When discussing goals for a program and policy advocacy strategy, particular

attention should be paid to how they fit with the overall mission of Chintan and how they

contribute toward it. For example, Chintan’s policy advocacy on e-waste management directly

relates to its mission of achieving a sustainable municipal waste system that enhances the

livelihoods of those working in the sector.

Goals should be developed both at the programmatic as well as the advocacy strategy level to

guide Chintan’s decisions.11 It can be useful to define goals as the inverse of the problem that

being addressed. For example, if the core problem is ‘waste segregation is not occurring in

residential households’, the goal could be ‘increasing the percentage of residential households

segregating waste’. The following five attributes are often presented as hallmarks of a well-

defined goal:12

Specific

Measureable

Attainable

Realistic

Time-bound

Once they are clearly defined, Chintan can begin to unpack each goal to reveal the underlying

components that are necessary for their achievement.

Below is a sample of how Chintan could have used the strategic framework in its E-Waste

advocacy campaign.

Expected Results

Which indicators

will be used to

measure success?

What is the

current status of

the indicator?

How far do you

want the indicator

to move?

How will

indicator data be

collected and who

is responsible? Goal: New

National Policy on

E-Waste

Management

Revision of draft E-

Waste rules

The government has

drafted E-Waste

regulations that do

not address the

informal sector.

The revision of a

National Policy on

E-Waste that

includes the

informal sector

Policy Tracking of

the issue by the

Advocacy Staff.

11 “A Guide to Measuring Advocacy and Policy,” 17. 12 “Developing Monitoring and Evaluation Framework for Budget Work Projects: A How-To Guide For Practitioners,” 13.

8

One possible goal for the E-Waste campaign may have been the development of a new national

policy on E-Waste management. The campaign could be considered a success if the draft E-

Waste regulations that were issued by the government were revised. The current status of the

policy can refer to recent activity surrounding the issue area. The next box details the extent to

which you want the current status to shift, in this case the revision of a draft policy. The final box

ensures that someone is assigned to collect the necessary information.

2. Interim Outcomes

Interim outcomes act as the bridge between achieving goals and the activities undertaken to

contribute to their achievement. They are important to illustrate progress towards goals, which

may be long-term or difficult to achieve. They are measured through changes that happen within

a specific target audience and are often the direct results of activities and tactics taken to move

toward the policy goal and often must be achieved in order to reach that goal. Measuring

interim outcomes will allows you to:13

Examine if there is evidence of progress towards your ultimate policy goal.

Assess which milestones you are reaching and which you are not, allowing you to adjust

your activities accordingly.

Determine which efforts are working best and how to better allocate resources to those

that have the most impact.

Examine if the interim outcomes you are achieving demonstrate success, even if the

policy goal has yet to be achieved.

In advocating for policy changes it is particularly important to identify and track progress

towards the achievement of interim outcomes. Even with policy change as your end goal, other

more immediate outcomes can be just as important for tracking Chintan’s overall organizational

strength, its impact on societal conditions, and can help build momentum towards policy change.

Moreover, by assessing a range of interim outcomes through monitoring and evaluation,

Chintan’s efforts can be viewed in light of their contribution to multiple impact categories even

if the policy goal is not achieved.14

Framing interim outcomes as the necessary conditions to create a policy or behavioral change

may be more conducive to discussing what to track in a given effort. Organizational Research

Services distilled six categories of outcomes, five of which we present as a strong basis for you

to begin to think about and identify interim outcomes:15

13 “Advocacy Evaluation Mini-Toolkit: Tips and Tools for Busy Organizations,” 5. 14 “Advocacy and Policy Change Evaluation: A Brief Overview,” 2. 15 “A Guide to Measuring Advocacy and Policy,” Harvard Research Project, 22.

9

Tracking progress towards interim outcomes can be both a formal and informal exercise in data

gathering. Formal measures of tracking progress in interim outcomes include focus groups,

polling, media tracking, surveys, and interviews. Informal methods of measuring progress

towards interim outcomes can also be very useful in providing information on the impact of your

effort. Integrating data collection into organizational routines is a simplified and introspective

way to assess progress. This approach may include:16

Discussion of progress during program and staff meetings,

Written record of impressions after engaging in activities,

Reflection sessions on progress.

.

Chintan currently only engages in a basic level of interim outcomes tracking and evaluation, and

solely does so at the programmatic level. The types of information already evaluated that fall

under the category of interim outcomes include:

The number of children “mainstreamed” into government schools whom no longer work

as wastepickers.

Improvement in numeracy and literacy levels in children attending Learning Centers

over time.

Public awareness and satisfaction with Chintan’s work in coordinating household waste

collection from waste-pickers.

16 A full list of indicators can be viewed in Figure 3 in the Appendix.

Example Interim Outcome Categories

1. Shift in Social Norms: How Chintan’s advocacy and policy goals are aligned with

social values and behaviors in society.

2. Strengthened Organizational Capacity: How well Chintan’s organizational structures,

staffing, leadership, and strategic planning are aiding in its efforts.

3. Strengthened Alliances: If Chintan’s alliances and partners are aiding momentum

towards its goals through coordination, collaboration, and mission alignment.

4. Strengthened Base of Support: The level of support for policy changes across the

general public, interest groups, opinion leaders, and politicians.

5. Improved Policies: Chintan’s ability to articulate evidenced based policies and ability

to help policies progress through the policy process (policy formally proposed, levels

of support, adoption, etc.).

10

Chintan should similarly begin establishing interim outcomes related to its advocacy campaigns

to assess its efforts and impact. Furthermore, it must address these topics on a regular basis,

incorporating the information gathered from monitoring and tracking into its decision-making

process. Expanding the organization’s focus to include metrics on its advocacy work will be

instrumental in tying Chintan’s programmatic activities to the advocacy work it engages in while

in pursuit of its mission.

Below is an example of how Chintan could have used the strategic framework to identify interim

outcomes in its E-Waste campaign:

Expected Results

Which indicators

will be used to

measure success?

What is the

current status of

the indicator?

How far do you

want the indicator

to move?

How will

indicator data be

collected and who

is responsible? Interim Outcome:

Increased Public

Awareness of E-

Waste Recycling

Positive responses

among survey

respondents.

Unknown. Need to

collect baseline

data.

60% of target

respondents convey

information related

to Chintan’s E-

Waste messaging.

Survey of target

audience by

Advocacy Staff and

volunteers.

Interim Outcome:

Increased Media

Coverage of E-

Waste Regulation

Number of stories

on E-Waste that

contain Chintan’s

messaging on E-

Waste management.

To be obtained

through counts of

media coverage on

the topic from 3 top

media outlets over

previous year.

50% increase of

media coverage of

Chintan’s desired

policies over the

next year.

Media tracking by

Advocacy Staff and

volunteers.

Interim Outcome:

Build political will

for a revision of

draft E-Waste

rules

Number of elected

officials who

publicly support the

inclusion of the

informal sector in

revised E-Waste

regulations.

Unknown. To be

obtained through

preliminary

conversations with

politicians.

5 political

supporters

throughout the

government.

Tracking of

interactions with

and actions taken by

politicians by

Advocacy Staff.

Three interim outcomes would be important to track to evaluate progress towards the goal of a

new national policy on E-Waste management. Several indicators were identified during a

brainstorming session, which were then narrowed down to one key indicator for each, which

could be used to evaluate its progress. At the beginning of a new campaign, the status of an

indicator may be unknown, particularly if it is a new issue for Chintan or the campaign’s target

audience. Chintan must then rely on its own institutional knowledge and past campaign

experience to set realistic expectations for how far they believe it is possible to move the metric

and make progress on their interim outcomes. Once again, it is important to identify who will be

responsible for tracking the interim outcome, and where the data will come from.

11

3. Activities

Activities consist of what is actually being done to achieve both interim outcomes and goals.

Inputs, such as the target of your advocacy effort, the financial resources available, and staff,

shape the direction of these activities. Outputs, or measures of effort, are the results of activities

and quantify what or how much is produced from a given activity.17 This is the very bottom level

of the theory of change and therefore is critical information to collect and analyze. It is at this

level in which Monitoring mainly takes place, where the work that Chintan does is constantly

reviewed to ensure it is being carried out according to plan.18

Activity tracking can take on a variety of forms and is highly specific to the program or

advocacy effort Chintan is interested in monitoring and evaluating. A complete list of possible

activities to track can be found in Exhibit 3 in the Appendix, but some of these include tracking:

Public information distribution

Policymaker education

Network building

Due to organizational constrains such as time and resource availability, it is crucial that Chintan

is able to identify and prioritize the outcomes that are most meaningful to revealing progress

towards achieving your interim outcomes and policy goals. In helping to guide your thinking in

this respect, we offer the following questions you can consider as you prioritize the resources

allocated to monitoring and evaluating a given program or advocacy campaign:19

17 “Monitoring and Evaluating Advocacy: A Companion to the Advocacy Toolkit,” 13. 18 “Developing Monitoring and Evaluation Framework for Budget Work Projects: A How-To Guide For Practitioners,” 57. 19 Adapted from “Monitoring and Evaluating Advocacy: A Companion to the Advocacy Toolkit,” (19-20) and “Advocacy

Evaluation Mini-Toolkit: Tips and Tools for Busy Organizations,” (1-2).

12

Chintan already has a simple, yet robust, system for tracking many of its programmatic activities

and their outputs. The organization delegates the collection of its data to Field Officers who

input information to pre-formatted excel sheets. This system conforms to accepted best practices,

as those doing the monitoring are individuals who are directly involved in their implementation

and are in the best position to gauge whether they are on track.20 The tracking sheets are

submitted to a central location where they are used to inform quarterly donor reports. Currently,

this information is available for output metrics such as:

The amount, in kg/day, of different wastes and recyclables collected by

wastepickers on a given contract.

The number of children attending its Learning Centers.

The number of trainings and workshops held.

Nearly all of the metrics that Chintan tracks lie on the programmatic side of their efforts, and

relate to outputs (activity tracking) without connection to any interim outcomes or policy goals.

Furthermore, there is a lack of monitoring their advocacy activities. By expanding the

foundation already in place, Chintan will be able to more effectively use the information it

collects, and have a broader insight into assessing its progress towards its interim outcomes

their effect on moving policy.

20 “Developing Monitoring and Evaluation Framework for Budget Work Projects: A How-To Guide For

Practitioners,” 57.

Questions to Identify Activities to Monitor

1. What will users of the monitoring and evaluation want to know? Are some metrics

more important than others? Is there a new strategy that you are particularly curious

about whether it is effective?

2. Does Chintan have a unique contribution to the interim outcome or policy goal? Due

to the prevalence of external and contextual factors in long-term policy advocacy

strategies, it may be more appropriate to focus on outcomes that are more directly

connected with the advocacy effort.

3. Who will conduct the monitoring? It is important to consider who within the

organization, or who external to Chintan, will conduct the evaluation. This will be

determined by both the resources available, and the interim outcomes and policy

strategies you are pursuing, as some are better suited to informal tracking and others

to the expertise and perspective of outside evaluators.

4. What will we do with the information once we collect it? Does it have a clear

connection to informing progress towards interim outcomes or goals? Will it serve

another purpose?

13

Below is an example of how Chintan could have used the strategic framework to identify

activities that would contribute to progress in your interim outcomes in its E-Waste campaign:

What must

be done to

achieve

interim

outcomes?

What interim

outcome will

success on the

activity

contribute to?

Which indicators will

monitor progress?

Where is

the

output

now?

How far do you

want to move the

output?

How will

data be

collected

and who is

responsible? Activity:

Public

Information

Campaign

Increased

Public

Awareness

of E-Waste

Recycling.

Increased

Media

Coverage of

E-Waste

Regulations.

Number of

Facebook posts.

Number of houses

face-to-face contact

and information

distributed at.

Number of Earned

Media mentions of

Chintan’s desired

E-waste policy.

Started at

zero. No

baseline

informatio

n.

3 Facebook

posts per week

on E-Waste.

30% of

households

contacted given

information.

3 earned media

mentions per

month.

The

Advocacy

Staff, with

the help of

interns, will

monitor and

track the

public

information

campaign.

Activity:

Policymaker

Education

Build political

will for a

revision of draft

E-Waste rules.

Number of face-to-

face meetings with

politicians

discussing E-Waste

Regulations.

Distribution of

report on E-Waste

and the informal

sector.

Started at

zero.

Campaign

has not

launched.

Contact 10

politicians and

secure face-to-

face meetings

with 7.

Distribute

report to

politicians

partner and

organizations;

press release on

website and

social media

accounts.

The

Advocacy

Staff will

identify

politicians

and track

interactions.

Activity:

Network

Building

Build

political

will for a

revision of

draft E-

Waste rules.

Increased

Public

Awareness

of E-Waste

Recycling.

Number of

organizations

identified and

contacted.

Types of

constituencies

represented.

Number of

meetings among

members.

Currently

we have

one

partner,

Safai

Sena, on

our E-

Waste

campaign.

Identify and

contact 8

organizations

working in this

issue area.

Meet in-person

with 5

organizations.

Interns and

volunteers

will identify

possible

organizations,

the Advocacy

Staff will

track contact

with

organizations.

Three activity categories were identified as the most important to monitor while assessing their

achievement of interim outcomes. At the beginning of a new campaign, many of these indicators

may not have any baseline information. Once again, Chintan should rely on its own experiences

with previous campaigns, and seek out best practices from other organizations, to create

achievable targets in each indicator area. As for the other two components, it is essential to

identify who will collect this information and where it will come from.

14

Feedback Loops

All the components of the system interact in an iterative process, constantly informing decisions.

Measuring digital outreach and social media activities, and their contribution to interim

outcomes such as issue visibility, can play a direct role in shaping the advocacy strategy that

Chintan pursues for a particular policy goal. The feedback mechanism works in the opposite

direction as well, with information gathered throughout the evaluation of advocacy efforts used

to guide the programmatic work of Chintan. Ultimately, the information exchange should occur

seamlessly and provide invaluable information and context to how Chintan is approaching and

achieving its mission.

15

The Framework in Action

This strategic advocacy strategy will ultimately be tailored by Chintan to fit its needs. However,

the framework will follow the same process no matter how it is adapted by Chintan. It is

necessary to build the framework within the theory of change in order to maximize its use in

tying together the many activities Chintan engages in to its ultimate mission. Once established,

Chintan can delve into the three component parts and begin creating an exhaustive list of

possibilities for each for each in relation to its programs and policy advocacy campaigns. The

following matrix provides a complete view of the strategic advocacy plan for the E-Waste

campaign, which has been presented throughout this section:21

Strategic Results Results Indicators Baseline Targets Means of

Verification

Expected Results

Which indicators

will be used to

measure success?

What is the

current status of

the indicator?

How far do you

want the

indicator to

move?

How will

indicator data be

collected and

who is

responsible? Goal: New

National Policy on

E-Waste

Management

Revision of draft E-

Waste rules.

The government has

drafted E-Waste

regulations that do

not address the

informal sector.

The revision of a

National Policy on

E-Waste that

includes the

informal sector.

Policy Tracking of

the issue by the

Advocacy Staff.

Interim Outcome:

Increased Public

Awareness of E-

Waste Recycling

Positive responses

among survey

respondents.

Unknown. Need to

collect baseline

data.

60% of target

respondents convey

information related

to Chintan’s E-

Waste messaging.

Survey of target

audience by

Advocacy Staff and

volunteers.

Interim Outcome:

Increased Media

Coverage of E-

Waste Regulation

Number of stories

on E-Waste that

contain Chintan’s

messaging on E-

Waste management.

To be obtained

through counts of

media coverage on

the topic from 3 top

media outlets over

previous year.

50% increase of

media coverage of

Chintan’s desired

policies over the

next year.

Media tracking by

Advocacy Staff and

volunteers.

Interim Outcome:

Build political will

for a revision of

draft E-Waste

rules

Number of elected

officials who

publicly support the

inclusion of the

informal sector in

revised E-Waste

regulations.

Unknown. To be

obtained through

preliminary

conversations with

politicians.

5 political

supporters

throughout the

government.

Tracking of

interactions with

and actions taken by

politicians by

Advocacy Staff.

21 Adapted from “Monitoring and Evaluating Advocacy: A Companion to the Advocacy Toolkit.” and “Developing

Monitoring and Evaluation Framework for Budget Work Projects: A How-To Guide For Practitioners.”

16

Course of Action Results Indicators Baseline Targets Means of

Verification

What must

be done to

achieve

interim

outcomes

What interim

outcome will

success on the

activity

contribute to?

Which

indicators will

monitor

progress?

Where is the

output now?

How far do you

want to move

the output?

How will

data be

collected and

who is

responsible? Activity:

Public

Information

Campaign

Increased

Public

Awareness

of E-Waste

Recycling.

Increased

Media

Coverage of

E-Waste

Regulations.

Number of

Facebook

posts.

Number of

houses face-

to-face

contact and

information

distributed at.

Number of

Earned Media

mentions of

Chintan’s

desired E-

waste policy

Started at zero.

No baseline

information

3 Facebook

posts per

week on E-

Waste.

30% of

households

contacted

given

information.

3 earned

media

mentions per

month.

The Advocacy

Staff, with the

help of interns,

will monitor

and track the

public

information

campaign.

Activity:

Policymaker

Education

Build

political

will for a

revision of

draft E-

Waste rules.

Number of

face-to-face

meetings with

politicians

discussing E-

Waste

Regulations.

Distribution

of report on

E-Waste and

the informal

sector.

Started at zero.

Campaign has

not launched

Contact 10

politicians

and secure

face-to-face

meetings with

7.

Distribute

report to

politicians

partner and

organizations;

press release

on website

and social

media

accounts.

The Advocacy

Staff will

identify

politicians and

track

interactions.

Activity:

Network

Building

Build

political

will for a

revision of

draft E-

Waste rules.

Increased

Public

Awareness

of E-Waste

Recycling.

Number of

organizations

identified and

contacted.

Types of

constituencies

represented.

Number of

meetings

among

members.

Currently we

have one

partner, Safai

Sena, on our

E-Waste

campaign.

Identify and

contact 8

organizations

working in

this issue

area.

Meet in-

person with 5

organizations.

Interns and

volunteers will

identify

possible

organizations,

the Advocacy

Staff will track

contact with

organizations.

17

Five steps can be applied to each component of the framework to build them out and tie them

together:22

Chintan works in a burgeoning issue area with many actors influencing the policy arena. As

such, it is particularly susceptible to external influences and events shaping its work. Consistent

review of programs, activities, and interim outcomes will help Chintan ensure its efforts are

responding to the external environment and having their intended impact.

The process of developing a strategic monitoring and evaluation framework for your programs or

your advocacy efforts should be a collaborative endeavor. Ideally, these discussions should take

place on a regular basis, checking in on progress and including new programs, advocacy

strategies, and campaigns into your current framework.

22 Adapted from “Monitoring and Evaluating Advocacy: A Companion to the Advocacy Toolkit.” and “Developing Monitoring

and Evaluation Framework for Budget Work Projects: A How-To Guide For Practitioners.”

Five Steps to Filling in the Framework

1. Results: A description of the component (goal, interim outcome, or activity). These

should already be defined while developing a theory of change.

2. Indicators: How the result will be measured and what success looks like.

3. Baseline: Where the indicator or output currently is at.

4. Targets: How far Chintan would like to see the indicator or output move from the

baseline state (what is success for the factor?).

5. Means of Verification: How the data will be collected and who is responsible.

18

Review

After Chintan develops a theory of change, formalizes its components, and begins the monitoring

and evaluation process, it must ensure the knowledge gleaned from the exercise is used and

informs its decision-making process. While the proposed framework aims to simplify as much as

possible the elements involved in this strategic thinking process, it inevitably requires time and

resources. Using a framework helps focus your attention on the key aspects of your programs

and advocacy efforts that are most relevant to achieving interim outcomes and organization-wide

goals. This review can occur in the following stages: 23

1. Informally at monthly meetings among program managers. Chintan can build on its

current discussions by incorporating more explicit language on its activities and progress

towards interim outcomes, drawing observations and data from its monitoring and

evaluation activities.

2. Formally at quarterly progress meetings. It may be useful for Chintan to devote an entire

meeting, once per quarter, to discuss its progress and strategies that may or may not be

working.

3. Annual Review Session. An annual review session can be very useful for Chintan to take

a step back and view its activities and efforts from a broad level. In this space, the

organization can revisit its theory of change and examine how its components have

changed throughout the year. It may be necessary to update interim outcomes, revise

activities, and update it with new programs and advocacy efforts. This review will help

Chintan ensure its approaches and efforts are effective and efficient in reaching its overall

mission.

23 “Developing Monitoring and Evaluation Framework for Budget Work Projects: A How-To Guide For Practitioners,” 73-75.

19

Recommendations:

Chintan has many of the necessary elements of a monitoring and evaluation programs to

implement a strategic advocacy framework. However, it has not engaged in efforts specific to its

advocacy campaigns. We present the following recommendations for your consideration as you

work to develop a formal strategic advocacy framework:

1. Place Your Advocacy Work in a Larger Context

Chintan should work to create a theory of change diagram that models how its various

activities, interim outcomes, goals, and mission interact. This exercise will be

instrumental in helping you begin the process of creating a strategic advocacy framework

and incorporating monitoring and evaluation techniques to track your progress.

2. Develop Interim Outcomes

Chintan does not engage in any formal development of interim outcomes. This

component of the strategic advocacy framework is crucial to tying your activities to your

goals, and ultimately to Chintan’s mission. We recommend that Chintan begin to

formally establish interim outcome markers for its advocacy goals.

3. Become a Learning Organization

Even if fully implemented, the strategic advocacy framework that we propose will end up

useless without continuous review by Chintan. Information and data that you will collect

throughout the process should be reviewed and discussed at regular progress meetings,

and informally among staff. Working to develop a culture of self-learning will engrain

the belief in the value of this information and the necessity of its incorporation in the

decision-making process.

Actions:

To implement the recommendations, we suggest Chintan take the following actions:

Immediate Actions:

o Each program should meet to discuss how the framework could be applied to their

program.

o Program Managers meet to discuss the results of their meetings with each other.

Medium-Term Actions:

o A small number of Interim Outcomes are created on a trial basis in each program

and are monitored and tracked internally.

o Chintan begins holding a quarterly meeting where the results of the Interim

Outcome tracking is discussed. Challenges and successes are highlighted and

areas of cross-organization collaboration are identified.

20

Section Two: Social Impact Measurement

Introduction

This section provides a brief introduction on how to undertake a social impact assessment of

Chintan’s ongoing program and advocacy campaign and gives detailed instruction on data

collection methods. These methods work within the strategic advocacy framework to develop an

information baseline, allowing you to perform comprehensive and in-depth social impact

evaluation in the future. We also provide detailed data collection samples based on Chintan’s

current programs to show how these methods are operated and applied in actual research. We do

caution that a thorough impact evaluation is a time-consuming and resource-intensive process.

The report outlines both qualitative and quantitative data collection and analysis techniques that

can be useful in measuring the impact of your programmatic work. However, even in their

simplified form, they can be both time and resource intensive. Moreover they rely on baseline

data collection that does not yet exist. Once Chintan develops a sound baseline and

longitudinal data through robust monitoring and evaluation, you may be able to engage in

more in-depth program impact measurement activities.

What is Social Impact?

Social impact is the consequence of certain public or private actions on particular individuals,

groups or overall society. Social impact measurement tries to identify and materialize the change

brought about by public actions, such as programs, projects or certain policy implementation.

Based on Chintan’s mission and current programs, we have identified five major social impact

categories associated with Chintan’s objectivity and ongoing programs:

1. Cultural impacts: The impact on social norms, customs, awareness and public opinion.

Example: Measuring the change of people’s knowledge of the deleterious effect of E-

waste on individual health and environment.

2. Socio-economic impacts: The impact on socio-economic indicators, such as individual

income, employment, tax and investment.

Example: Measuring the improvement of the income levels of wastepickers after

formalizing their job.

3. Community impact: The impact on infrastructure, services, volunteer organizations and

activity networks.

Example: Measuring the increased number of sorting trash bins in different communities.

4. Health impact: The impact on mental, physical and social well-being.

Example: Measuring improved health conditions of children within wastepicker

communities.

5. Life style impact: The impact on people’s living habits, behavior and interaction with

other people.

21

Example: Changes in people’s purchase and use of electronic products that contain less

hazardous components.24

Why Measure Social Impact?

As Chintan develops and expands as an organization, measuring social impact will play a crucial

role in evaluating interim outcomes that link Chintan’s program activities with their ultimate

mission. It tracks the direct implementation impacts from important programs on social

population groups to show the incremental progress toward the overall change. Identifying,

measuring and communicating social impact enables Chintan to track their achievements and

make improvements or adjustments on current strategies and programs.

In addition, this measurement will allow you to:

Identify and set agendas based on social issues associated with the program.

Enhance sustainable, positive effects on beneficiaries (citizens, communities).

Build up a trust relationship between Chintan, the municipal government and community

residents that is beneficial for future program implementation.25

Social impact Identification

Before conducting any data collection and analysis work, it is always important to identify what

social change Chintan primarily wants to enable. This could be long-term change due to a

specific policy adoption or an intermediate outcome through certain activities. We recommend

Chintan identify the key impact based on the scope of program and policy as the first step to

establishing an effective assessment design. This can most easily be identified through

exercising the theory of change model. While thinking about your potential impact, it is useful

to ask questions such as these:26

What is the ultimate impact and intermediate outcomes that you would like to see for

citizens, government officials and city environment?

How large is the scope of your influence based on organizational scale and capacity?

Are there any potential unexpected outcomes of our programs, either positive or negative,

you should consider?

Tracing Impact

There are many confounding factors that may also lead to an observed outcome after policy

implementation. In order to establish a more convincing causal relationship between program

implementation and the social outcome, you could establish a counter-factual: a comparison

group that is very similar to the observed people or community, to test what would happen if

24 Adapted from “A Comprehensive Guide for Social Impact Assessment,” 5. 25 “A Comprehensive Guide for Social Impact Assessment,” 8. 26 “Social Impact Assessment,” 1.

22

beneficiaries had not experienced the program or policy. Due to time and resources constraints,

we do not recommend Chintan adopt this method. However, it is still important for you to

“scope” your impact, trying to identify the most obvious impact that could be directly associated

with the implementation of program.

The diagram below displays an impact value chain that allows you to better understand the

relationship between program activities, impact and the expected goals and where the role of

social impact evaluation fits in. 27

To review, Inputs are the resources necessary for the operation of program, such as money,

human capital and time. These are directed towards the major activity conducted in the program,

such as trainings, meetings, and grassroots events. Outputs are the measure results from the

organization’s activities, e.g. the weight of waste that has been collected, the number of students

that have been admitted to learning center, the number people that signed the pledge etc.

Outcome are the actual change that occurred in the social system. In a formal program

evaluation, the overall change compared to the changes in a counter-factual group determines

how large the social impact is. Finally, goal alignment involves the evaluation of the difference

between the expected outcomes and what impact has actually been achieved. All together, these

measures provide insight into the refinements of future program implementation.

27 “Double bottom line project report: assessing social impact in double bottom line ventures.”

Input Activity Output OutcomeGoal

Alignment

23

Introduction to Methodology

Data Collection Methods

There are both quantitative and qualitative methodologies that could be applied to meet different

measurement needs. Quantitative methodologies, such as systematic surveys, are research

methods that involve numerical tabulations and statistical comparison. 28 Qualitative

methodologies aim to understand, report and evaluate the meaning of events for people in

particular situations. They focus on participants’ own interpretations of their experience and

attitudes and construction of reality. Most frequently used methods include key informant

interviews, focus group, case studies and participant observation.

There are several useful data collection approaches that can be easily operated by Chintan

independently, including: (1) longitudinal surveys, (2) key informant interviews, and (3)

focus group discussions. These methods will help you establish baseline data for future

comprehensive and in-depth impact evaluation give you an overall understanding about to what

extent the social change has been made by Chintan’s program or advocacy campaign.

Selecting Impact Variables

The first step to begin an impact assessment is to identify the domain of impact and select

variables to be measured. Impact variables are the measures of hypothesized outcomes and

impact that could be obtained either by qualitative or quantitative methodologies. Table 1

provides an example of a framework for researching impact in the E-waste campaign. The value

chain of impact is elaborated in stakeholders, domains of impact, an associated series of impact

variables and the appropriate data collection methods for measurement.

28 “Collecting and Using Data for Impact Assessment,” 4.

24

Table 1. Framework for Researching Impact of E-waste Campaign29

Stakeholders Domains of Impact Impact Variables Source of Information

Community resident

Increased awareness of

E-waste recycling

Increased knowledge of

E-waste category

Increased knowledge of

E-waste deleterious effect

of environment and

individual

Survey

Case Study

Improved health

condition

Decreased health issues

of residents

Survey

Behavior change Donate electronic for

reuse

Consumption of

electronic products with

less toxic constitutes

Survey

Key informant Interviews

Case Study

E-waste pickers

Enhanced skill of

handling E-waste

Increased knowledge of

E-waste category

Use of scientific methods

to collect E-waste

Focus Group Discussion

Survey

Municipal officials

Enhanced political will of

E-waste management

Increased knowledge

about E-waste

management

Increased expression of

support for building E-

waste recycling sites

Key informant interview

Focus group discussion

29 Adapted from “Collecting and Using Data for Impact Assessment,” 3.

25

Frameworks

1. Longitudinal Survey

A longitudinal survey is a primary quantitative method that tracks the development and changes

of the targeted group due to specific interventions and generates primary data for analysis.

Longitudinal surveys enable you to track changes of impact overtime and make a stronger case

for impact on certain populations.30

Longitudinal surveys usually involve one baseline survey and several follow-up surveys that

relate the change back to the baseline at earlier stage of program implementation. The time-

period can span for several years and the interval of surveys can either be six months, one year or

two years to allow the impact to be reflected in the sample group. A long time horizon survey

caters exactly to Chintan’s evaluation needs and is the recommended data collection tool.

Advantage and limitations

Longitudinal surveys allow you to keep record of the development and change happening over

time and thus the more effectively collect strong evidence of the impact of program. However, it

is usually very costly because of the requirement of a large sample size for representativeness

and a long time commitment. It is also difficult to maintain the same people in the study that

were interviewed for the baseline.

Survey questions

The design of survey questions plays a determinant role in obtaining sufficient and high quality

data. The questions should be clear, understandable and elicit a meaningful response. A pilot

test survey is helpful in evaluating the value of the questions. Closed-ended questions allow for

easier data analysis by using coded responses. However, open-ended questions collect larger

diversity in responses.31

Survey Design

The survey questionnaire must be rigorously structured. It should have a strong logic

organization and coherent section arrangements. We recommend you use shorter surveys in

length rather than longer surveys to solicit more meaningful responses.32 A survey should not

take more than 30 minutes to finish to ensure respondents give useful responses while not being

burdensome.

Survey sampling

The sample for longitudinal survey would ideally be a random sample drawn from the population

that is divided between a participant group and non-participant group (frequently withholding

30 “Collecting and Using Data for Impact Assessment,” 4. 31 Adapted from “Collecting and Using Data for Impact Assessment,” 4. 32 “Collecting and Using Data for Impact Assessment,” 5.

26

intervention from the non-participant group until a later date). However, it is difficult and can be

costly to engage in this type of rigorous sampling. Using a Simple Random Sample (SRS) or

Stratified Random Sample among the targeted group or beneficiaries is a simpler technique and

can still provide useful data. SRS is made of randomly selected individuals, with each having the

same probability of being selected. Stratified random sample is several SRS drawn from several

subgroups of population according to demographic characteristics (gender, ethnicity, education,

living areas), allowing differential impacts on different populations to be explored As a rule,

each sample must include 32 valid responses to generate statistically significant results.

Exhibit 4 in the Appendix provides a sample questionnaire for surveying households’ awareness

and knowledge about E-waste and behavior of disposing E-waste after Chintan conducting E-

waste drive in Delhi. This survey target is a random sample of households drawn from major

communities in Delhi. It includes both a baseline survey and follow-up surveys (with an interval

of six months after then program ends).

Analyzing Survey Results33

Analyzing survey results centers on what is the best way is to summarize and present the data

collected from the questionnaires. Here we introduce the most three common types of data that

result from longitudinal surveys and show how each of them could be analyzed in a simple and

understandable way.

1. Categorical Data: This type of data uses specific names or labels as the possible set of

answers. It is usually referred to as “nominal data”. Each piece of categorical data cannot

be ranked “higher” or “lower” than another piece, each simply represents different

categories. For example:

How do you dispose your electronic products?

o Donation

o Sold to scrap dealer

o Disposed with household wastes

o Put on the street

Categorical data is usually the easiest data to analyze because you only need to calculate

the share of responses in each category. To analyze, you can simply calculate the number

of responses in each category and divide them by the total number of responses. This is

referred to as the relative frequency in statistical analysis. The relative frequencies should

sum up to 100%. Below is an example of a (relative) frequency table for the above

question.

33 “How to Design and Analyze a Survey”

27

Table Two: Relative Frequency Table with Categorical Data

Answers Responses Share

Donation 30 30/100=30%

Sold to scrap dealer 40 40/100=40%

Disposed with household wastes 16 16/100=16%

Put on the street 14 14/100=14%

Total 100 100%

2. Ordinal Data: Ordinal data can help you to ask “how much” questions. Ordinal response

type data presents answers that make sense in an order. Two examples of a series of

ordinal data may be: “Never, Rarely, Sometimes, Often, Always” and “Strongly Agree,

Agree, Neutral, Disagree, Strongly Disagree.”

When analyzing ordinal data, you should present the data in a frequency table to see the

distribution of results directly. Do not convert the responses to numbers and calculate the

average of those numbers, this could lead to misleading interpretations of the data.

Never Rarely Sometimes Often Always 3% (3) 60% (60) 5% (5) 2% (2) 30% (30)

3. Interval data: Interval data has two main characteristics: (1) the data needs to be ordered

and (2) there needs to be a meaningful range between the values. Interval data is useful

when combined with segmented data from other questions. For example, if you want to

know about people’s awareness of electronic waste from different income levels, you

could segment this question based on a previous interval-style question about their

income. You can use a contingency table to present your finding. Below is an example

about how people at different income levels dispose their electronic waste.

Income Level

(Total/Share)

Donation Sold to scrap

dealer

Disposed

with

household

wastes

Put on the

street

Total

<10000 Rs 8%(2/25) 32%(8/25) 24%(6/25) 36% (9/25) 12.5% (25)

10000-20000 Rs 37% (28/75) 43% (32/75) 12% (9/75) 8%(6/75) 37.5% (75)

20000-30000 Rs 26% (13/50) 48% (24/50) 12% (6/50) 14% (7/50) 25% (50)

30000-40000 Rs 36% (18/50) 24% (12/50) 30%(15/50) 10% (5/50) 25% (50)

Total 30.5% (61) 38% (76) 18% (36) 13.5% (27) 100% (200)

If the interval is even, you can also treat interval type of data as ordinal data and directly

display it in the frequency table.

28

2. Key Informant Interviews

Key informant interviews are a qualitative data collection method that involves holding in-depth

interviews of 15 to 35 people to gain first-hand knowledge. With a loose structure, key informant

interviews aim to foster the free flow of information among interviewees by identifying a list of

key issues to be discussed. Chintan can utilize this method when you want to understand the

motivation, behaviors and perspectives of different stakeholders or when you need to further

interpret quantitative data analysis results.34

Advantage and limitations

The data collected through key informant interviews usually comes from those directly impacted

by the program or policy. Chintan can gain unexpected ideas, insights, and a new understanding

of stakeholder’s experiences with a relatively easy to use method. However, interviews can

easily generate bias if the informants are not representative of the target group or if interviewer

or interpreters bias is strong.35

Steps to conduct a key informant interview 36

1. Formulate survey questions: The first step of conducting key informant interview is to

precisely define the study by listing relevant study questions. The study questions should

be kept minimum with no more than five and should be specific; for example, “Do waste-

pickers enjoy a better livelihood after the training program?” “Does the program have a

positive impact on the health condition of the community?”

2. Prepare a short interview guide: Interviewers must be aware of what topics need to be

covered in the interviews and what main study questions need to be asked. Interviewers

should develop a short but clear interview guide listing the major issues and topics to be

covered in each interview.

34 “Conducting Key Informant Interviews,”1. 35 “Collecting and Using Data for Impact Assessment,” 1 36 Adapted from “Collecting and Using Data for Impact Assessment,” 2-4.

1. Formulate Survey

Questions

2. Prepare an Interview

Guide

3. Select Key Informants

4. Conduct Interviews

5. Analyze Interview Data

29

Table 2. An example of interview guide for “Project Armaan” 37

3. Selecting key informants: The key informants (30-35 people) should be selected based on

their specialized knowledge and perspectives of the selected topic of interest. First, the

interviewers should identify the groups and organizations from which interviewees be

drawn. It is better to cover all stakeholder groups to capture a diverse range of opinions

and interests. For example, groups with potential interviewees for the e-waste campaign

could include municipal officials, project staff, waste pickers, affected households, and

corporate representatives.

4. Conduct interviews: There are several steps within conducting the interviews that will

help secure unbiased, effective results:

a. Establish rapport with interviewees by explain the purpose of the interview,

intended uses of the results and guarantee anonymity. Additionally, you should

minimize the use of jargon and technical terms.

b. Sequence the questions with factual questions first and questions requiring

opinion and perspectives second.

c. Phrase questions to elicit more meaningful and detailed answers. Avoid simple

“yes” or “no” questions. For example, ask questions like “Can you tell me what

you know about E-waste drive?” instead of “Do you know about the E-waste

drive in this area?”

d. Use probing techniques to seek more details and elaboration through follow-up

questions.

e. Maintain a neutral attitude. Interviewers should avoid giving off a strong bias

towards one position or another.

37

“Conducting key informant interviews in developing countries,” 8.

Learning Quality at Education Centers

• Curriculum

• Learning materials

• Attendance

Effects of the Program

• Positive changes in children

• Evidence of changes

• Challenges and areas for improvement

General Questions

• Long-term sustainability of the program?

• Suggestions for the growth of the program?

• Current challenges?

30

f. Create as detailed records as possible to ensure accuracy. You can also use a tape

recorder or mobile phone recorder if available.

5. Data Analysis: To reduce the large amounts of information you collect into manageable

themes for review and examination, you can create a one to two page summary sheet.

This summary should include information on key informants, the reasons for their

inclusion, major observations and their implications. Abbreviations are useful as a

descriptive code to help organize the responses. These codes label data under appropriate

categories and cover main concepts, ideas and key themes such as students’ learning

outcome. A short sheet is then prepared that lists page numbers devoted to particular

items, which later become subheadings in the text. A storage system for the information,

such as an excel sheet or word document can help organize and store the final results.

Finally, tables, figures, or charts of the results are useful to present findings that can then

be communicated to policymakers and other stakeholders.

In Exhibits 5, 6, and 7 in the Appendix we provide a sample templates for conducting key

informant interviews concerning the social impact of children participating in Project Armaan

and relevant data analysis methods.

31

3. Focus Group Discussions

Focus group discussions are a qualitative data collection method that gathers people together

from similar experiences and backgrounds to discuss a specific topic of interest. The group

discussion is guided by a group facilitator (moderator) who gives an introduction of the

discussion topic then helps facilitate an interactive discussion among the focus group

participants.38 Focus group discussions can be useful for Chintan to better understand

stakeholders’ attitudes, opinions and needs while learning barriers to program implementation.

Advantages and limitations39

Focus group discussions are low cost and provides almost immediate results. They can generate

rich and detailed information regarding a group’s idea, experiences and opinion about an issue

and minimizes false or extreme opinions through interactions within the group setting. However,

the flexible format makes it susceptible to facilitator’s bias and thus undermines the reliability of

the results. There is no quantitative data generated to make general conclusion of the population.

The information may be difficult to analyze and must be interpreted under group context.

Steps to conduct a focus group discussion40

1. Select the Team: A qualified team plays a key role in conducting a successful focus group

discussion. The team should be small with a facilitator to guide and record the discussion.

The facilitator should have adequate knowledge of the topic and be skillful at holding

group discussions. Backgrounds in sociology or public policy would be helpful for the

group.

2. Select the Participants: First, identify the groups and institutions that should be

represented in the focus group based on what information you hope to get out of the

discussion. Separate focus groups should be held for each group identified. Consultations

with key informants can help guide the selection of each focus group. It is also advisable

to consult several key informants to eliminate bias of personal preference. The focus

group under each category should be 7-11 people to allow for a free flow of discussion.

All participants should share a similar background and traits related to the discussion

topic to allow participants to freely express their opinions without being subject to group

sentiment. Ideally people should not know each other.

3. Develop a Discussion Guide: A clear outline with the topics and issues to be covered is

useful for facilitator to explore, ask and probe questions and makes data collection more

38 “Conducting Focus group Interviews,” 1. 39 “Conducting Focus group Interviews,” 1. 40 “Conducting Focus group Interviews,” 2-4.

1. Select the Team

2. Select the Participants

3. Develop a Discussion

Guide

4. Conduct the Interview

5. Analyze Results

32

efficient. The guide should be flexible to allow for unanticipated but relevant issues that

may be brought up. Use several carefully designed questions to steer each topic to keep

the discussion more focused. Table 3 is an example of discussion guide to assess waste-

picker’s door-to-door collection of neighborhood.

Table 3. Example of Doorstep Waste Collection Discussion Guide41

Question Key Points Notes

How do you think the waste

pickers’ doorstep waste collection

in your neighborhood?

Waste-pickers’ professionalism

Collecting time

Outcome and result

Waste-pickers’ efficiency

Provide-user relations

Attitudes toward waste-pickers

4. Conduct the Interview: There are several steps within conducting the interviews that will

help secure unbiased, effective results:

a. Establish rapport: The facilitator must create a thoughtful, permissive

atmosphere, provide ground rules, and set the tone of the discussion in the first

few moments of the discussion.

b. Phrase questions: Adopt questions that elicit more meaningful information rather

than impede the discussion. Open-ended questions are preferable to stimulate

detailed stories and perspectives. Facilitators can then probe participants further

by expanding the discussion through “when, where, why, how” questions. 42 Table

4 has sample questions regarding attitudes towards wastepicker livelihoods:

Table 4. Example of focus group questions regarding attitudes toward waste

pickers’ livelihood

Question Notes What do you perceive as the greatest challenge for

waste pickers to earn a consistent and safe

livelihood now?

What do you think the government’s role should be

in relation to waste pickers? (Probe: should it be

responsible for formalize their job?)

Do you think current government set priority to

waste pickers in waste management process?

c. Control the discussion: The facilitator is responsible for maintaining an even and

participatory flow of discussion among all participants. To balance the

participation, the facilitator can try to address questions towards reluctant

participants or give non-verbal hints when someone talks for an extended period

(stop taking notes, look in other direction) or politely intervene the talk and

41 “Conducting Focus group Interviews,” 2 42 Adapted from “A Handbook of Data Collection Tools: Companion to ‘A Guide to Measure Advocacy and Policy’,” 4.

33

refocus the discussion.

d. Minimize group pressure: Facilitator should be cautious when a general idea has

been adopted under group pressure without any general discussion or agreement.

It is useful to probe alternative views to relieve group pressure and elicit different

ideas.

e. Record the discussion: Immediately after the discussion, the team should draw a

diagram of the seating arrangement, conduct facilitator and assistant facilitator

debriefing, label and file notes, tape and other materials. The team should also

summarize the information, the team’s impressions, and implications of the

information for the study.

5. Analysis of results: Qualitative assessment of responses should be conducted soon after

the focus group discussion to identify any potential trends, patterns, and majorly held or

most frequently aired opinion. Each transcript should be dissected while segregating

useful and non-useful parts of the discussion. Each focus group question can also be

broken out separately and accompanied with a summary statement that describes the

general response.

Combine Data Sources and Review

Strategically using and integrating relevant data sources into the monitoring and evaluation of

Chintan’s program can enrich future programs while informing the impact analysis. There are no

perfect means or sequence of collecting background, conducting surveys, and completing focus

groups or key informant interviews that must be followed. These methods should overlap and

complement each other.

Conducting qualitative analyses throughout the assessment provides rich, explanatory

information that provides evidence or refutes research and secondary data. Some of the questions

developed for interviews or focus group discussions may also be adapted into questionnaires for

surveys. Qualitative studies also allow for strategic adjustments and building upon original

surveys as circumstances change. The original survey remains valid for future comparison with

follow-up surveys or changes in hypotheses and survey designs.43

43 “Collecting and Using Data for Impact Assessment,” 9.

34

Recommendations

Chintan has already implemented a data collecting system that regularly documents field work.

However, Chintan needs to expand their data collection scope and move to more rigorous

methods in order to inform their advocacy strategies and eventually measure social impact.

Obtaining first-hand knowledge on social change impacts allows you to appropriately evaluate

programs or advocacy campaigns, identifying the successful elements and make adjustments.

Here we provide the following suggestions for appropriately using these approaches to better

conduct social impact measurement.

1. Identify Key Impact Categories

Conducting longitudinal surveys are time consuming and resource intensive. Chintan

should focus on researching one or two key social impact outcomes that have a strong

causal relationship with your ultimate mission. We recommend you pay special attention

to the positive change of key beneficiaries of the program instead of conducting

comprehensive assessments on the overall environmental change at this stage. Some

considerations include; the improvement of waste pickers’ livelihoods, increased

awareness of zerowaste, increased opportunities for children of waste-pickers to attend

formal school and more.

2. Begin Baseline Quantitative Data Collection

If Chintan is going to move forward in conducting post-impact evaluation, it needs

baseline data. Much of this data will be developed through a robust monitoring and

evaluation framework, presented in the previous section. By strategically combining

quantitative and qualitative methodologies, Chintan will have a more in-depth

understanding of their social impact. Collecting qualitative data would give you more

detailed information about perspectives, emotions and feedback.

3. Identify Technical Partners

Chintan should work to set up good cooperative relations with consulting companies,

research institutes or universities to make social impact measurements more formal and

professional. With the help of outside resources, Chintan can take a step towards building

experimental designs to test the actual causal relationships between programs and social

impact.

i

Appendix

ii

Exhibit One: Theory of Change Framework

Programmatic Activities

1

2

3

4

Interim Outcomes

1

2

3

Policy Advocacy Activities

1

2

3

4

Interim Outcomes

1

2

3

Chintan's Program

WorkImpact

Policy Strategy

Policy Goal

Mea

sure

men

t

Evalu

ation

Mea

sure

men

t

Evalu

ation

iii

Exhibit Two: Theory of Change Example for the E-Waste Campaign

Programmatic Activities

Waste-picker Trainings

E-Waste Collection Drives

E-Waste Utilization Workshops

Interim Outcomes

Increased Waste-picker Awareness

Increased Public Awareness

Increased E-Waste Collection

Policy Advocacy Activities

Public Information Campaign

Network Building

Policymaker Education

Interim Outcomes

Increased Public Awareness of E-Waste Recycling

Increased Media Coverage of E-

Waste Regulation

Build political will for a revision of draft E-Waste

rules

Chintan's E-Waste

Program Work

Impact of Activities

Advocacy Strategy for E-

Waste Campaign

New National E-Waste

Policy

Mea

sure

men

t

Evalu

ation

Mea

sure

men

t

Evalu

ation

iv

Exhibit Three: Overview of Strategic Advocacy Framework Components44

Activities Interim Outcomes Goals Public Information Campaign Organizational Capacity Policy Adoption

Earned Media Expansion (Newspaper,

TV, Radio)

New Partnerships

Policy Change

Network Building

New Political Partners Policy

Implementation

Research and Reports Increased Issue or Organizational

Visibility

Behavioral Change

Policymaker Education Public Will Shift

44 Adapted from “Monitoring and Evaluating Advocacy: A Companion to the Advocacy Toolkit.” and “Developing Monitoring

and Evaluation Framework for Budget Work Projects: A How-To Guide For Practitioners.”

v

Activities Activity Definition Possible Indicator

Public Information

Campaign

Using traditional media,

social media, and stakeholder

engagement to expand the

audience.

New section on the

webpage

New facebook post

Article on website

Number of grassroots

events held

Earned Media Expansion

(Newspaper, TV, Radio)

Reaching out to print, radio,

or television media to gain

visibility on the issue.

Number of outreach

attempts to reporters.

Number of different

types of media

contacted

Network Building

Gaining more visibility and

strength by bringing together

outside groups or

organizations to partner on an

issue

Number of coalition

groups

Types of

constituencies

represented

Number of meetings

held between

members

Research and Reports

Report that relates to a

particular policy area or

program undertaken by

Chintan

Item Produced

Number of downloads

Policymaker Education

Meet with politicians to

explain desired policy, its

expected impact, or desired

program and its expected

impact.

Number of politicians

contacted, met with

Commitments from

politicians

Presentations

Making a case for a policy or

project in-person to an

organization or individual

Number of

presentations held

Types of audience for

each (politician,

corporation, school)

Number in attendance

vi

Interim Outcomes Outcome Description Definition Possible Indicators

Organizational

Capacity

The ability of the organization

to lead, manage, and

implement programs and

strategies

Increased knowledge

about advocacy and

organizing

Improved media contacts

Improved data collection

and analysis skills

New Partnerships

Mutually beneficial

relationships with

organizations and individuals

who participate in an

advocacy strategy

Number or new

relationships

Strength of current

relationships

Amount of collaboration

between organizations

Improved alignment of

partnership efforts

New Political Partners Politicians who adopt an issue

and support it

Number of politicians

recruited

New geographic regions

represented

New levels of government

represented

Number of supporting

actions of politicians

Increased Issue or

Organizational

Visibility

Recognition of the problem,

familiarity with proposed

program, or familiarity with

the organization

Number of media stories

Percentage of audience

with knowledge of policy

or program (through

polling)

Public Will Shift

Willingness of public to

support an issue or take part

in a program

Percentage of audience

willing to participate in a

program

Attendance at an

advocacy event (training,

presentation, meeting)

Political Will Shift

Willingness of politicians to

act to support an issue or

program

Number of officials who

support the policy or

program

Votes for or against a

policy

vii

Goals Outcome Description Definition Possible Indicators

Policy Adoption Passage of a policy or

adoption of a program

Policies or programs

formally adopted or

begun

Policy Change

Revision of a current policy

to address a new aspect of a

problem.

Policy formally

amended

Policy Implementation Proper implementation of a

policy or program.

Level or funding or

resources allocated

Expected outcome

experienced in

practice

Behavioral Change

Changes in behavior of target

audience in accordance with

policy or program

Number participating

in program

Results of policy seen

in practice

viii

Exhibit 4: Sample Survey

Chintan’s E-waste Assessment of New Delhi

Questionnaire for Households45

Date:____________

Location:_____________

Interviewer:_____________

Interviewed Person:

Name

Telephone

E-mail

Ward/Suburb

Town

Introduction

Chintan is a not-for-profit organization based in New Delhi, in collaboration with *****, is

collecting data on e-waste generation and management in Delhi in order to know the current e-

waste generation and management in the City. The study will enable the authorities to determine

the necessary steps required for handling e-waste. We would be grateful if you could spare some

time to answer a few questions:

45 Adapted from “E-waste Assessment Tanzania,” 65.

ix

Questions about Awareness and Behavior

Questions Response 2.1 Yes No

a) Do you know what e-waste or

waste of electrical and electronic

equipment is?

b) Are you aware that components

of e-waste need a special treatment

in order to be safely disposed of?

c) Do you have waste collectors in

your community?

2.2 Yes, Everything Yes, But No E-waste No

Do waste collectors come and pick-

up waste at your door? Do they

collection e-waste too?

2.3 Yes No I don’t know

Is the way e-waste is currently

collected convenient to you? How

could it be improved?

Comments

Product Tracing of Household Electronics

Products

Where was it bought

and in what

condition? (new/ used

& working/ broken)

Number of

years used

Number of

years stored

In what condition was

the product at the end

of life? (Working –W;

Broken – B; Broken

but Fixable - F )

Refrigerator Washing Machine Microwave Personal Computer Mobile phone Laptop TV Radio

x

Usual Method of Disposal of Electronic Products (please mark with x)

Donation

Sold to

second-

hand dealer

Sold to

scrap

dealer

Disposed with

household waste

Put on the

street Other

Fridge

Washing Machine

Microwave

Personal

Computer

Mobile phone

Laptop

TV

Radio

Number of person in the household (please tick appropriate box)

1 2 3-4 5-8 More than 8

Total Household Income per Month (Rupee) (please tick appropriate box)

Less than

10,000

10,000-30,000 30,000-50,000 50,000-70,000 70,000-100,000 More than

100,000

xi

Exhibit 5: Key Informant Interview Sample Questions of Assessment of Project

Armaan

1. (Questions for representative of children) How has participating this project been of benefit to you?

What specific knowledge have you learned through participating the project? What other. Are there

any extracurricular activities that you find interesting and benefit from them?

2. (Questions for representative of parents) How is your thought of sending child to informal study

center? Has children’s participation of the project brought any change to your family? What are the

factors that you hesitate to allow children participate this project? (probing economic burden, sexual

bias)

3. (Questions for teachers) How is students’ learning quality in the study center (probing improvement

of numeracy and literacy skill). What are the challenges now for improving children’s learning

outcome or mainstreaming more children into formal school (probing facility, learning materials,

children attendance etc.)

4. (Questions for project staff) How do you see the sustainability of this project in the future? (Probe

funding, government support etc.)

xii

Exhibit 6: Example of a Coding System for the Evaluation of Project Armaan46

General Description Code Transcripts Page Number

Learning outcome LEA-OUT

Literacy skill LEA-OUT-LIT 7,9,33,44

Numeracy skill LEA-OUT-NUM 15.54.176

Computer skill LEA-OUT-COM 28,37,104,135,170

Extracurricular knowledge LEA-OUT-EXTR 3,35,62,78

Project Effects PRO-EFF

Children life PRO-EFF-LIFE 11,26,33,55,100

Family PRO-EFF-FAM 5,33-4,45

Project Sustainability PRO-SUS

Funding PRO-SUS-FUN 17,86-7,187-3

Government Support PRO-SUS-GOV 139-41,198

Exhibit 7. Example of Data Analysis for the Key Informant Interviews47

Recommendations for Improving Mainstream Rate

Recommendation Number of Respondents

Improving parental involvement and awareness 23

Conducting teacher training programs 21

Adding other subjects to the curriculum 16

46 Adapted from “E-waste Assessment Tanzania,” 26. 47 Adapted from “E-waste Assessment Tanzania,” 29.

1

References

Agency for International Development. (1989). Conducting Key Informant Interviews in

Developing Countries. Washington DC: Kumar, Krishna.

Center for Good Governance. (2006). A Comprehensive Guide for Social Impact Assessment.

Chandurkar, D. and Sen N. (2014). Developing Monitoring and Evaluation Framework

for Budget Work Projects: A How-To Guide for Practitioners.

National Foundation for India.

Chintan Environmental Research and Action Group. (2014). Annual Report: 2013-14.

New Delhi.

Chintan Environmental Research and Action Group. (March 2013). Project Armaan:

An Initiative to Help Children Stop Picking Trash and Get and Education. New Delhi.

Clark, C., Rosenzweig, W. (2004). Double Bottom Line Project Report: Assessing Social Impact

in Double Bottom Line Ventures.

“The Evaluation Exchange.” Harvard Family Research Project, Harvard Graduate School of

Education, Vol.13, Num.1, (2007).

Learning for Action. (2013). Advocacy Evaluation Mini-Toolkit: Tips and Tools for Busy

Organizations. San Francisco, CA.

Magashi, A., Schluep, M. (2011). E-Waste Assessment Tanzania.

Cleaner Production Centre of Tanzania & Empa Switzerland.

Organizational Research Services. (2007). A Guide to Measuring Advocacy and Policy.

Seattle, WA: Gienapp, A., Reisman, J., and Stachowiak, S.

Organizational Research Services. (2007). A Handbook of Data Collection Tools: Companion to

‘A Guide to Measuring Advocacy and Policy’.

Seattle, WA: Gienapp, A., Reisman, J., and Stachowiak, S.

Overseas Development Institute (2011). Background Note: A Guide to Monitoring and

Evaluating Policy Influence. London: Jones, H.

Red Ochre. (2011). Social Impact Assessment.

Accessed via:

http://www.vai.org.uk/wp-content/uploads/2011/07/110707-Social-Impact-Toolkit.pdf

2

United Nations Children’s Fund. (2010). Monitoring and Evaluating Advocacy:

Companion to the Advocacy Toolkit. New York: Coffman, J.

USAID. (2006). Collecting and Using Data for Impact Assessment.

USAID. (1996). Conducting Focus Group Interviews.

USAID. (1996). Conducting Key Informant Interviews.

Zapier. The Ultimate Guide to Forms & Surveys. Peters, C., Schreiber, D., Guay, M.,

and Baedell, S.

Accessed via: https://zapier.com/learn/ultimate-guide-to-forms-and-surveys/#toc.