Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

1 | Value for Money

Foreword

As evaluators, what can we do to provide the most value for money from our evaluations? This is a

question that matters not just for evaluators, but for the clients that commission these evaluations

and ultimately use our findings.

Between 2014 and 2016, we published a series of blogs that unpacked and discussed how to

increase the value of an evaluation, while managing its costs. The series was stimulated by

discussions that pushed for greater value for money of evaluations. In the series, we covered the

evaluation cycle, how to strategically choose what to evaluate, key questions to ask during the

design and implementation of the evaluation, and how to share evaluation insights.

In the series, we also share practical suggestions that have been applied in IEG, but can be adapted

also to different contexts. The series had over 20,000 readers, which is why we decided to publish

the collection.

To reflect some of our more recent insights, we’ve updated the blogs and provided additional

context and resources.

We hope you will find this mini-publication useful.

2 | Value for Money

Table of Contents Foreword ................................................................................................................................................ 1

Value for money in the Evaluation Business? ........................................................................................ 3

Unpacking the "Value" in the Value for money of Evaluation ............................................................... 5

Value for money: Value-for-Whom?...................................................................................................... 7

Evaluation: Does More Money Mean More Value? .............................................................................. 9

Independent versus self-evaluation—is there a place for both? ........................................................ 11

The first step to a great evaluation? Make the right choices .............................................................. 14

Value for Money—Getting It Right In Designing and Implementing the Evaluation ........................... 17

What Comes After the Evaluation is Completed? ............................................................................... 20

The Complexity of Estimating the Value and Costs of an Evaluation .................................................. 23

Biography ............................................................................................................................................. 25

3 | Value for Money

Frequent readers know that I’m obsessed

with evaluation being as effective as it can be.

As much as I am passionate about evaluation

and the learning that it allows, its usefulness

lies in influencing change, hopefully for

the better.

As promised in my blog When Rating

Performance, Start with Yourself, we at IEG

worked hard to develop objectives and

metrics that tell us whether we are achieving

what we want. Our annual work program,

approved in June of every year, is online for

everyone to see.

Our metrics for the work program were built

on a two-tier system:

▪ intermediate outcomes over which we have greater control, and

▪ medium-term outcomes, which describe how we affect the World Bank Group

The first is about our ability to generate

meaningful evidence from evaluations and

through that a greater understanding of what

works. The second is about whether the

World Bank Group uses the information to

improve its services for clients and, in turn,

enhance its development effectiveness.

The intermediate outcomes include indicators

such as feedback from clients and

independent assessments of the quality of

our evaluations (our meta-evaluation panel,

see Who Evaluates the Evaluators? is in the

process of testing quality criteria). Other

indicators are strategic choice of our

evaluations, timeliness, and efficiency. In the

medium-term, we are hoping to see that

“To determine value for money, we need the cost (which we as evaluators know, at least for producing the evaluation) and the value.”

JULY 29, 2014

Value for money has become a much-buzzed about concept in international development in recent years. How can we apply it to evaluation and how would it change the way we do business if we did?

4 | Value for Money

policy and operational choices are better

informed by evidence from independent

evaluation, and that this leads to better

outcomes for clients.

Some of these might work better than others,

and we will revise them as we learn more.

But there is one indicator that we want to

expose ourselves to that’s particularly

interesting and tricky: value for money of

evaluation. The concept is easier when we

think of goods and services we buy: is this car

worth the sticker price; is that meal worth the

money I spent? And even in these examples it

is clear: there is an intangible metric—our

taste, which is the yardstick against which to

measure “worth.”

This becomes more difficult when it comes to

knowledge. The payoff of knowledge might

not be obvious at the time when it is

generated. The results of better knowledge

might take time to materialize and people

might internalize the knowledge to the extent

that they actually might not remember where

they got it from. Yet, for some knowledge

products the equation is clearer: each time

we buy a book, a report, a software

application, we acquire knowledge with a

clear value for money proposition that in

today’s Internet age is challenged by free

applications, open data, and reports that put

a lot more knowledge in people’s hands

without charging them for it.

For evaluation the challenge becomes even

greater. Many people still have negative

associations with evaluation—a bad

scorecard, an embarrassing assessment, or

worse: fear of losing a project, a job, or a

funder! In many institutions where I have

worked, control over the budget of evaluation

was one way to ensure independence or

curtail it. So, could someone argue that an

evaluation wasn’t value for money work when

it actually contains some inconvenient truths?

Or, how about those evaluations that shed

light on facts that are known, but no one

dares talk about—how does one value the

transparency that this evaluation brings

about? And then there are evaluations that

generate genuine new insights, but are they

worth the price?

If people were free to put a price-tag on

evaluation, how many would say it’s too

expensive, what would be their willingness

to pay?

To determine value for money, we need the

cost (which we as evaluators know, at least

for producing the evaluation) and the value.

How can we best assess it when, just like

other knowledge products, it is hard to

determine whether policy or operational

choices would have been exercised differently

if the evaluation had not been available? And

how does the value relate to the cost?

We don’t have the answers. Do any of you?

5 | Value for Money

My blog in July on Value for money generated a lot of feedback. Many thanks for the thoughtful comments and suggestions. The sense I got from the comments is that we need to unpack the value proposition of evaluation before we can contrast it with the money—cost and price—part of the equation. Theory of Change The value of evaluation lies in helping to improve development results. Our aim is that evaluation leads to better informed decisions and actions before and during the life-cycle of an intervention, which in turn results in greater benefits at a lower cost. The Value of Independent Evaluation The value of independent evaluation to an institution like the World Bank Group lies in its transparency, accountability, and learning. The importance of independence was discussed in a previous blog.

Specific evaluations add value by:

▪ Producing information no-one else has compiled. That’s what we did in our PPP evaluation where work across The World Bank Group was so dispersed that it took our evaluation to get an overall sense of the portfolio. The value of this information to our client, the new Cross-Cutting Solutions Area for PPPs, can be estimated by the saved cost to them.

▪ By stepping back from the daily routines of designing and implementing interventions, independent evaluation can generate new insights by shifting perspectives and combining evidence in ways that allow people to see improvements that they otherwise might have missed. For instance, our evaluation of sustainable finance of transport investments found that in spite of policy commitments from the highest levels, the share of projects

“The value of evaluation lies in helping

to improve development results. Our

aim is that evaluation leads to better

informed decisions and actions before

and during the life-cycle of an

intervention, which in turn results in

greater benefits at a lower cost.”

SEPTEMBER 16, 2014

The value of independent evaluation goes beyond cash equations

6 | Value for Money

with objectives to sustain transport investments had declined to 10 percent. If our recommendations are heeded, countries could attain savings and higher economic returns on their investments in the transport sector.

▪ Evaluation evidence (or validating self-evaluation findings) on results and performance generates an understanding of what works and what needs to be fixed. Our validations of project completion reports provide information which then gets aggregated in the annual Results and Performance report. The value of our work would be to help decision-makers and implementers take corrective actions when interventions do not produce expected results. In other words, to avoid or reduce losses.

Realizing the Value of an Evaluation Value is not generated by simply producing information. It’s the changes in behavior, practices and policies that matter. That means we have to reach out and share lessons and knowledge in ways that get absorbed and influence behaviors. We are fortunate to have the resources to do that and have committed ourselves to revving up our work on this front. To know whether we succeed, we have an established system to track the implementation of recommendations over time. Our results framework also includes an external assessment of the actual use and impact of our evaluations. Estimating the Value-Added But, our measures do not quantify the expected value improvements that should

follow from evaluations. How would we make this work? For example, in our evaluation The Big Business of Small Enterprises we found that the WBG spends $3 billion annually on targeted support to small and medium sized enterprises. But only 34 percent of the interventions state whether they address market failures and only 13 percent provide evidence to that effect. And while the overall outcome ratings of these projects are positive, we also found that few projects had quantified data and evidence to demonstrate the extent to which SME development had been helped and what wider economic effects this has had. Assuming that as a result of our evaluation, targeted SME interventions would have a clear line-of-sight on the market failures they will address and how this will affect the sector’s growth and multiplier effects, the value of the evaluation could be enormous. Does it translate into a number, a dollar amount? Only if we start seeing data that helps quantify outcomes. And, let’s not forget the intangible value of more or better information.

7 | Value for Money

When I first wrote about Value for money in the Evaluation Business, a number of people commented about the importance of addressing the question of “value for whom” and whether evaluation was generating value for a diverse set of stakeholders. As I’ve written in the past, knowing your stakeholders is important to me as an evaluator. To be effective we need to understand who our stakeholders are, what is at stake for them, and what they want to get out of the evaluation. So, what is the value we are creating for stakeholders, and can we please them all? Yes and no. We have a range of stakeholders that look to us for independent evaluation and evidence about what works and what needs fixing.

Within the World Bank Group our main stakeholder groups are:

▪ The Bank Group shareholders, represented in the Board, who look to IEG and who use our work to engage with senior management on strategic issues and policy directions and who rate our work highly for its usefulness;

▪ Senior management which derives the

highest value from our major evaluations and from our Results and Performance report that summarizes performance trends of the portfolio overall, also with the aim of raising strategic issues. Feedback from this group about the quality of our work, and hence, its value added has decreased and we are working hard to improve our standing. Recent evaluations and how they were received indicate we are on the right

“To be effective we need to understand who our stakeholders are, what is at stake for them, and what they want to get out of the evaluation.”

OCTOBER 28, 2014

Evaluators need to ask themselves tougher questions about what difference their recommendations will make and the value they’ll add.

8 | Value for Money

track, but we will follow-up with our client surveys and

▪ Colleagues who work in The World Bank

Group’s country offices and regions on country engagements and in the new Global Practices and Cross-Cutting Solution Areas handling projects. Signals from this group are mixed. On the one hand, our annual client satisfaction survey tells us that operational staff value our work, but individually we often get questions that show dissatisfaction.

Answering the right questions The comments on the first value for money blog raised other concerns, namely whether we were answering the right questions or had come too late in the process to make a difference. These are valid concerns, and they arise in all evaluation functions, not just in IEG. Some of these issues come from different expectations of what evaluation can do. For instance, we often get asked why we cannot provide hands-on support to improving quality. The simple answer is it would interfere with management’s responsibilities and undermine our independence. In other cases when it comes to asking the right questions, we share the approach papers for our major evaluations and concept notes for our major learning products with stakeholders and ask for feedback. At the end of the evaluation process, we have a similar interaction to ensure we get the facts right and that the recommendations make sense.

Clients and users The users of our evaluations also go beyond the World Bank Group, even though we do not have direct client relationships and rely on Bank Group operational staff to follow-through on lessons and recommendations. These stakeholders include civil society organizations and other bilateral and multilateral organizations that rely on us as an arms-length institution that is close enough to see what is going on, while removed enough to have an independent evidence-based view. Does this mean we serve every stakeholder and answer every question? Certainly not! Reflecting on the value for money question there are three ways in which we as evaluators can push ourselves further:

▪ We can be sure that the questions we ask in our evaluations hone in on specifics that deepen the understanding of results and past experience;

▪ We can ask ourselves tougher questions about what difference our recommendations will make once implemented, and what value-added they will create; and

▪ We can capitalize on our work by making it more accessible to different user groups, creating greater value for them, at relatively little cost to us.

9 | Value for Money

In the last couple of posts in this series on value for money in evaluation I looked at value and how to measure it. Today, I want to turn to the question of money and the relationship between price and cost. In Value for money in the Evaluation Business I wrote about the perceived or actual "price" of a negative evaluation: a bad scorecard, an embarrassing assessment, or worse - fear of losing a project, a job, or a funder. Today, I want to unpack the question of cost, both direct and indirect. Our attention often gets absorbed by direct evaluation costs—what is the budget, what have we spent it on, how much do consultants and fieldwork cost—which are important considerations, especially when there is so much data and information available that needs to be considered in ever more complex projects, programs, or policies. At the same time, efficiencies can be achieved

with new technology that allows for the analysis of data and text in ways that surpass more time-consuming methods that potentially put evaluations at risk of inconsistencies. Access to information ahead of time can also reduce the amount of required fieldwork (where a lot of the cost sits). But in discussing value for money, we also need to look at indirect costs—sometimes monetary, often not—that can enhance or diminish an evaluation. Transaction costs. All of us know how "expensive" meetings are: they take a lot of time, distract us or take time away from other things, and depending on how they are run, they can feel like an utter waste of time. For evaluators meetings can be particularly tricky as the people we’re talking to may be reluctant to share information during the evaluation interviews or may fear negative results when discussing draft findings.

“in discussing value for money, we also

need to look at indirect costs—

sometimes monetary, often not—that

can enhance or diminish an evaluation.”

JANUARY 13, 2015

Looking beyond the price when assessing the cost of an evaluation

10 | Value for Money

My own approach to turning this handicap into an advantage and eventually a value preposition? Be extremely well prepared: This requires reading all there is to read and understand prior to meeting stakeholders. It has enabled me to engage with people in a deeper conversation about what matters to them most, their work, their project, the results they want to achieve, and what stopped them from doing so. When they see that the evaluation is addressing their problems and recommending ways to fix them, they appreciate its value. Increase the quality of the engagement: How we engage with stakeholders can determine the cost of our evaluation and its value-added. For instance, meetings that tackle problematic findings in a frank and constructive way enrich the evaluation process, while engagements that avoid key problems or deal with them in confrontational ways are not productive and could increase the cost. Cost of quality assurance. Quality is paramount to evaluation! Poor quality will come at the cost of reputational risk and reduce the effectiveness of evaluation. In addition, there is a real cost to quality assurance—the time needed to review, discuss and revise reports. What can help minimize these costs, without making the wrong trade-offs about quality, is having a clear and agreed understanding of standards and expectations, and using modern techniques that help triangulate evidence from different sources to crystallize well-sourced, clear messages. Getting it right is hard, but it is also indispensable if we are to increase the value of evaluations while reducing the cost of their production, both in

terms of time spent and in credibility vis-à-vis stakeholders. Cost of discussing draft findings. Related to the cost of quality assurance, is the cost of engagement with stakeholders on the draft final report. There is the simple time factor where everyone’s "time is money". Cost of sharing lessons. Very often, the cost of an evaluation is estimated to include only the work up to when the report is completed and published. Encapsulating evidence and knowledge in a document without sharing it diminishes its value. Getting more mileage–more value—out of evaluations requires additional investments in disseminating their findings. In the past, that might have meant simply sending copies of the evaluation to interested parties. Today, we have a panoply of communications tools that help us to not only share the information, but also to tailor and package it to specific audiences who otherwise would not read an entire, often lengthy and detailed, evaluation report. In short, there are costs we do not count, but that can make the difference between an evaluation that is costly and one that brings added value.

11 | Value for Money

As we come back to our blog series on the value for money of evaluation, let’s start with the question why we have to spend all this money on self- and independent evaluation. Couldn’t we improve the value for money equation by saving on one or both of them? Spending money on monitoring and evaluation affects the cost of business. Business efficiency has been one of the reasons why institutions and private sector entities have been hesitant to invest in monitoring, and even more so in evaluation. To them, the cost associated with evaluation does not pay back in benefits or profits derived from the information. Or so they think! And this perception would be even more pronounced with the appearance of doubling the cost with separate self- and independent evaluation. But, that presumes there is no cost of “not knowing.” In today’s competitive world,

successful companies and organizations invest in understanding markets, critical development issues, and other players in a crowded field. Much of that data gathering is forward looking, gathering information about future opportunities, and less about past results. Still, why separate self-evaluation and independent evaluation? Can’t we get away with just one. In an ideal world, we would have one evaluation system that is completely streamlined and efficient, produces the right kind of information at the right time, and does not shy away from difficult or unwanted messages. But, as many of us development professionals know, that is a tough call. Value of Self-Evaluation. Self-evaluation has the advantage of being much closer to the action. It therefore has a great potential to

“In an ideal world, we would have one

evaluation system that is completely

streamlined and efficient, produces the

right kind of information at the right

time, and does not shy away from

difficult or unwanted messages. But, as

many of us development professionals

know, that is a tough call.”

JANUARY 26, 2016

Why spend money on self- and independent evaluation? Couldn’t we improve the value for money equation by saving on one or both of them?

12 | Value for Money

promote learning, if institutional incentives are right. The value of self-evaluation is to fix problems as they arise, avoid repeating mistakes unnecessarily, and to account for resources and actions. Either way, unnecessary expenses can be spared, harm avoided, and relationships maintained or even improved. Value of Independent Evaluation. This second tier of evaluation derives its value exactly from being arm’s length. It gives credibility to the institution as a whole, which increases trust and confidence of shareholders and others. In addition, it can make up (in part) for weaknesses in the self-evaluation system. In particular, every company and organization has “well-known secrets”, things everyone knows are not working, but everyone understands that one ought not to talk about them. Independent evaluation, with its mandate to speak “truth to power” can and should raise these issues. Importantly, independent evaluation produces value by helping to fix systemic issues, and generating new insights when assessing a set of interventions from an arms-length perspective. This can lead to greater institutional efficiencies and better results, and to identifying opportunities that would not be seen from the day-to-day operational perspective. Estimating Value Added. The value of these gains from evaluation (self and independent) are hardly ever estimated. On the operational side, more effort is spent on pressing for higher quality evaluations or defending results and performance of past programs. As evaluators we are still struggling to see whether our evaluations are effective, meaning recommendations are taken up and implemented. This leaves limited space to determine the value added from evaluation.

Cost of Evaluation Systems. A common lament from operations managers is that the cost of collecting data outstrips monies available for managing the project. This is even more so in humanitarian assistance or the private sector. The trade-off seems to be that instead of evaluation (direct and indirect costs), money could be spent on serving more people better, making a project work more efficiently, or reducing the cost of a private sector deal. To some extent, I am sympathetic to this position. It is not always obvious how evaluation systems (self or independent) speak to the information needs of decision-makers or program implementers. Data collection requirements seem endless, reports are unwieldy, and messages nuanced. Simple, sharp and clear answers about "what’s next" don’t emerge easily. And, often there are calls for more and more data collection. Nonetheless, there is an important middle-ground between too many details and an effective evaluation system.

Where does this leave us in the end? Yes, there is space, even need for both self- and independent evaluation systems. They complement each other. Better self-evaluation will result in greater agility, timely course-corrections, and ultimately efficiency and results of operations. In addition, high quality self-evaluation will aid independent evaluation, enable it to focus on strategic issues and resolve systemic problems that contribute to institutional effectiveness and efficiency. To achieve this vision, all sides need to act together. Decision-makers and implementers need to work with the designers of self- and independent evaluation systems to clarify information needs. Challenges will arise as different needs should be weighed against each other. For instance, immediate business

13 | Value for Money

needs might not pay due attention to development outcomes. Nonetheless, there is probably room for finding a better balance to achieve:

▪ Greater efficiency in self- and independent evaluation systems whereby data collection is "right sized",

▪ Greater effectiveness of self- and independent evaluation systems whereby evaluation evidence is used in decision-making, corrective action, and learning for future policies and programs,

▪ Filling information gaps by developing a deeper understanding the value added of evaluation evidence, the improvements it has (or could have) led to, and resultant implications in terms of costs and savings.

14 | Value for Money

Evaluators must make strategic choices at

every stage of the evaluation cycle. This

matters because value can be generated or

lost—and costs incurred or saved—

throughout the lifecycle of an evaluation.

While some of the factors that affect value-

creation are outside the control of evaluation,

there are plenty of things that can be

anticipated and managed by evaluators.

There are three distinct questions that evaluators can ask in order to create value rather than destroy it.

▪ What to evaluate and when;

▪ How to evaluate; and ▪ With whom and how to share

evaluation results, insights, and knowledge.

In this part of our value for money (VfM) of evaluation series, I want to unpack each of these questions, starting today with the

importance of making strategic choices on what to evaluate and when. What to Evaluate When. The choice of what gets evaluated is particularly important for VfM: value and cost (money) are driven by what gets evaluated and how. This is even more so when looking at the portfolio of evaluations of an institution. Choices need to balance two sets of considerations.

▪ Coverage of the institution’s work to generate evidence and evaluation of the health of the institution as a whole; with

▪ Strategic issues, which once deeper evidence is generated, can be the biggest change agents for institutional results and performance.

This balance can be achieved by the right combination of business lines that focus on different units of accounts (from projects

“The choice of what gets evaluated is

particularly important for making sure

an evaluation is providing value for

money. The value and cost (money)

are driven by what gets evaluated

and how.”

APRIL 25, 2016

What can evaluators ask in order to create value rather than destroy it?

15 | Value for Money

through country strategies to policies and corporate issues) with different types of evaluations (validation of self-evaluation, independent evaluation, and synthesis products that build on existing evaluation evidence).

Mirroring the Institutional Portfolio. To evaluate the health of the portfolio overall, one needs adequate evaluation coverage of an institution’s work. To achieve this, it is important to understand its entire portfolio of activities. More often than not, it will take

several variables in combination to describe the portfolio. They can include size, geographical location, sector, type of instrument. In other words: characteristics that need to be taken into account to ensure representativeness of the overall portfolio.

For instance, if an institution works predominantly in Africa, but the evaluation portfolio is focused on Asia, the results would not be an accurate representation of the health of the institution’s portfolio as a whole. The challenge lies in finding the right balance between evaluating enough to ensure appropriate coverage, but not so much that costs become disproportionate. In some cases, a 100% sample makes no sense, in others it is needed to provide in-depth analyses of important aspects of the portfolio. Making Strategic Choices. For other evaluations, it is important to

understand which of them can make a strategic difference. This is particularly so for large, costly evaluations. Their value derives from (and their cost can be justified by) their ability to bring about evaluative evidence and insights that can stimulate strategic or

16 | Value for Money

systemic changes. These evaluations might tackle controversial issues or focus on aspects of ongoing change processes where just-in-time evaluation evidence can help inform decision-makers’ choices. But, there is no simple way when to do one or the other, and not just one thing that gets you to the answers about what’s strategic. Ask Your Stakeholders. Stakeholders who want to use evaluation evidence need to receive that information at the time when they are debating and deciding what to do. It is therefore important to understand

▪ The issues they are tackling. In some cases, stakeholder readiness to deal with issues is crucial to materialize the value that evaluation evidence brings to the table. But only, if they need evaluation evidence to understand the direction of change. In other cases, it might be more important to evaluate controversial issues that (some) stakeholders would rather not see addressed; and

▪ Decision-making processes, especially milestones when evaluation evidence is needed. It takes thinking ahead as much time as it takes to complete an evaluation—to pick up strategic issues in a timely way and have an evaluation available right in time for decision-makers’ use.

Understand the Past with a View to the Future. When an institution is making a strategic shift, an evaluation of past policies and programs will help understand how far the institution has come, how big and where the gaps are. There are sensitivities that need to be managed well when making strategic choices of this nature. For one, the vision for the future has to be settled, to ensure the

evaluation applies the right lens for the future. Otherwise, it will not generate its potential value, but waste resources. In addition, it is important to manage well the concern that the past will be evaluated against a new yardstick. In such case, the risk of damaging relationships is high—a cost that might outlive the specific evaluation—and might outdo the value of evaluation evidence that helps prioritize areas for improvement. Tackling Complex Issues with a Set of Evaluations. Some issues are fundamental to moving an institution to the next level. But they are too big and complex to tackle at once. Instead, a structured program of strategically selected evaluations can help shine light on the issue from different directions. This is exactly what we are trying to achieve with IEG’s Strategic Engagement Areas, which were selected as fundamental for achieving the Bank Group’s twin goals, but too big for a sole evaluation.

17 | Value for Money

Value can be generated or lost and costs can be incurred or saved throughout the lifecycle of an evaluation, starting with what to evaluate and when; how to evaluate; and with whom and how to share evaluation results, insights, and knowledge. As part of this series on getting value for money from evaluation, I wrote about the importance of making strategic choices on what to evaluate and when. Today, I want to focus on opportunities to increase value for money during the evaluation process. A number of readers might ask: why focus on this? Isn’t this part of evaluation practice well established by now? Yes it is, but choices made in designing and conducting evaluations can and often do influence the overall cost and value of evaluation. Ultimately, the value of an evaluation lies in it being used. For that to happen, the

evaluation and its underlying design and process must be credible, timely, and useful. Credibility is necessary! Why else would anyone take note and act on findings or recommendations? A large part of credibility is derived from evaluators’ professional standing. They are known for their expertise. Their word counts. But, evaluation is more than an expert opinion. Making the right choices about scope and methods also has an important part to play. For instance, if sampling methods introduce biases, or methods rely on too few sources of information for meaningful triangulation or assessment of results, validity of findings and credibility of the evaluation suffer. Now, does that mean sampling 100% of a program, or asking every conceivable question in a survey are the right answers to credibility? Certainly not, as wastefulness in an evaluation undermines credibility just as much—

“Choices made in designing and

conducting evaluations can and often do

influence the overall cost and value of

evaluation.”

MAY 24, 2016

Why focus on Value for money? Isn’t this part of evaluation practice well established by now?

18 | Value for Money

especially when the evaluation critiques the efficiency of the program it is evaluating.

Timeliness is quintessential for an evaluation to be used. When a decision needs to be taken, for example, to renew a program, make changes to it, or stop it altogether,

19 | Value for Money

evaluation results need to come in well ahead of that decision. Nothing is more disappointing, and costly than to miss such an opportunity to influence decisions. This is the reason for my previous blog: making strategic choices about what to evaluate when should take into account key milestones in decision-processes. That way, an evaluation can start early enough. Another difficulty is when evaluators face trade-offs between scope and timeliness, especially when looking at larger complex programs. There are no simple recipes for such situations, but choices have to be weighed at the outset of an evaluation and managed throughout. Ultimately, is greater detail adding more to the credibility of or knowledge generated by the evaluation? But, what good does it do, when this information surfaces after decisions have been taken? Usefulness. Michael Quinn Patton is the grand master of utilization-focused evaluation. He has provided insightful discussions, guides, and checklists to ensure evaluations are focused on use. I couldn’t agree more with them on the need to identify specific primary users. I would add here that at least in my experience in multilateral organizations, very often we do not have a single prime user. We need to satisfy different demands with the same evaluation. And, even if we did evaluations that are primarily focused on the needs of the Executive Board, to whom IEG reports, we generate important insights for management and operational staff as well. And, they need to take up lessons and recommendations in their actions. Thinking of evaluation reports as composed of building blocks—shorter summaries for executives, detailed annexes for operational staff—one can satisfy different needs with the same evaluation and report.

In addition, usefulness is often determined by the questions the evaluation asks and aims to answer. If these do not speak to the interests of key stakeholders, utility and with that perceived or actual value-added might not be attained. The “So What”? The Importance of Recommendations. A critical aspect of the value for money of evaluation derives from its recommendations. Recommendations define the issues that the evaluation prioritizes and suggests will make the biggest difference in the program—its performance and results—if addressed. Recommendations derive from evidence, findings, and conclusions, and often focus on observed shortcomings to correct them. But, especially when assessing large complex programs, many avenues for improvement or scaling up of success might be possible. Recommendations are sometimes exposed to tests whether they are actionable, but less to the question what difference will they make? Once implemented, what value-added will they generate? To foster ownership of recommendations it is important for evaluators to engage with program managers and seek the answer to the "so what?" question. The more these stakeholders understand the value added from implementing the recommendation, the higher the likelihood of ownership and follow-up action will be.

20 | Value for Money

In previous blogs, we looked at the various ways in which we can create or increase the value (for money) throughout the evaluation life-cycle. Now it is time to look at what comes after! What happens after an evaluation is completed can impact its value. Yet, this is a phase in the cycle that is often undervalued and inadequately resourced. For the longest period, evaluators thought their job was done when the evaluation report was completed, sent to decision-makers, and discussed by authorities. Gradually, the realization set in: recommendations didn’t implement themselves. In this blog, I look at some of the follow-up processes and measures that can help ensure that evaluations result in meaningful action. Often, institutions do not schedule time or money for this work, but it is essential to get ownership, learning, action and change from an evaluation.

Early Engagement: Credibility and Ownership. My previous blog focused on a couple of things that help increase value (and have cost implications) during an evaluation. Among them was building credibility through transparent evaluation metrics and processes. This kind of engagement during the evaluation process is critical to building ownership of its findings. Surprising stakeholders with findings that are critical and point to problems, is not helpful to getting evaluation findings heard, insights understood, and recommendations taken seriously. In engaging with decision-makers at the World Bank Group, IEG is, for example, experimenting with participatory processes for developing evaluation recommendations jointly with evaluees. Our engagements allow evaluators and program managers to discuss findings and identify how best to translate them into recommendations. The opportunities: greater ownership as the basis for greater implementation of agreed actions. The challenge: keeping the focus on

“Surprising stakeholders with findings

that are critical and point to problems, is

not helpful to getting evaluation

findings heard, insights understood, and

recommendations taken seriously.”

JUNE 29, 2016

Five tips to make your evaluation more influential

21 | Value for Money

recommendations that are hard to accomplish, but are necessary to achieve step improvements. Sharing Evaluation Knowledge: With the Right People, At the Right Time, In the Right Language. The answer to better resourcing the outreach phase of an evaluation does not lie in simply spending a lot of money. Outreach activities can be expensive, and even more so if they add to the cost without commensurately increasing the value of an evaluation.

Evaluations will increase their chance of influence when they: ▪ Differentiate diverse

audiences, recognize their diverse information needs and learning styles;

▪ Package messages, products and channels to meet these needs to make evaluation evidence and lessons easily accessible; and

▪ Inspire and understand the incentives of behavior change.

Diverse Audiences, Diverse Information Needs. Life is much easier if you have a single audience, because the entire evaluation can be targeted to their questions, level of technical detail, etc. But, this is often not the case. At the World Bank Group,

for example, IEG’s audiences range from members of executive boards, to senior and mid management, and operational staff and counterparts. Each have a stake in the evaluation, each an interest in learning something about the program under evaluation. For instance, policy makers in board rooms, ministries, and senior management want to know whether strategically a policy, strategy, or program is the right thing to do. Does it contribute to the overall goals of the institution; does it generate the returns – financially and in

22 | Value for Money

development outcomes – commensurate with resources invested? At the operational levels, the interest is greater in the nuts and bolts of what has worked, how and why. And, affected communities want to get feedback, especially if they were consulted during the evaluation, on issues they raised and what will be done about them. Making Messages Accessible. This description of different audiences and their needs already conveys that there cannot be a “one size fits all”. At community level, conveying messages will require different means – including translation into local languages, focus on details that matter to the community, and delivery through means accessible to people – than for instance a board room of a multilateral. For the latter, the interest is in short, focused, strategic messages, while operational people would want more detail, both evidence that proves the points made in the evaluation, and to learn from the details. And for wider audiences and different means of communication—including social media—there is a need to capture and convey messages in different ways. This is why IEG has invested in developing a range of prototypes from infographics to videos. Incentivizing Behavior Change. Ultimately, evaluation recommendations are about changing what we do and how. Traditionally, evaluators have thought of this step as tracking progress in the implementation of recommendations. The World Bank Group’s management action record is an example of a sophisticated system that tracks recommendations over a four-year period. It is an important step in taking stock of what has changed since the evaluation was completed. In future, we will raise the discussion to reflect on systemic, institutional changes as a result of evaluations conducted under each of our Strategic Engagement

Areas. This is one step in the direction of recognizing that incentivizing behavior change requires a different kind of engagement than one of (solely) compliance checking. Enhancing Value Creation. In most organizations where I have worked, it would have been impossible to finance all of these outreach activities. At IEG, we are fortunate to be resourced in ways that we can invest in making our evaluation work more accessible to users, hence facilitate learning and influence change. These investments enhance the value of the work done. It does require making strategic choices about which investments in outreach will generate the highest returns.

Since publishing this blog, IEG has

experimented with deep dives into the

management action records for groups of

evaluations. For instance,

recommendations of evaluations related

to environmental sustainability and their

follow-up actions were reviewed in the

2017 Results and Performance report.

The deep dive differentiated the types

of recommendations IEG had made over

time, and illustrated which kind had

received more attention than others.

The analysis raised strategic issues, which

were discussed at the World Bank Group’s

Executive Board and with

senior leadership.

Among others, IEG was tasked to work

with World Bank Group managements

to revisit and update the system for

ensuring recommendations lead to

learning and behavior change.

23 | Value for Money

Thanks to the many of you who viewed,

shared, and commented on our Value for

money of evaluation blog series blogs.

Really an extraordinary participation.

Now, here comes the challenge! How do we estimate, or even calculate the value for money (VfM) from any given evaluation? In previous blogs, I mapped out the points at which value and costs are incurred. I suggested a number of ways to increase value and reduce the costs of an evaluation. These suggestions are valuable in their own right, as they improve evaluation practice. But, they do not yet spell out how value or cost can be estimated. Many of the costs can be calculated, for instance the direct cost of an evaluation, or the expense associated with outreach activities. But, there are indirect costs that

often do not get taken into account. They are noticed in particular when evaluations uncover controversial issues and create the need for difficult discussions. These meetings are often experienced by stakeholders—people whose programs are evaluated as much as evaluators—as an additional cost. Other costs, often hard to calculate, are those of reputational risk—feared or actual—that can arise if there is no evaluation at all, or if an evaluation is inaccurate. The value, however, is harder to calculate. An evaluation that is not timely is a missed opportunity to make better informed choices. But, estimating the value would have to compare the choices made with and without evaluation evidence, and assume the evaluation would have influenced decision-making.

“The effects on an evaluation can often

be seen only after years and depend on

how soon recommendations get

implemented, how quickly they result in

behavior change, and how well that

translates into changes in policies,

strategies, or projects.”

JULY 19, 2016

How do we estimate, or even calculate the value for money (VfM) from any given evaluation?

24 | Value for Money

In addition, choices along the life-cycle of evaluations can increase the VfM equation at some point but reduce it at others. The most obvious case is when cost-savings lead to evaluation findings that are not robust. But, just spending more money is not the answer—wise choices about evaluation design (objectives, scope, and questions) are. Ultimately, the value of evaluation needs to be estimated through the changes that occur as a result of it. If, for instance, a project normally generates a return of “100” but as a result of an evaluation introduces changes that increase its return to “120” the value of the evaluation would be “20” assuming all changes introduced in the project were motivated by the evaluation. Typically, however, the value increase of a project is neither calculated nor attributed to an evaluation as easily as in this example for a number of reasons, including the following:

1. Quantification. Economic and financial returns on an investment include some things that are more easily quantifiable—for instance, reduced vehicle operating costs as a result of an improved road surface—but others are less so. Establishing a link between an evaluation finding, its recommendation, and the implementation thereof to a financial return is even harder for larger programs or sector strategies: the returns might be higher, especially when systemic problems get resolved, but harder to calculate;

2. Attribution. Evaluations are similar to

other knowledge work: they stimulate discussion and help rethink

approaches. But, many factors play a role when it comes to making course-corrections or decisions about new policies and strategies. Suggesting that all outcomes of these changes are attributable to evaluation would not be justifiable.

3. Time Lag. The effects on an

evaluation can often be seen only after years and depend on how soon recommendations get implemented, how quickly they result in behavior change, and how well that translates into changes in policies, strategies, or projects.

None of these points should stop us from attempting to assess the influence, impact and value for money or our evaluation work. Please share your examples, if you have done so.

In commenting on this blog, ITAD

(itad.com) shared a link to one of their

publications:

The Value of Evaluation: Tools for

Budgeting and Valuing Evaluations,

August 2016,

http://www.itad.com/knowledge-

products/value-evaluation-tools-

budgeting-valuing-evaluations/.

The publication offers a number of

techniques for valuing evaluations at

various stages. It also discusses the

application and challenges of each of

these methods.

25 | Value for Money

Biography

Caroline Heider is the Director General of the Independent Evaluation Group at the World Bank (IEG), a position she has held since 2011. Ms. Heider has dedicated the past 30 years of her career to evaluating the work of development and humanitarian organizations, transforming findings into lessons, and promoting innovative ways for institutions to apply the knowledge derived from evaluations toward accelerating development effectiveness. As a senior leader, Ms. Heider has a proven track record in leading change, strengthening institutions, and building evaluation capacity

through testing and trying new methods to get to better evidence and greater insights. She has first-hand experience evaluating policies and programs in over 30 countries around the world.

Ms. Heider is a leading voice in the international evaluation community. She is a lifetime member of the International Development Evaluation Association (IDEAS) and a member of the American Evaluation Association. She chaired the Global Evaluation Advisory Committee of UN Women for the first years of its existence. In the past, she has been a member of the Australasian Evaluation Society and served a two-year term as vice-chair of the UN Evaluation Group.

Before IEG, Ms. Heider headed the Office of Evaluation at the World Food Program. She has also held leading positions in the evaluation offices of the Asian Development Bank and several UN agencies, including the International Fund for Agriculture Development, the UN Development Program, and UN Industrial Development Organization.