Assessments and the WB Assessment Manual

Linn Hammergren

November 10, 2010

Purpose

• Discuss why assessments are important (and what happens when you don’t do one)

• Discuss assessment inputs• Provide ideas on methodologies and

alternative strategies for doing them• Outline a strategy for conducting an

assessment

Why assessments are important

• Because obviously you can’t resolve a problem until you identify it and its causes

• Because the obvious answers to the two (implicit) questions are often wrong

• Because an assessment allows you to understand the broader context for an operation and the likely impediments to its success

• Because an assessment provides a format for discussing strategies with the client

• Because an assessment gives you baseline data and thus a means of monitoring progress

Why assessments are not done or are incomplete

• They cost money• They take time• Bank thinks identification mission is sufficient• The client thinks they are unnecessary• They reveal inconvenient details• However, taking into account how long and how much it

takes to “prepare a PAD suitable for Board approval” the investments in a good assessment are marginal.

• However inconvenient the truth of the matter, it is better to find it early on than to have it emerge post-approval

Examples of findings an assessment might discover

• Belief (Mexico) that primary problem with money claim cases is delay to judgment– Finding: only 20% of cases reach judgment– Finding: longer delays for enforcement

• Belief (Mexico and many LAC countries) that judges are “overloaded with work”– Finding: annual filings often quite low and even when not, most cases

go “nowhere” and so require little work.– Finding: much of caseload is unattended backlog

• Belief (Colombia, Honduras) that new CrPC improves case processing:– Finding: most cases reaching verdicts are in flagranti– Finding: major crimes less attended.

• Belief (Ghana and other African countries) that fast-track and commercial courts are more efficient, but studies show low caseloads and delays

Inputs to assessments• All prior diagnostics and studies – or at least the most important

ones. Many assessments ignore prior work (especially if not by the WB), which is a waste of resources

• Discussions with the principal client (often the high court judges) as to their perception of the “problem” to be resolved

• Discussions with other stakeholders to get their views, which are often different

• Review of any existing statistical material, legal framework, history of prior and on-going programs

• Fieldwork of various kinds (more on this later)• In short, a good part of the assessment can occur before the

fieldwork and helps focus the latter. This saves time and resources.

A participatory approach?

• This is controversial although Bank seems to endorse• The client is usually the host government or the judiciary, and both will have

a self-interested cut on what is wrong and how it should be fixed.• Participation is important for client buy-in, but in some cases may prove

overly restrictive – e.g. when client attends and thus censors all meetings • If the client is open to new approaches and inconvenient truths, it can be

very helpful, but if this is not the case, it can limit findings.• Still, to the extent possible:

– Clients should be involved in assessment design– They should be invited to provide any material they consider essential

(and what they consider essential is also revealing)– They should be involved in at least part of fieldwork– They should be informed about progress and problems encountered– They should be included in analysis and discussion of conclusions

Why not do a completely participatory assessment?

• Even in the best of cases this is not likely to produce the best results because:– Local appreciations are local– Participants lack the big picture (as regards their

“problems” and the likely solutions)– Participatory planning encourages Christmas Tree

projects – an ornament for everyone.• External experts are important because they have

broader experience, and if good can suggest a variety of

different interpretations and solutions.• The wisdom of crowds is rarely technically based

A problem-oriented approach using institutional, political economic analysis

• Can be as participatory as possible – techniques are not incompatible

• To maximize utility for project design – if there are no problems, an operation is unnecessary

• Problems can be defined by combination of CMU input, literature review, primary client views, broad-based interviews and team’s observations

• Initial problem definition tested and refined in later stages

• Seek explanations (and remedies) through institutional and political economic analysis

What is a problem?• Defined here as shortcoming in desired outputs and

downstream impacts, but there is often confusion on this point

• When judges, police are asked about problems, often mention salaries, equipment, overwork

• Not the problems of interest here – best treated as potential causes

• Challenge – to get “system operators” to redefine “their” problems in terms of output/impacts. Can ask about consequences of “overwork” for clients, for example

Institutional analysis looks at:

• Relevant state and non-state organizations and actors

• Formal laws, structure, mandates, and procedures; human and other resources; territorial distribution and other factors affecting use and outputs

• Real practices and informal rules• Incentive structures – will have to be inferred• How the above interact to shape outputs and

create problems or performance gaps

Political Economy• Bank has shied away from this (or mislabeled institutional analysis

to this end)• Political economy analysis (PEA) looks at power and access to

resources, and at intra- and extra-sector interactions to affect both– Institutional analysis focuses on how organizational

arrangements influence behavior– Political economy explains why – who benefits?

• PEA may be omitted from public report, but not from assessment. – If politicians need compliant judges or are in cahoots with

criminals, this will affect the program– Example – IADB citizen security projects in Caribbean neglect

ties between politicians and “dons” (gang lords).• Bank need not engage in politics but should recognize its effects

Assessment stages and content – the Manual

• Preparation -- scope, resources, timing, and team• S 1: Preliminary background• S 2: Preliminary problem identification• S 3: Field collection of information – institutional analysis• S 4: Summary overview and analysis• S 5: Comprehensive analysis of partial findings,

prioritization of problems/causes, identification of areas for interventions

• S 6: Recommendations for Program

Preparation: Initial Considerations• Initial definition of assessment scope by TTL, CMU, client

– Often targeted at institution or problem– Team may need to negotiate some modification, especially as regards

incorporating more context (Court’s problem is not always the Court)

• Resources and timing --- usually decided (too?) early– Both often underestimated, but there is never enough– 2-4 weeks in field (with more time for writing) absolute minimum, but superficiality

can be combated with experienced team (knows country, knows methods) , access to documents, or with very limited focus, small country, etc.

– Even with large team, more time, lack of focus/organization can produce ambiguous results

• Assessment team: skills needed; identification of local/foreign experts; decisions as to inclusion of Bank and/or client members

• Norms, ground rules, roles: set by team and TTL• Design and selection of methodologies

Stage 1: Preliminary background• Get team up to speed on country and overall characteristics, history,

broader development problems• Less necessary if use local experts or internationals with past

experience• However, will be needed as part of assessment document, and also

to ensure common team understanding of where they are starting• May be useful to have team meetings to discuss, especially if prior

exposure to country varies greatly• Example: team for Honduras labor assessment (1 local, 1

international with country experience, 3 substantive experts with no country exposure or language skills) – would have been helpful to start here

Stage 2: Preliminary problem identification

• Start with working hypotheses based on: – What people say (from interviews, written documents,

international assessments, opinion polls)– Team’s initial observations – this is where international

experience counts as a point of reference• Over course of evaluation, problem definition may change – as may

early guesses as to underlying causes• Even if findings contradict conventional beliefs, latter should be

mentioned in study. What people think is important and will affect what can be recommended

• Also useful to tie problems to groups mentioning them – not everyone agrees on what is wrong or why (which is why reviews should never be limited to judges and government authorities).

Stage 3: Data collection

• Some can be done prior to fieldwork– using written or internet documents (including laws, budgets, statistics) and out-of-country interviews

• Rest will inevitably be a mix of interviews, observation, surveys, statistical reviews, and whatever else the team can muster

• Aim – to do more than repeat what informants tell you; test their versions with other data sources and also explore alternative hypotheses

Methodology (Fieldwork and other Data collection)

• Assessments use multiple methodologies; several described in manual• Combine hard with soft data sources• Soft data (from interviews, observation) provide insights; hard data

(statistics, surveys) used to check and to generate hypotheses directly• No data source is perfect, and selection depends on what is available, time

and financing, what is being sought• Also depends on team skills – do they know statistics/surveys? Are they

experienced with specific topics?• Assessment may suggest further studies • Most studies will only focus on formal systems – this can be a problem

where most citizens uses informal ones.• Even if someone on the team has experience, legal-pluralism does not lend

itself to a fast assessment. (i.e. Countries with 100 or more local traditions?)• But, legal pluralism may not be amenable to international assistance either.

S. 3 Softer data sources• Document review

– WB often overlooks prior studies, even its own – team should be provided with this material

– Laws may not reveal reality, but they are important insofar as they impede change

• Interviews --- will continue to be a major source of information– Important to triangulate – multiple points of view– Structured (set questions) can be quantified, but richer information comes from

unstructured. May thus combine two.– Try to transcend client’s definition of interview list – need to reach further.

• Observation – a widely ignored source of information– However requires skill and most probably wider experience– A good court administrator or IT expert doesn’t need to know the language to

see problems– However, a local and an international specialist may achieve much more

S. 3: Harder sources• Surveys – might be considered “soft” as they are

quantified opinions, but in any event, seem to carry more weight

• Aggregate statistics -- often not available and in any case, usually limited

• Sampling – costly but revealing. – However, samplers should be carefully monitored to ensure 1)

their sample is good and 2) their data are also good– Can always treat as “illustrative” if quality is in doubt.– In large, heterogeneous countries, what holds in one region may

not in another

S 3, Harder data, some further observations

• Common lack of good statistics is a finding in itself – and may lead to recommendations on improvements

• Most systems only permit estimations of workloads, congestion, and possibly resolution rates; for delays, case profiles, differential outcomes need something more.

• Surveys, sampling, and related methods usually take more time and money than allowed in assessment. Can take “illustrative” short cuts– Quick review of random sample of filings from courts “of interest”– DB style survey of users on delays, outcomes – but their

perceptions often don’t coincide with numbers (case of Mexico)– If courts agree, can use staff for some data collection, but

supervision required and best to keep simple– See Dilbert (next slide) on iffy data

A little more on hard data

S. 3: Less conventional data sources and analysis

• Evaluating quality of judicial decisions – many have tried, but no consensus on how to do it. Also all methods are labor-intensive, time-consuming and ultimately, highly subjective

• Focus and study groups – also quasi-soft, but unlike surveys allow more detail and discussion; useful for explaining statistics and sampling results.

• Incentive analysis – more an art than a science and also the result of other methods (focus groups, statistics, surveys). Some interesting, if hypothetical conclusions:– Mexico: lawyers are less interested in curbing delays than clients– Generally – delay does not bother judges until someone (their bosses)

tracks it– Generally – counting judgments encourages “cherry picking” and

manipulation of numbers

S 4: Summary Overview and analysis• Conducted on the basis of initial information, before

fieldwork is completed• Allows fieldwork to focus on further testing of hypotheses

-- may call for surveys, focus groups, or sampling• Except in longer-term studies may not be possible, but

will at least generate areas “for further study.”• If done, should orient collection of harder data – for

example, extent of and causes of delays; hypotheses as to who uses courts and with what results

S. 5: Comprehensive analysis• Initial list:

– Performance and impact problems– Interrelated causes of each– Initial ideas on solutions

• Prioritization:– Of most significant or most urgent problems (can include

perceptions as well)– Of causes, by susceptibility to change and broader impact– Potential points of program entry with short and long-term

results, difficulty of addressing, sources of support and opposition to change.

S. 6: Recommendations• Assessment TOR may not require, but is the obvious last

step• This and prior analysis should involve local stakeholders,

as if they are not in agreement, nothing will happen• May be useful to hold workshops or seminars to discuss

tentative findings, conclusions, and recommendations• Results and report content may consequently be

negotiated• Quick fixes (tactical measures) and longer term reforms

(overall strategy)

Assessment Do’s and Don’ts• There are no recipes, but have a plan, tentative organization, and

role assignments even if they will change along the way• Go beyond “descriptive studies” –an overview of the parts and how

they fit together is useful, but assessments must do more than this and when they don’t, counterparts complain. Save the details for the Annexes?

• Likewise, assessment should not be a legal review – often a problem of local experts with little research experience

• Do try for a multi-disciplinary team – to spread the perspectives• Mix internationals and locals – they each have their use• Beware, however, of expert bias – trainers inevitably recommend

training (even if not a priority) and IT experts will want systems; generalists with some substantive knowledge may be preferable, saving the dedicated experts for the identification mission.

Assessment Do’s and Don’ts (cont.)• Triangulate sources and mix methodologies to test “what everyone knows”• Avoid sending multiple team members to interview one person

(simultaneously or sequentially) – a waste of time and can intimidate informants. “Round robin interviews” may work better with groups.

• Take informant claims seriously, but not as the unvarnished truth• Protect informants and verify/respect their wish to be identified or not.• Gather all relevant information, vetting for inclusion later – don’t censor

along the way. (Consider multiple versions of report?)• Encourage team meetings on findings and intermediate conclusions – this

can reveal gaps, contradictions, and synergies.• Be prepared to leave things out of the report (not only for sensitivity but for

limited relevance). Data collection is a process, but the report does not need to repeat all the steps.

• Recognize areas for more research later, noting why they are important and how they might be explored.

Assessment is an iterative process• The first cut is only that• Assessments should include provisions

and recommendations for further studies• However, assessments should be used –

and in many cases they remain on the shelves

• Can also be useful in motivating clients, other local participants to find out more.