Activity 1 Discrepancies in ranking

Activity 1 Discrepancies in ranking

• Read the following 2007 indicator sets in small groups

1. the World Bank’s Worldwide Corruption Indicators on ‘Control of Corruption’

2. Transparency International’s Corruption Perceptions Index (CPI)

3. the Global Integrity Index.

Module 5 Composite and original indicators

• Note the position of China, Thailand and India on each index. What do you observe?

• 10 minutes


0

1

2

3

World Bank - Controlof Corruption

TransparencyInternational - CPI

Global Integrity Index

Corruption indices for China: Variations in rankings

China

Thailand

India

Corruption in China in 2007: 3 indices; 3 stories


Corruption in China over time (2004-06) Two indices; two stories

According to the World Bank, China is doing better in 2006 than it was in 2004

According to TI, China is doing worse in 2006 than it was in 2004

World Bank - Control of Corruption (China) 2004-2006

0

5

10

15

20

25

30

35

40

2004 2005 2006

Transparency International - CPI (China) 2004-2006

3.1

3.15

3.2

3.25

3.3

3.35

3.4

3.45

2004 2005 2006

Activity 2 Disaggregation exercise

Review the worksheets distributed and discuss the following

with your group:• What is each index measuring? • What types of indicators make up the

composite score for each index? • What types of data make up each component

indicator?


• Note feedback on flipcharts

• 10 minutes


0

1

2

3

World Bank - Controlof Corruption

TransparencyInternational - CPI

Global Integrity Index

Corruption indices for China: Variations in rankings

China

Thailand

India

Corruption in public sector as perceived by

“experts”

Corruption in public & private sector (+ some indicators at household level) as perceived

by “experts” + opinion polls (incl. NGO experts)

Existence, effectiveness and citizen access to anti-corruption mechanisms,

assessed by national experts


Tip 1: Ignore an index’s label and dig underneath

• Different scores on 2 different indices does not mean that one index is right and the other is wrong.

• Only that the 2 indices are assessing different concepts

• Quick tip:1) Know what want to measure 2) Ignore an index’s “label” 3) Unpack component questions / indicators4) Which index really measures what you want to

measure? 5) Or do you need to generate your own

research?

Activity 1 Discrepancies in ranking

Module 5 Composite and original indicatorsActivity 1 Discrepancies in ranking

Selected sources

(out of 14 in 2007)

Subject asked Respondents Coverage

World Economic

Forum (WEF)

Undocumented extra payments or bribes connected

with various government

functions

Senior business leaders;

domestic and international companies

125 countries

Bertelsmann Transformation Index (BTI)

The government’s capacity to punish and

contain corruption

Network of local experts inside

and outside the organization

125 countries

Economist Intelligence Unit (EIU)

The misuse of public office for

private (or political party) gain

Expert staff assessment

166 countries

What data makes up the Corruption Perceptions Index?

Module 5 Composite and original indicatorsAnnual variations in methodology

Transparency International - CPI (China) 2004-2006

3.1

3.15

3.2

3.25

3.3

3.35

3.4

3.45

2004 2005 2006

Rank: 71th

Rank: 78th

Rank: 70th

Out of 145

Out of 158

Out of 163

16 surveys – out of 18

14 surveys – out of 16

9 surveys– out of 12


• What are the advantages and disadvantages of composite indicators?

• What are the alternatives to indices?

• Small group work• 5 minutes

Activity 4 Composite and original indicators


• What are the main advantages of composite indicators?

• What are the main advantages of original indicators?

• What are the alternatives to quantitative assessments?

Activity 4 Composite and original indicators

Module 5 Composite and original indicatorsMini-lecture Composite and original indicators

Composite indicators of corruption combine existing indicators into a single measure. This aggregate measure normally ranks a country’s performance on a multi-country index.

• International efforts to measure corruption began to take off in the late 1990s with the use of composite indices.

Module 5 Composite and original indicatorsMini-lecture Composite and original indicatorsTransparency International’s groundbreaking Corruption Perceptions Index (CPI)

• released in 1995• purpose = a warning bell• Ranks 180 countries by their perceived levels of corruption. •Scale = 0 (highly corrupt) - 10 (highly clean) •Rankings determined by expert assessments and opinion surveys which combine 13 sources covering a period of 2 years (2008 survey).

Module 5 Composite and original indicatorsMini-lecture Composite and original indicators World Bank Institute’s Control of Corruption Indicator= 1 of 6 Worldwide Governance Indicators (WGI).

• produces indicators for 212 countries over the period of 1996 to 2007 • data from 35 separate sources, 25 of which were used for the Control of Corruption indicator


Why aggregate?

• to reduce measurement error, which can occur when survey responses do not reflect the reality of the wider population.

• to widen country coverage, which is necessary for generating broad cross-country comparisons.

Module 5 Composite and original indicatorsMini-lecture Composite and original indicatorsCriticisms of composites

• reliance on third-party data reduces the conceptual clarity of what is actually being measured.

• composite indicators, “have no explicit definition, but instead are defined implicitly by what goes into them”. • limited by the available data

• variation in data sources helps to explain discrepancies in rankings of a country between different indices


Criticisms of composites

• wide country coverage inconsistent data coverage, particularly for the poorest countries which are often not included in many underlying data sources.

•makes comparison difficult, both between countries and over time, because some data sources are discontinued and new ones become available.

• lack of actionability for local stakeholders planning anti-corruption reforms.


• Multi-country composite indicators useful for providing signals where a country’s performance is slipping or improving

•have been used primarily by international donors and investors for making decisions about aid and capital allocations

• not designed for informing policy decisions and reforms


original indicators of corruption: indicators that are based on originally generated rather than “third party” data.

• part of “second generation” of assessment tools which tend to focus on single countries, and also tend to use locally generated data.

•major strength = produce highly detailed and contextualized information that can be disaggregated to reveal local nuances (e.g. patterns between corruption incidence and income levels.)


• based on national surveys of experts and the public•can be single-country in focus, but many repeat methods across countries for comparison:

•E.g. Open Budget Index, UNECA’s Africa Governance Indicator, the Global Integrity Index, TI’s Global Corruption Barometer

Weaknesses:• a high degree of contextualization makes cross-country comparisons difficult. •comparability over time necessitates consistent repetition of methods which can be prohibitively costly.


• How are corruption indicators used in your country? (e.g. for policy making in a specific sector)

• What makes an indicator useful for each of these purposes? In your experience, are some indicators more or less useful than others?

Activity 5


• Actionable indicators: – allow disaggregation to pinpoint bottlenecks within

public administration systems– indicate need for corrective action– are based on specific and localized data– measure things that are under the control of

policymakers

• Policymakers say that global composite indicators are not locally specific enough to tell what needs to be fixed– But some indices are drawn on national sources, and can

be useful indicator sources (e.g. TI’s Global Corruption Barometer)

– Nonetheless, actionable indicators require locally generated original data, tailored to national priorities

Activity 5 The need for actionable indicators


• You have been distributed a handout containing indicators drawn from the 3 global corruption indices we looked at earlier:– Corruption Perceptions Index – Worldwide Corruption Indicators on ‘Control of

Corruption’ – the Global Integrity Index.

• Which, if any, of the indicators on the worksheet is actionable?

Activity 6 Identifying actionable indicators


Actionable Non-actionable

Can citizens access the asset disclosure records of members of the national legislature? (Global Integrity)• In law, citizens can access the asset disclosure records of members of the national legislature.• In practice, citizens can access legislative asset disclosure records within a reasonable time period.

Does the state provide direct public subsidies to parties for their non-electoral activities? (CRINIS Project)

In practice, do the parties keep accounting books and a registry of their assets? (Answers should relate to the national party organisation) (CRINIS Project)

How often do firms make extra payment to influence the content of new legislation? (Business Enterprise Environment Survey)

How many judges and magistrates do you think are involved in corruption? (Afrobarometer)

On average, what % of total annual sales do firms pay in unofficial payments to public officials? (Business Enterprise Environment Survey)

To what extent can the government successfully contain corruption? (Bertelsmann Transformation Index)


• Do you find the concept of actionable indicators useful? To what extent is it already a feature of (anti-) corruption monitoring in your country?

• “Not everything that can be counted counts.” – Albert Einstein

• What are the pitfalls of actionable indicators?

Activity 7 Discussion


• Actionable does not mean action-worthy

– E.g. Measuring whether a country has an independent anti-corruption commission when there is no guarantee it will reduce corruption

– E.g. measuring the speed of judicial proceedings when not clear increasing the speed will ensure justice is done

– Risks: measuring things because they are easily measurable, leading to “reform illusion”; attribution of results to an AC intervention can only be estimated, amidst a range of other factors

Activity 7 Action-worthy indicators

Module 5 Composite and original indicatorsActivity 8 Determining what is action-worthy

•How does a country determine what is action-worthy?

10 minutes to discuss

Module 5 Composite and original indicatorsActivity 8 Determining what is action-worthy

• Assessing performance of interventions only makes sense if these are first deemed appropriate for the particular country context

•Prior research can identify what reforms to prioritize, and ones that risk further entrenching corrupt systems e.g. political economy studies

•Sustained resourcing is required for systematic data collection over time (baselines and periodic monitoring to establish progress and regress)

Activity 1 Discrepancies in ranking

Documents

Transcript of Activity 1 Discrepancies in ranking