Post on 21-May-2020
The European Commission’s scienceand knowledge service
Joint Research Centre
Tools for developing and auditing composite index frameworks and
scoreboardsDániel Vértesy, PhD
Research Fellow | European Commission, Joint Research Centre, Competence Centre on Composite Indicators and Scoreboards (CC-COIN), Ispra, Italy
UN Expert Group Meeting on Methodologies of Regional Integration Indexes 26/09/2018, Addis Ababa (ET)
3
1 IntroductionThe JRC’s work on Composite Indicators and Scoreboards
5
• information explosion in terms of the 3V’s (Volume, Variety, Velocity), but
• human ability to analyse information will increase only marginally
± 2 is the average number of objects an average human can hold in working memory
[Miller's Law]
Data Revolution
7
6
Making sense of Data, Information & Knowledge
• Diffusion of evidence-based policies
• Today’s information deluge
• Fundamental role for the JRC role in managing knowledge in support of EU policy making
7
Knowledge Centres
Disaster Risk Management Knowledge Centre
Knowledge Centre for Territorial Policies
Knowledge Centre on Migration and Demography
Knowledge Centre on Bioeconomy
Knowledge Centre for Food Fraud and Quality
Provide relevant dataknowledge intelligence
in a specific policy field
8
Competence Centres
Competence Centre on Composite Indicators and Scoreboards
Competence Centre on Text Mining and Analysis
Competence Centre on Microeconomic Evaluation
Competence Centre on Modelling
Competence Centre on Foresight
Provide scientific expertisestate-of-art methods tools
to inform policies in a any policy field
9
Competence Centre COIN
Provide
scientific expertisestate-of-art methodstools
to inform policies in any policy field
The European Commission’s Competence Centre on Composite Indicators and Scoreboards
Our aimsServe as a referencepoint for EU policymakers on monitoringmulti-dimensionalphenomena
Maintain the globalrecognition in auditingcomposite indicatorsand scoreboards
10
Competence Centre COIN
What we doSupport to EU Services (> 40 indices & scoreboards)E.g., Social Scoreboard for the European Pillar of Social Rights (26/04/2017; CAB Juncker), Cultural and Creative Cities Monitor (06/07/2017; Commissioner Navracsics)
Collaborations with international partners (> 100 audits)E.g., Global Innovation Index (INSEAD, WIPO), Commitment to Reducing Inequality Index (Oxfam), Financial Secrecy Index (Tax Justice Network)
Methodology Jointly with the OECD quality control framework (conceptual coherence, multivariate analysis, sensitivity analysis, multi-criteria decision analysis, statistics and policy)
Training & Community of Practice on Composite Indicators and Scoreboards 5-9 November 2018, Ispra, Italy (16th Training – and 3rd CoP)
11
Community of Practice on Composite Indicators and Scoreboards
Discussion Topics• Fairness, Inequality and Poverty• Social Rights & Social Trust• Resilience & Risk Management• Sustainable Development & Sustainable
Governance
• Education & Skills• Ageing societies & Migration• Innovation & Digital Transformation• Competitiveness & Attractiveness
▶ How should indices and scoreboards be used for policy and what should be avoided?▶ How can indices and scoreboards be communicated to the wider public?
Ispra, IT8-9 Nov 2018
12
[COIN Data Hub, inventory in preparation]
150•Banks’ contribution to EU Single Resolution FundDG FISMA
•Index for Risk Management (INFORM)DG ECHO
•Consumer Conditions & MarketsDG JUST
•Europe 2020 IndexDG REGIO
•Small Business Act Principles DG GROW
•Innovation Output IndicatorDG RTD
•Cultural and Creative Cities Monitor DG JRC
•ASEM Sustainable Connectivity Index (Oct. 2018)EEAS
•Systemic Risk Heat MapsECB
composite indicators and scoreboards
developed or used by the EC
50% of the EC indices/scoreboards are developed
with JRC contribution
Ever increasing number of monitoring tools
13
Example 1: Cultural and Creative Cities Monitor
https://composite-indicators.jrc.ec.europa.eu/cultural-creative-cities-monitorhttps://composite-indicators.jrc.ec.europa.eu/cultural-creative-cities-monitor
14
Example 1: Cultural and Creative Cities Monitor
15
Example 2: JRC support to the European Central Bank
ECB – Systemic Risk Country Heat MapL L L L L L L L L L L L L L L L L L L L L L L L L LL L L L L L L . L L L L L L L L L L L L L
L L L H L L L L L H L L H LH H H H H H H H H H H H H H
H H H H H H H . H H H H H H H HH H H H H H H H H H . H H H H H H H H H H H H
H H H H H H . H H H HH H H H H H H . H H H HH H H H H H H H H H . H H H H H H H H H
. . . . H H . . . . . . . . . H . .H H H . H H H H H H
L H L L . L L H L H L H L HH H H H H H H H H . H H H H H H H H H H
L L L L . L L L L L L L L L L L LH H H H . H H H H .
H H H H H H H H H HH H H H H H H H H H H H H H
H H H H H H . H H H H H H .
L L L L L L L L L L L L L L L L L L L L L L L L L LL L L L L L L L L L L L L L L L L L . L L
L L L H L L H L L H L L L LH H H H H H H H H H H H H H
H H H H H H H H H H H H . H H HH H H H H H H H H H H H H H H H H H H . H H H
H . H H H H H H H H HH H H H H H H H H . H H
H H H H H H H H H H H H H H H H H . H H. . . . . . . . H . H . H . . . . .
H H H H H H H H . HH L H H H H L L L L . L L L
H H H H H H H H H H H H H H H H . H H HL L L L L L L L L L L L L . L L L
. H H H H H . H H HH H H H H H H H H H
H H H H H H H H H H H H H HH H H H . H H H H H . H H H
Low systemic risk ------------------------------> High systemic risk
16
A compass towards better living and working conditions in the EU
• 3 dimensions & 12 areas• 14 headline & 28 secondary indicators• 93 indicators in total (gender/age)
https://composite-indicators.jrc.ec.europa.eu/social-scoreboard/
Example 3: Social Scoreboard for the European Pillar of Social Rights
17
2 Developingcomposite index frameworks:What, why, how to measure?
18
Step 10. Presentation & dissemination
Step 9. Association with other variables
Step 8. Back to the indicators
Step 7. Robustness & sensitivity
Step 6. Weighting & aggregation
Step 5. Normalisation of data
Step 4. Multivariate analysis
Step 3. Data treatment
Step 2. Selection of indicators
Step 1. Developing the framework
Finally endorsed (after 2 rounds of consultation) by the OECD high level statistical committee in March 2008
The 10 Steps Guide for Composite Indicators
19
Developing an index...
A technical challengeA technical challenge A conceptual challengeA conceptual challenge
Indicators as neutral metrics
=> Descriptive
Trigger action by comparison
=> normative
20
The power of numbers
Numbers as targets: strong advocacy tool- See the case of the Millennium Development Goals (MDGs)
“While quantification is the key strength of global goals, it alsoinvolves simplification, reification and abstraction, which havefar-reaching implications for redefining priorities”Setting MDG goals/targets influenced policy priorities and hadnormative effects on development discourses;All MDG goals/targets “led to unintended consequences indiverting attention from other important objectives and reshapingdevelopment thinking”
(Fukuda-Parr, S., Yamin, A.E., Greenstein, J., 2014)
21
Indicators for Policy
Surge of indicators of policy• Internationalization & globalization (need for comparison)• Calls for accountability (the “audit explosion” – Power, 1994)
• legitimize interventions (ex ante) & demonstrate success (ex post)
• Indicators offer de-personalized accounts on the state of the world to allow for rational decisions to change the world for the better (Rottenburg and Merry, 2015)
=> Aim to provide unbiased, comparable information on a single phenomenon of interest for different units
BUT: quantification involves simplification=> Normative choices
22
Indicators are shaped by policy needs & discourseIndicators, in turn, influence policy discourse
Indicators are embedded in a socio-political context• Indicators & indicator frameworks are value laden; reflect policy discourse
(Godin, 2002)
• Indicator developers & users better be aware of consequences:• The “agora model” (Barré, 2001, 2010)
indicators are “debating devices” – it’s the process that matters!• Honest broker (Pielke, 2007)• “Indicators in the wild” – open up scope of data & expertise, process, framing
of problems (Rafols, 2017)
23
Composite Indicators
Advantages:
• Support decision makers by summarizing complex or multi-dimensional issues
• Provide the “big picture”, highlight common trends
• Measure a latent phenomenon that is not directly measureable
• Attract public interest by benchmarking
Pitfalls:
• Offer misleading, non-robust policy messages if they are poorly constructed or misinterpreted
• May invite politicians to draw simplistic policy conclusions
• Easier to “manipulate” than individual indicators; the selection of sub-indicators and weights could be the target of political challenge
Important to ensure their qualityImportant to ensure their quality
The development process is crucial: it helps• Better understand how a system functions• Identify latent dimensions, redundancies or
trade-offs between components
The development process is crucial: it helps• Better understand how a system functions• Identify latent dimensions, redundancies or
trade-offs between components
24
The Quality of Composite Indicators
“Composite indicators sit between analysis and advocacy, but quality discriminates the plausible from the rhetorical” (Saltelli, 2007)“Composite indicators sit between analysis and advocacy, but quality discriminates the plausible from the rhetorical” (Saltelli, 2007)
Advocacy
Analysis
Quality
Codified and continuously refined methodology• The OECD-JRC Handbook (JRC-OECD, 2008)• Audits – robustness and sensitivity
analyses (i.e. Saisana et al, 2011; Paruolo et al, 2013)
25
3 Auditingcomposite index frameworks
26
What is the purpose of auditing indicators?
• Test for statistical coherence• Are the conceptual and statistical frameworks aligned?• Is the structure statistically justified? • Are all indicators sufficiently driving index scores?
(accounting for information loss)
• Test for robustness and sensitivity • How do modelling assumptions influence results?
27
100
• Human Development Index UNDP
• 3 child-poverty indicesUNICEF
• Multidimensional Poverty Assessment ToolUN IFAD
• Global Innovation Index WIPO
• ICT Development IndexITU
• Financial Secrecy IndexTJN
requests for JRC audits by international partners (UN, UNICEF, Tax Network, WEF, INSEAD, WIPO, UN-IFAD, FAO, Harvard U., Yale U., Columbia U., Cornell U.)
The EC JRC is an in-house science centre providing technical advice to a variety of projects throughout Europe. As part of that mission it is also an expert reviewer of project methodologies, says SiddharthBannerjee, Nesta project lead for the EDCI. It’s important to be rigorous about methodology. It’s how bias can sneak in when not expected,” he adds. “The EC JRC may not be well known outside certain circles, but it is the ultimate indicator of what a composite index looks like.”
Examples: Requests for JRC audits
28
• Assess if few indicators dominate the framework
(rindicator,index>0.95)
• Assess if indicators behave as “noise” in the framework
(-0.3<rindicator,index<0.3)
• Assess if indicators are negativelyrelated to the index
(rindicator,index<-0.3)
Statistical coherence
Corr. rSens itivity Indices (r^2) Interpretation
FRAMEWORK CONDITIONSHum an resources1.1.1 New doctorate graduates 0.691 0.477 driver1.1.2 Population completed tertiary education 0.561 0.315 w eak driver1.1.3 Lifelong learning 0.844 0.713 Main driverAttractive research system s1.2.1 International sc ientif ic co-publications 0.889 0.791 Main driver1.2.2 Scientif ic publications among top 10% most cited 0.898 0.806 Main driver1.2.3 Foreign doctorate students 0.831 0.691 driverInnovation-friendly environment1.3.1 Broadband penetration 0.508 0.258 w eak driver1.3.2 Opportunity-driven entrepreneurship 0.732 0.535 driverINVESTMENTSFinance and support2.1.1 R&D expenditure in the public sector 0.731 0.534 driver2.1.2 Venture capital investments 0.507 0.257 w eak driverFirm investments2.2.1 R&D expenditure in the business sector 0.809 0.655 driver2.2.2 Non-R&D innovation expenditure -0.065 0.004 silent2.2.3 Enterprises providing ICT training 0.777 0.603 driver
Indicators vs. SII scores
SII 2017: Corr. Indicators vs. Aggregate Index
29
The correlation table says it all…
The European Innovation Scoreboard 2017The European Innovation Scoreboard 2017
30 JRC-COIN © | Step 7: Statistical Coherence
• Ideally, an indicator correlates more with its own dimension than with others
• See e.g., Sustainable Society Index, 2012
Are the indicators in the right dimension?
31 JRC-COIN © | Step 7: Statistical Coherence
• Similarly, sub-pillars are ideally relatively more correlated to their own pillar than others
• See e.g. Global Innovation Index, 2017
• Positive, strong correlation between Sub-pillar and pillar
Are the indicators in the right dimension?
32
• …• Assess if indicators statistically fit
better under different dimensions than those in the framework
• Assess if dimensions should be merged or split
• Assess if bias has been introduced in the index (e.g., due to population size, population density, GDP)
Statistical coherence
33
• Identify the sources of uncertainty in the index development
• Assess the impact of the uncertainties to the index scores or ranks
complement scores/ranks with confidence intervals
Robustness & Sensitivity
Including/excluding variables Normalisation
Missing dataWeights
Aggregation
Country 1
10
20
30
40
50
60
Country 2 Country 3
34
• Identify the sources of uncertainty in the index development
• Assess the impact of the uncertainties to the index scores or ranks
complement scores/ranks with confidence intervals
Robustness & Sensitivity
35
How important are underlying assumptions?• Identify key modelling choices• Test their impact on the final composite
scores and rankings
Assumptions influence outcome
(Source: Hardeman and Vertesy, 2015)
36
• Identify which uncertainties are more crucial in determining the final classification
• Explain why certain countries notably improve or deteriorate their relative position given the assumptions
• Robustness ≠ Quality
Robustness & Sensitivity
37
COIN helps to ensure that composite indicators are …
developed sensibly and used responsibly.
‘Sensible development’ of a composite indicator implies a quality control process based on both conceptual and statistical considerations.
‘Responsible use’ calls for care in drawing conclusions and recommendations without taking into account the conceptual context in which composite indicators were developed.
Indices are powerful advocacy tools
ec.europa.eu/jrc
@EU_ScienceHub
EU Science Hub - Joint Research Centre
Joint Research Centre
EU Science Hub
Stay in touch!
ThanksQuestions?You can find me at daniel.vertesy@ec.europa.eu
Further info: https://composite-indicators.jrc.ec.europa.eu/
Contributions to this presentation by Michaela Saisana and JRC colleagues in the Composite Indicators and Scoreboards (COIN) Team are gratefully acknowledged.
Disclaimer: The views expressed are purely those of the writer and may not in any circumstance be regarded as stating an official position of the European Commission
40
Becker, W., M. Saisana, P. Paruolo, and I. Vandecasteele. 2017. ‘Weights and Importance in Composite Indicators: Closing the Gap. Ecological Indicators 80: 12–22.Cohen, A., Saisana, M., 2014, Quantifying the Qualitative: Eliciting Expert Input to Develop the Multidimensional Poverty Assessment Tool, Journal of Development Studies 50(1): 35-50.Davis, K. E., Fisher, A., Kingsbury, B. and Merry, S.E. (eds.) (2012) Governance by Indicators: Global Power Through Quantification and Rankings. Oxford: Oxford University Press.Ferretti, F., Guimaraes Pereira, A., Hardeman, S. & Vertesy, D. (2017) “Research Excellence Indicators: time to re-imagine the ‘making of’?” Paper under revision for Science and Public PolicyOECD/JRC, 2008, Handbook on Constructing Composite Indicators. Methodology and user Guide, OECD Publishing, ISBN 978-92-64-04345-9. Paruolo P., Saisana M., Saltelli A., 2013, Ratings and Rankings: voodoo or science?. J Royal Statistical Society A 176(3), 609-634.Power, M. (1994). The audit explosion (No. 7). Demos Paper No. 7. Rottenburg, R., Merry, S.E., Park, S-J. and Mugler, J. (eds.) (2015) The World of Indicators: The Making of Governmental Knowledge through Quantification. Cambridge: Cambridge University PressSaltelli, A. (2007). Composite indicators between analysis and advocacy. Social Indicators Research, 81(1), 65-77.Saisana, M., and Saltelli, A., 2011, Rankings and Ratings: Instructions for use, Hague Journal on the Rule of Law 3(2), 247-268.
References and related reading
41
Saisana M., D’Hombres B., Saltelli A., 2011, Rickety Numbers: Volatility of university rankings and policy implications. Research Policy 40, 165–177.Saisana, M., Saltelli, A., 2014, JRC Statistical Audit of the WJP Rule of Law Index® 2014 (p.188-198). In the World Justice Project Rule of Law Index® Washington, D.C.: The World Justice Project.Saisana, M., Weziak-Bialowolska, D., 2013, JRC Statistical Audit on the Environment and Gender Index (p.143-153), in IUCN The Environment and Gender Index (EGI) 2013 Pilot. Washington, D.C.: IUCN.Saisana, M., Saltelli A., 2012, Corruption Perceptions Index 2012. Statistical Assessment, EUR 25623, European Commission, JRC-IPSC, Italy.Saisana M., 2010, ELLI-Index: a sound measure for lifelong learning in the EU, EUR 24529, European Commission, JRC-IPSC, Italy.Saisana M., Saltelli A., 2010, The Multidimensional Poverty Assessment Tool (MPAT): Robustness issues and Critical assessment, EUR 24310, European Commission, JRC-IPSC, Italy.Saisana M., 2008, The 2007 Composite Learning Index: Robustness Issues and Critical Assessment, EUR 23274, European Commission, JRC-IPSC, Italy.Saisana M., D’Hombres B., 2008, Higher Education Rankings: Robustness Issues and Critical Assessment, EUR 23487, European Commission, JRC-IPSC, Italy.Vértesy, D. 2016. ‘A Critical Assessment of Quality and Validity of Composite Indicators of Innovation’. Paper presented at the OECD Blue Sky III Forum on Science and Innovation Indicators,19–21 September 2016, Ghent, Belgium.
References and related reading