Transferring knowledge into policy and the role of Wikiprogress

Kate Scrivens

Wikiprogress Project Manager

OECD Statistics Directorate,

Monitoring Progress & Well-

being Division

KNOWLEDGE & POLICY: OVERVIEW

Introduction

Theories of knowledge use in policy

Case study 1: Programme for International Student

Assessment (PISA)

Case study 2: Climate change measurement

The policy impact of well-being/QoL measurement

The role of Wikiprogress

INTRODUCTION

The OECD

The OECD’s work on well-being measurement

Wikiprogress

What do we mean by “knowledge”?

Well-being and Quality-of-Life measures as policy

knowledge

INTRO THEORY PISA CLIMATE WELL-BEING WIKIPROGRESS

WHAT IS THE OECD?

The Organisation for Economic Cooperation and

Development:

Currently 34 member countries

Historical origins: administering the Marshall Plan,

fostering economic growth

Today’s mission: “Better Policies for Better Lives”

A think-and-do tank, semi-autonomous, functioning

through peer review and consensus-building


THE OECD AND WELL-BEING

2004: the 1st OECD World Forum on Statistics, Knowledge and Policy

The well-being and ‘Beyond GDP’ movement: diverse and international

2009: the Stiglitz Report

2011: the launch of the Better Life Initiative

BLI, How’s Life?, Subjective Wellbeing Guidelines, etc.

High-Level Expert Group on Well-being (‘Stiglitz 2’), 5th World Forum to be held in 2015


WIKIPROGRESS

An online platform for sharing knowledge and data

A community portal for anyone interested in well-being/progress measurement and policy

Hosted by OECD but open-source, independent

Purpose:

Provide resources and information for researchers, practitioners and policy-makers

Raise public awareness about well-being/progress issues


WHAT IS ‘KNOWLEDGE’? Knowledge = information + understanding

Where does our information come from?

Scientific research

Policy research

Official statistics (surveys, administrative data, etc.)

Other data (Open government data, ‘big’ data, etc.)

Media

Experience

Other people (friends, experts, advisors, conference presenters, etc.)

We cannot assume that knowledge and information are objective and free of bias


THEORIES OF KNOWLEDGE USE IN POLICY

Models of research use in policy

Types of policy outcomes resulting from knowledge use

Political economy theories and the role of knowledge:

Ideas

Institutions

Interests

Communicating for policy: complexity vs accessibility


MODELS OF RESEARCH USE IN POLICY

The policy process is rarely linear and clear-cut

6 models of research use (Weiss, 1979, “The many

meanings of research utilisation”)

Problem-solving model

Knowledge-driven model

Interactive model

Political model

Tactical model

Enlightenment model


PROBLEM-SOLVING MODEL Based on the idea that decisions are made in a linear

series of steps

A problem is defined

Different courses of action are identified

Pros and cons of each option are clearly set out

The option offering the best solution is selected

The policy is implemented

The outcome is evaluated

Represents most people’s ideal of evidence-based policy-making

BUT ignores the role of politics in the policy process


KNOWLEDGE-DRIVEN MODEL Sees academic research as the driving force of policy

development

While academics may assume this to be norm, it is rare

in real-life policy

Academic knowledge in the social sciences is usually

not undisputed or authoritative enough to lead directly

to implementation


INTERACTIVE MODEL Recognises the plurality of policy actors

Input into the policy process can come from many

different sources (both formal and informal

Practitioners, aides, researchers, lobbyists, journalists,

friends, etc.

Difficult to trace the role of specific kinds of

‘knowledge’ in influencing policy

Each actor provides a piece of the puzzle, but it is rare

that any one source alone can provide an answer


POLITICAL MODEL Political interests around an issue predetermine the

position that decision makers take

Discussion around an issue has gone on for so long

that opinions are hardened and new evidence is

unlikely to change these positions

Information is used selectively as ammunition to

support previously held positions


TACTICAL MODEL Information used for purposes unrelated to the actual

content it presents

Examples

A politician using statistics in a speech only to give the

impression of evidence-based thinking

A mayor using a report as an excuse to delay city

planning reform (“we need to study the issue more”)


ENLIGHTENMENT MODEL Research and data are diffused through various

channels Conference presentations, news reports, discussions with colleagues,

indicator reports, scientific evidence etc.

Generalisations emerge which provide decision makers with the means of framing issues in a certain way

Makes policy makers more sensitive to emerging issues and helps to define problems

Weiss argued that this was the most common way that social science research contributes to policy making


TACTICAL MODEL Information used for purposes unrelated to the actual

content it presents

Examples

A politician using statistics in a speech only to give the

impression of evidence-based thinking

A mayor using a report as an excuse to delay city

planning reform (“we need to study the issue more”)


ROLE OF INDICATORS The BRAINPOoL project (following on from POINT), adapted

these general models to the role of indicators.

Instrumental use, where indicators are seen as objective information tools to improve policy making, solving problems and consciously influencing decisions

Conceptual use, which sees the prime value of the indicator to be more intangible, influencing how policy makers define a problem

Political use, which consists of 3 sub-categories Strategic use: used to bolster a position/decision that has already been

taken Tactical use: where decisions are postponed or avoided with the excuse

that data is being awaited Symbolic use: where indicators are used only to convey a message or

present an image


TYPES OF POLICY OUTCOMES FROM

KNOWLEDGE USE Content change

Change in the content of policy and/or resources

allocated

Agenda setting change

Change in policymakers’ priorities and where attention

is drawn to new issues that were previously ignored

Framing shift

Changes the way that policymakers understand a

problem or the possible responses to it


TYPES OF POLICY OUTCOMES (CONTD.) Procedural change

Changes how policy itself is made by procedural/ institutional

change that leads to new actors or new evidence being part of

the process of decision-making

Behavioural change

Change in the way in which policy is implemented

From Sumner et al. 2009, “Making science of influencing:

Assessing the impact of development research”


POLITICAL ECONOMY THEORIES & KNOWLEDGE Mainstream economic theory provides a framework of

understanding based on idealised models of human

and market behaviour

Theories of political economy aim to take into account

the ways policies are developed and implemented in

the real world

Political economy theory tends to cover three broad

areas (the 3 ‘I’s):

Ideas

Institutions

Interests


IDEAS Ideas, in this context, refer to our implicit and explicit

assumptions about how the world works

No matter how objective and scientific we think we are being, we tend to operate within certain cognitive and normativeframeworks

Ideas are powerful – and they are most powerful when we are not even aware of their influence: many things we take to be “fact” are socially constructed

Dominant ideas tend to change gradually over time, but they can also change quickly after a shock

Ideas (norms, theories, shared understandings, worldviews, paradigms, ideologies) both constrain and enable collective action


INTERESTS In all societies there are different groups who have different

interests in the outcomes of policies (interest groups)

Groups that have more resources, more power and are

better-organised tend to have the most influence on policy

outcomes

This is especially the case where the costs of a policy are

concentrated within one group, and the benefits are

distributed across society (or vice versa)

However, politicians need votes as well as support from

elites: where widespread public opinion can be mobilised,

this can provide the political will to override powerful lobbies

For some, interests and ideas are two sides of the same coin


INSTITUTIONS Institutions are the rules and laws that govern society, and

the bodies and processes which implement them

Institutions shape the way in which decisions are made (e.g.

consensus, majority vote, expert-driven, evidence-based,

negotiation) and policy action is implemented

Institutions matter a great deal for determining the impact of

knowledge on policy

Institutional change tends to be slow


COMMUNICATING STATISTICS FOR POLICY Statistical products generally have three types of audience

1. Statisticians/researchers

2. Policy makers/analysts

3. Public (media, civil society, non-experts

Each of these audiences requires a different amount and

complexity of data


Public

Policy

Tech

Public

Policy

Technical

Size of

audience

Amount/

complexity

of data

KEY POINTS FROM THE THEORY There is no one way in which knowledge can be

transferred into policy

Multiple actors from different sectors of society can play a role in the policy process

The impact of knowledge on policy can take many forms, and is often diffuse, subtle, accumulative and hard to trace

Scientific knowledge, no matter how convincing and clear-cut it is, will be filtered through the dominant ideas, interests and institutions of a society, which will shape its impact on policy

The communication of knowledge for policy needs to be tailored to different purposes and audiences


WHAT HELPS INDICATORS IMPACT POLICY? According to the BRAINPOoL project, four “success” factors

can be identified for policy indicators

Salience for decision makers

Indicators are successful when they can demonstrate real relevance for policy or strategy

Salience for a broader audience

Producing a simple, attractive message that links to a meaningful concept while avoiding certain ‘taboo’ words and concepts is key to impacting a broad audience

Credibility

Data quality and the appearance of neutrality are important

Relationships

Relationship-building with the intended audience to strengthen the legitimacy of indicators is key


CASE STUDY 1: PISA

The Programme for International Student Assessment

(PISA) – description and objectives

Implementation of PISA

Policy impact

Lessons from PISA on how knowledge gets transferred

into policy


PISA: DESCRIPTION AND OBJECTIVES The Programme for International Student Assessment

(PISA) is an international survey which aims to evaluate education systems worldwide to inform national education policy

More than 70 economies have participated in the assessment to date

The survey is held every three years, with the main focus rotating between three core subjects: literacy, mathematics and science

The most recently published results are from the assessment in 2012, in which over half a million students took part


IMPLEMENTATION OF PISA PISA was created in 1997, with the first round of

assessments taking place in 2000

Each round of PISA operates on a 5-year cycle (3 years’ preparation, 2 years’ reporting)

The development and implementation of PISA is based on collaboration and consensus with a wide range of experts and practitioners

The institutional structure of PISA is geared towards high levels of cooperation and exchange between the OECD and national experts and policy makers in participating countries


POLICY IMPACT OF PISA Large-scale reform has been implemented in several

countries as a direct result of PISA “shock”

E.g. Germany, Switzerland, Denmark, Japan

The influence of PISA has increased over time and it is

now widely recognised as the most important

international evaluation of education systems

It is a good example of instrumental use of indicators

that has led to a wide range of policy outcomes

(content, procedural, agenda-setting, framing,

behavioural change)


LESSONS FROM THE PISA EXPERIENCE Why has PISA been so immensely influential?

It meets all 4 BRAINPOoL criteria for successful indicators (saliency for policy and public, credibility and relationships)

It covers a well-defined problem (education) whose importance is widely-agreed across different groups (few opposing interests)

It represents the dominant idea that education is important for economic productivity

The institutions and tools to implement education reform are well-established


CASE STUDY 2: CLIMATE CHANGE

The evolution of knowledge on climate change

The policy impact of climate change evidence

Lessons from the example of climate change


KNOWLEDGE ON CLIMATE CHANGE Scientific concern over the global warming and the role of

human activity began in the 1950s

By the 1990s, a scientific consensus had emerged around the need to act

1988: the Intergovernmental Panel on Climate Change (IPCC) was formed

1992: United Nations Framework Convention on Climate Change

1997 (enforced 2005): Kyoto Protocol (first legally binding agreement)


KNOWLEDGE ON CLIMATE CHANGE (CONTD.) By 2000s, volume of evidence on the need to take

policy action to combat climate change is

overwhelming

2007: Stern Report on Economics of Climate Change

(estimated 1% of world GDP needed to be invested)

Nov 2014: Latest IPCC Report released:

“Without additional mitigation efforts beyond those in place today, and

even with adaptation, warming by the end of the 21st century will lead to

high to very high risk of severe, widespread, and irreversible impacts

globally”


POLICY IMPACT OF CLIMATE SCIENCE While the scientific knowledge is unanimous, policy

action has lagged behind

Countries’ commitments have tended not to be

matched by action

The Kyoto Protocol took years of negotiations to be

enforced, and the US has refused to ratify the treaty

Implementation of policies to reduce carbon emissions

are extremely uneven across countries (tending to be

stronger in EU than non-EU countries)


LESSONS FROM CLIMATE CHANGE Climate change is a complex issue, which touches

upon a wide range of economic, social, technological and institutional challenges

While the message and evidence is clear, the political and institutional barriers to action are significant and numerous: Strongly entrenched opposing interests promoting a

contradictory narrative National institutions are not well-equipped to deal with cross-

cutting issues with high costs of action and diffuse, delayed outcomes

Lack of political consensus around the best way to act on climate change (few established policy tools and levers)


LESSONS FROM CLIMATE CHANGE (CONTD.)

Public knowledge of climate change is a powerful

determinant of climate change policy adaption

Countries in which the public is aware of the causes of

climate change are significantly more likely to adopt

climate change mitigation policies


WELL-BEING AND QOL MEASUREMENT The OECD and well-being measurement

The policy impact of OECD well-being work

Lessons learned

Towards the development of tools for using well-being

measures in policy


GDP “measures neither wit nor our courage, neither our wisdom nor our learning, neither our compassion nor our devotion to our country, it measures everything in short, except that which makes life worthwhile”

GDP is a flawed measure of people’s actual well-being, and

therefore an incomplete measure of a nation’s true progress

THE FIRST SOCIAL INDICATORS MOVEMENT The current well-being/progress/QoL/Beyond GDP

movement is not new, but the political momentum is

In the 1960s and 1970s, a movement to develop social indicators emerged in North America and Europe

NASA project in the 1960s

1970: OECD Social Indicators programme established (measuring health, education, employment and quality of working life, time and leisure, command over goods & services, physical environment, social environment, and personal safety)

1974: launch of Social Indicators Research

However, by the mid 1980s, most social indicator work disbanded or significantly reduced due to lack of policy support


THE OECD WORLD FORUMS In 2004, the OECD held the 1st World Forum on

Statistics, Knowledge and Policy in Palermo.

Purpose was to bring together a wide range of societal actors – not just statisticians – to talk about statistics

Rather than being a technical conference, it looked at the fundamental role of statistics to create knowledge for policy, and to underpin democracy: “we measure what we value”

It recognised that statistics are not politically neutral, and that deciding the best way to measure ‘progress’ requires an inclusive approach


WHY ARE STATISTICS ‘POLITICAL’? Counting requires decisions about categorising, about

what (or whom) to include and exclude

Measuring any phenomenon implicitly creates norms about what is too little, too much, or just right

Numbers can be ambiguous, and leave room for political struggles over interpretation

Numbers are used to tell stories, such as stories of decline

Numbers create the impression that very complex phenomena are simple, countable and precisely defined

Numbers can create political communities out of people who share some trait that has been counted

Numbers, by seeming to be so precise, help bolster the authority of those who count


THE ISTANBUL DECLARATION The 2nd World Forum, held in Istanbul in 2007 led to the

Istanbul Declaration, where signatories pledged to:

Encourage communities to consider for themselves what “progress” means in the 21st century

Share best practices on the measurement of societal progress and increase the awareness of the need to do so using sound and reliable methodologies

Stimulate international debate, based on solid statistical data and indicators, on both global issues of societal progress and comparisons of such progress

Produce a broader, shared, public understanding of changing conditions

Advocate appropriate investment in building statistical capacity, especially in developing countries


FROM DISCUSSION TO MEASUREMENT OECD continues to hold the World Forums every 2-3

years (5th World Forum to be held in Mexico in 2015)

However, there has been a shift in OECD work from

agenda-setting/ framing to developing policy measures

Stiglitz report in 2009 gave impetus and laid out the

roadmap for methodological work

Greater emphasis on household perspective and patterns of

distribution (inequalities)

Recognition that well-being is multidimensional

Need for both subjective and objective measures

Focus on both current well-being and sustainability


OECD WELL-BEING FRAMEWORK


OECD WORK ON WELL-BEING


Plus: country reports, working papers,

Income Distribution Database,

Inclusive Growth Index, Big Data

analysis, Handbooks and Guidelines,

High Level Group, Forums

POLICY IMPACT OF OECD WELL-BEING WORK There is now a widespread consensus amongst

statisticians, policy makers, practitioners and researchers that policy needs to go ‘beyond GDP’

Many national leaders have made public declarations of the importance of measuring well-being (UK, Germany, United States)

There are many national and sub-national initiatives to develop broader indicators of progress (UK, Italy, Israel) - based on public consultation

Some national treasury offices have established well-being frameworks to assess national expenditure (New Zealand, Canada, Australia)


POLICY IMPACT OF OECD WELL-BEING WORK The OECD is only one of many actors working on these

issues, but widely seen as a leader (credible)

Continued to advocate for well-being measures after crisis

While there is much conceptual and symbolic use of well-being measures, examples of instrumental use is rare

To date, the policy impact of well-being measures has been closest to the enlightenment and interactive models - diffuse, with many actors, impacting primarily through agenda-setting and framing e.g multiple dimensions, importance of SWB


LESSONS LEARNED Lessons learned from the first social indicators

movement – why did it fail?

Social issues became overshadowed by economic concerns as the global economic situation worsened

Ideological shift towards conservatism and reduced government

Limited usefulness of social indicators to policy makers

Lack of unifying theoretical framework to compare with economic theory

Lack of common metric for aggregation comparable to money in economics


LESSONS LEARNED Many of these factors apply now too (crisis, reduced

government, lack of unifying theory) – what is

different?

Crisis has strengthened sense that traditional

measures were missing something important, and that

a disconnect exists between people’s experience and

aggregate economic measures

This has supported the narrative (idea) that change is

needed


GDP VS LIFE SATISFACTION IN EGYPT BEFORE 2011

Traditional measures did not predict the Arab Spring,

whereas indicators of the share of people “thriving” in the

country plummeted

LESSONS LEARNED But what are the barriers to instrumental use?

Ideas are changing but institutional change is needed

too

Well-being and quality of life are complex and multi-

dimensional

Unlike education, there are no ministries of well-being

Policy discussions on well-being/QoL tend to exist in

parallel with the economic policy process



Source: BRAINPOoL final report, 2014

LESSONS LEARNED GDP predominates policy discussions because

The prevailing policy belief is that economic growth is

the key goal for policy

It allows for the combination of multiple factors into a

single, powerful number that immediately

communicates its meaning

It is embedded in a widely-accepted theoretical

framework, with well-established institutions and policy

tools to support policy making focused on the

maximisation of aggregate productivity


LESSONS LEARNED How do we promote the goal of well-being as the

ultimate objective of policy-making?

Ideas are starting to change but this will be a slow

process

We need to continue to communicate knowledge on the

importance of well-being, tailored to different audiences

(including the general public)

We need to appeal to people’s interests, with a convincing

narrative for change (e.g. to those still focused on economic

growth, focus on the economic benefits of well-being policy)

We need to reform the way economics is taught to take into

account new thinking on well-being and inequalities


LESSONS LEARNED We need new institutional structures that are better

geared towards the delivery of well-being policy

Realising the potential of well-being data to design

better policies “requires change to policy-making

processes at both national and local level, incuding

breaking down silos between policy areas and

reforming the process for allocating budgets”

(Report of UK All-Parliamentary Group on Wellbeing Economics)

We need new frameworks and tools for well-being

policy analysis


POLICY TOOLS USING WELL-BEING MEASURES Composite indices

Inclusive growth index, ecological footprint, human development index

Accessibility of information over comprehensiveness/complexity

Need to be embedded in an analytical framework

Multiple criteria analysis

Designed to deal with decisions with multiple objectives – not all of which monetisable or easy to aggregate

Weighting and trade-offs are made explicit

Designed to assess different options, usually includes stakeholder deliberation


POLICY TOOLS USING WELL-BEING MEASURES Subjective well-being (SWB) measures

There is an increasing recognition of the need to measure and maximise SWB

SWB can be understood in a number of ways (affect, life satisfaction, eudomonia)

There are a number of ways in which SWB measurement can be used in policy (principally focused on life satisfaction)

To inform policy design, monitoring and evaluation as the dependent variable (e.g. work, health, city planning)

To provide input into other forms of analysis such as cost-benefit analysis

OECD has played a major role in increasing the profile of SWB measures in policy


QUESTIONS What do you think policy makers should be maximising

(e.g. SWB, capabilities, economic growth)?

How would you persuade a government leader of the

importance of taking into account well-being

measures?

How would you persuade the general public of the

importance of promoting policy for well-being?

How do you think knowledge on well-being is used in

policy in your country?


WIKIPROGRESS Wikiprogress is an online space for sharing knowledge on

well-being and progress

It serves 3 main functions

A knowledge base providing resources for anyone interested in ‘progress’ (articles, reports, data, organisations)

A community portal for researchers, practitioners and engaged non-experts (events, blogs, discussions)

Communicating to a general public on the importance of measuring well-being for better policies

We would like your help!

Wikiprogress depends on volunteer contributions

We would like your help in creating a new portal for young people


THANK YOU!

Wikiprogress.org

[email protected]

@katescrivens

mailto:[email protected]

Transferring knowledge into policy and the role of Wikiprogress

Government & Nonprofit

Transcript of Transferring knowledge into policy and the role of Wikiprogress