Towards more inclusive STI indicators
Ismael Ràfols1,2 and Jordi Molas-Gallart1, Richard Woolley1 and Diego Chavarro2
1Ingenio (CSIC-UPV), Universitat Politècnica de València2 SPRU (Science Policy Research Unit), University of Sussex, Brighton
Gent, OECD Blue Sky 2016
Re-shaping design and use for inclusion
• Indicators may be harming – via exclusion Current indicators are only (partially) appropriate for some types of science and
innovation. Biases against and potential suppression of creative and valuable types of
research (agro-, health,…). Threat to diversity. May count as positive harmful forms of innovation (Soete)
• Not only more, but other types of indicators needed Making visible “other” contributions and other types of research and innovation
(e.g. action research, co-creation) Enhancing visualisation for “opening up” perspectives rather than facilitating
“closing down”
• Understanding uses of indicators Better embedding in policy, evaluation, context Indicators used to pluralise debate, as tools for interpretation and
deliberation, not a substitute for judgement (Barré)
Uses of indicators: Pressing demands of research policy, management and evaluation --- Can indicators help?
Yes, indicators can help make decisions… Reduce time and costs Increase transparency and sense of objectivity Reduce complexity, accessible to managersbut do they lead to the “right” decisions?
Evaluation gap (Wouters):“discrepancy between [the] criteria [implicit in indicators] and the social and
economic functions of science”
*Academia – “excellence” *Innovation – economic “growth”
Missions not well covered: agriculture, public health, defence,development, social inclusion,…
Often related to marginalised / “neglected” populations
Space of problems
Space of research
Researchwell illuminatedby indicators
Problems, research, indicators and marginalisation
Problems, research, indicators and marginalisation
Multiple types of space:
STI Peripheries:research spaces notwell capturedby indicators
Researchwell illuminatedby indicators
Cognitive: SSH, engineering
Linguistic: non-English
Sectoral: low-tech, agriculture, creative ind.
Social: gender, minorities
Geographical: regional, “South”
Coverage: Knowledge production on rice by country
India China Japan USA0.0%
5.0%
10.0%
15.0%
20.0%
25.0%
30.0%2000-2009
CABI
Scopus
WoS
Publ
icati
ons
Rafols, Ciarli and Chavarro (in preparation)
Coverage: Knowledge production on rice by topic
Rafols, Ciarli and Chavarro (in preparation)
2002-2012
Streetlight effect in indicators: mistaking light with “problems”
Space of problems
Space of research
Researchwell illuminatedby indicators
Streetlight effect in indicators: mistaking light with “problems”
Space of problems
Space of research
Space of problemsSpace of research
Space of problems
Hypothesis: reduced indicator coverage may contract research space
Space of research
Space of STIindicators
The societal needs dealt by research that is under the streetlight effect, will be better rewarded.
Reduced diversity ofresearch efforts...
…reduced coverageof societal needs
Space of problems
Demands for expanding role of science in society…
Space of research
Space of STIindicators
Space of problems
Demands for expanding role of science in society…
Space of research
Space of STIindicators
Space of problems
…may require expanded sets of data and indicators
Space of research
Space of STIindicators
Space of problems
…may require expanded sets of data and indicators
Space of research
Space of STIindicators
• Comprehensive databases
• Better normalisationby context (e.g. citation bias)
• Indicators of process or interactions
• New data sources
narrow
broad
closing-down opening-up
range of appraisal's inputs(issues, perspectives, scenarios, methods)
effect of appraisal ‘outputs’ on decision-making
Leach et al. 2010
Appraisal: Broadening out vs. Opening up
narrow
broad
closing-down opening-up
range of appraisal’s inputs(issues, perspectives, scenarios, methods)
effect of appraisal ‘outputs’ on decision-making
Most conventionalS&T indicators??
Appraisal: Broadening out vs. Opening up
narrow
broad
closing-down opening-up
range of appraisal’s inputs(issues, perspectives, scenarios, methods)
effect of appraisal ‘outputs’ on decision-making
ConventionalS&T indicators??
Broadening out
Incorporation multiple analytical dimensions:
New analytical inputs: media, blogsphere.
Appraisal: Broadening out vs. Opening up
narrow
broad
closing-down opening-up
range of appraisals inputs(issues, perspectives, scenarios, methods)
effect of appraisal ‘outputs’ on decision-making
Journal rankings
University rankings Unitary measuresthat may be translated into prescriptionComposite
European InnovationScoreboard
Appraisal: Broadening out vs. Opening up
narrow
broad
closing-down opening-up
range of appraisals inputs(issues, perspectives, scenarios, methods)
effect of appraisal ‘outputs’ on decision-making
Appraisal: Broadening out vs. Opening up
ConventionalS&T Indicators??
opening-up
Making explicit underlying conceptualisations and contextcreating tools to facilitate exploration
NOT about the uniquely best methodOr about the unitary best explanationOr the single best prediction
From S&T indicators for justification and disciplining…… towards S&T indicators as tools for deliberation
Model 2: Plural and conditionalExploring diverse choices Facilitating options/choices in landscapes
Model 1: Unique and prescriptiveProposing “best choices”Rankings -- ranking list of preferences
An agenda for more inclusive metrics• Inclusiveness in the inputs
Broadening out: Create more diverse indicators – Indicators of open science, RRI, hidden, social innovation– Improve representation of SSH scholarship, languages other than
English, the “South”,…
• Inclusiveness in the outputs Opening up: develop toolkits that allow exploration of choices. New
ways of presenting indicators– Multi-ranking tools– Interactive visualisations
• Inclusiveness in the policy process (??) Develop new social processes on use of indicators
– STI indicators as tools for interpretation and deliberation (R. Barré)
Backup slides
24
Rice Varieties Classic Genetics
TransgenicsMol. Biology Genomics
PestsPlant protection
Weeds Plant protection
Plant nutrition
Production & socioeconomic issues
ConsumptionH. nutrition, food techs
What are the “options” in rice research?
Not a fined-grain perspective
25
US, 2000-12
How research priorities differ by country
26
India 2000-12
How research priorities differ by country
27
Thailand 2000-12
How research priorities differ by country
S&T Indicators Conference – First week
21st Science and Technology Indicators ConferenceValència, 14-16 September 2016, www.sti2016.org
The problem:• Some STI activities and their outcome are badly captured by indicators• Disincentive to carry out certain types of activity at certain spaces• Empirically: problem more acute at peripheral spaces in science • Potential effects: Reduction of diversity –particularly locally relevant?
Bias in citation within a subdiscipline
Good
Average
Bad
Van Eck, Waltman et al. (2013)
More basic
More applied
Clinical neurologyIs basic always better than applied?
Citations: not stable to changes in classification and granularity (Zitt et al., 2005; Adams et al., 2008).
Measures of “scientific excellence”
ISSTI SPRU MIoIR Imperial WBS LBS0
0.5
1
1.5
2
2.5
3
3.5
4AB
S Ra
nk
ISSTI SPRU MIoIR Imperial WBS LBS0
1
2
3
4
Cita
tions
/pub
Jo
urna
l-fiel
d N
orm
alis
ed
ISSTI SPRU MIoIR Imperial WBS LBS0
0.05
0.1
0.15
0.2
Cita
tions
/pub
Citi
ng-p
aper
Nor
mal
ised
Which one is more meaningful??
ISSTI SPRU MIoIR Imperial WBS LBS0
1
2
3
4
Jour
nal I
mpa
ct F
acto
r
Rafols et al. (2012, Research Policy)
Top Related