THE FUTURE OF BIBLIOMETRICS - SCONUL · 2019-01-16 · • 1960 – “The white heat of the...
Transcript of THE FUTURE OF BIBLIOMETRICS - SCONUL · 2019-01-16 · • 1960 – “The white heat of the...
THE FUTURE OF BIBLIOMETRICS UNDER THE MICROSCOPE
Philip Purnell SCONUL, Southampton – July, 2015
BACKGROUND & REF
INTELLIGENT INFORMATION FOR PROFESSIONALS 3
THE POWER OF CITED REFERENCES
2003 2007
2011
2008
1965 1998
2001
1930
Times Cited
Related Records
1986
2008 1984
1972
Cited References
“ … association of ideas …”
2002
CITATION BEHAVIOUR VARIES BETWEEN FIELDS
Where did all this come from? • 1955
– Eugene Garfield’s Science paper on “Citation Indexes for Science”
• 1963 – Science Citation Index (ISI >Thomson >Thomson Reuters)
• 1972 – U.S. National Science Foundation initiates Science Indicators (later
Science & Engineering Indicators), including publication and citation data
• 1980s – Uptake of science indicators in Europe; research by SPRU, CWTS,
Hungarian Academy, as well as ISI
• 1992 – Advisory Board for the Research Councils works with ISI on
National Science Indicators to benchmark UK
• 2004 – Elsevier’s Scopus and Google Scholar are launched
7
Where did research evaluation come from? • 1960
– “The white heat of the technological revolution”, Harold Wilson
• 1970 – “For the scientists, the party is over”, Shirley Williams
• 1980 – UGC/ABRC consensus on selectivity – 1986, Research Selectivity Exercise – 1989, Research Assessment Exercise
• 1990 – Evolution of research management and administration – 1992-2008, RAE - the standard model, evolving grades – 2014, Research Excellence Framework
8
We find bibliometric impact and peer review are coherent across institutions (chemistry data)
0.4
0.6
0.8
1
1.2
1.4
1.6
1.8
0 1 2 3 4 5 6
Aver
age
cita
tion
impa
ct fo
r uni
vers
ity 1
996-
2000
Relative citation impact of articles submitted for RAE2001
Grade 5*
Grade 5
Grade 4
Grade 3a
Grade 3b
Spearman r = 0.57, P<0.001
Ratio mapped/NSI = 1.93
Data and analysis: Evidence Thomson Reuters
Responses to evaluation Research trajectory changed from mid-1980s
1.1
1.2
1.3
1.4
1.5
1981 1985 1989 1993 1997 2001 2005 2009
Rela
tive
impa
ct o
f UK
rese
arch
pub
licat
ions
UK citation impact
5 yr moving av'ge
Arrows indicate RAE years, e.g. 2001 and 2008
Data and analysis: Evidence Thomson Reuters
Responses to evaluation Behavioural games - Goodhart’s Law
RAE1996 Science Engineering Social sciences Humanities and arts
Outputs % Outputs % Outputs % Outputs % Books and chapters 5,013 5.8 2,405 8.1 16,185 35.1 22,635 44.4
Conference proceedings 2,657 3.1 9,117 30.8 3,202 6.9 2,133 4.2
Journal articles 77,037 89.8 16,951 57.3 22,575 49.0 15,135 29.7
Other 1,104 1.3 1,122 3.8 4,154 9.0 11,128 21.8
RAE2001
Books and chapters 1,953 2.5 1,438 5.4 12,972 28.6 25,217 46.5
Conference proceedings 751 0.9 3,944 14.9 857 1.9 1,619 3.0
Journal articles 76,182 95.8 20,657 78.1 29,449 65.0 17,074 31.5
Other 618 0.8 408 1.5 2,008 4.4 10,345 19.1
RAE2008
Books and chapters 1,048 1.2 216 1.2 12,632 19.0 21,579 47.6
Conference proceedings 2,164 2.5 326 1.8 614 0.9 897 2.0
Journal articles 80,203 93.8 17,451 95.4 50,163 75.5 14,543 32.1
Other 2,125 2.5 301 1.6 3,018 4.5 8,287 18.3
Data and analysis: Evidence Thomson Reuters
12
Research policy and management is about ‘more, better research’
Research quality
Research black box
What we want to know
Output data have underpinned quantitative research evaluation
What we have to use
Research quality
Research black box
O U T P U T S
Journals and proceedings Citations
What we want to know
You can now use comprehensive research management information
Research quality
Research black box
Numbers – of researchers,
facilities, collaboration
O U T P U T S
Journals, books &
proceedings
IDEAS proposals,
applications and
partnerships
Trained people
Licences and spin outs
Patents
Deals and revenue
Citation and address
links
Skilled employment
Industrial contracts
Charitable awards
O U T C O M E S
Reports and grey literature
Citations
Social policy change
What evaluators want to know
What evaluation needs to use
Research scholarships
Innovation funds
Research grants
I N P U T S
What research users want to know
Data and analysis: Evidence Thomson Reuters These data, added to peer review, create a modern ‘gold standard’
They are proxy indicators, not metrics We use multiple ‘bearings’ to assess our uncertainty
Data and analysis: Evidence Thomson Reuters
CONTROVERSY, LIMITATIONS & TRENDS Article level indicators Rankings vs profiles Social impact vs citation impact Visualisation Scientific approach Balance with peer evaluation
17
CONTROVERSY
18
JOURNAL EDITORIAL POLICY SELECTION CRITERIA
40% of the journals:
• 80% of the publications
• 92% of cited papers
4% of the journals:
• 30% of the publications
• 51% of cited papers
0
20
40
60
80
100
120
0 1000 2000 3000 4000 5000 6000
# of journals
% o
f dat
abas
e
Articles Citations
… exponentially diminishing returns of extending a search for references in science journals … Samuel C Bradford 1934
LIMITATIONS TO BIBLIOMETRICS
Source data • Representation
Outputs • Papers, patents, products, procedures, spin-offs
Impact • Motivation for citation • Citation, Social, Economic
WHY DO PEOPLE CITE PAPERS?
List of reasons:
Implications:
LIMITATIONS TO BIBLIOMETRICS NEGATIVE CITATIONS
LIMITATIONS TO BIBLIOMETRICS SELF-CITATIONS
• REV BRAS FARMACOGN: First Journal Impact Factor in 2009=3.462
Journals whose rank in category is significantly distorted by self-citation are removed from JCR for 2 years, then re-evaluated
LIMITATIONS TO BIBLIOMETRICS MUTUAL CITATIONS
24
LIMITATIONS TO BIBLIOMETRICS MULTI-AUTHORED PAPERS
IMPACT OF A PAPER
26
BENCHMARKING A PAPER
27
IMPACT OF A PAPER
28
24 / 13.82 = 1.74 times journal expected citations
NORMALISATION ARTICLE LEVEL INDICATORS
29 Source: Thomson Reuters InCites
This paper has received 45/16.74 = 2.69 times the expected citations for this subject category
This paper has received 45/21.48 = 2.09 times the expected citations for this journal
Integrative & Complementary Medicine articles from 2005 have been cited 16.74 times
Articles published in this journal from 2005 have been cited 21.48 times
EVALUATING INDIVIDUAL RESEARCHERS
Computer Science
Medicine
METHODS & INDICATORS UNIVERSITY RANKINGS
METHODS & INDICATORS ALT METRICS
Alternative impacts (other than traditional citation counting) Usage/Views: Downloads Discussed: Journals, blogs, Twitter, Wiki, Book reviews Saved: mendeley, Cite U Like Citations: Web of Science Recommendations: F1000Prime Several publishers have started providing data BioMed, PLOS, NPG, Elsevier. BMJ study Funders showing interest UK MRC Limitations size & selectivity of database (stat significance) Standardisation of data Validity of indicators
MAPPING / VISUALISING SCIENCE
The Future of Bibliometrics - Summary • Here to stay
• Increasingly scientific – both data and evaluation
• Include all research outputs
• Social impact - Altmetrics
• Backward chaining from outcomes
• Visualisations
• Library to play leading role
• Maintain balance with peer review
34