Haustein, S. (2017). The evolution of scholarly communication and the reward system of science

The evolution of scholarly communication and the reward system of science

Stefanie Haustein @stefhaustein

OutlineScholarly communicationFrom the 16th century to Open Science

BibliometricsFrom library management to research evaluation

AltmetricsOpportunities and challenges

Conclusions and Outlook

Invisible Colleges

http://commons.wikimedia.org/wiki/File:Marin_mersenne.jpg#/media/File:Marin_mersenne.jpghttp://commons.wikimedia.org/wiki/File:Henry_Oldenburg.jpg

Père Marin Mersenne(1588-‐1648)

Henry Oldenburg(1619-‐1677)

Scientific Societies

http://upload.wikimedia.org/wikipedia/commons/thumb/6/61/Louis_XIV_und_Colbert_in_der_Akademie.jpg/640px-‐Louis_XIV_und_Colbert_in_der_Akademie.jpghttps://www.rcseng.ac.uk/museums/hunterian/images/lost-‐museums-‐2011/the-‐royal-‐society-‐repository/Image%201%20-‐%20GreshamCollege.jpg/image_previewhttp://en.wikipedia.org/wiki/Nullius_in_verba#/media/File:Bookplate_of_the_Royal_Society_(Great_Britain).jpg

L’Académie royale des sciences22 December 1666

The Royal Society28 November 1660

Scientific Journals

http://de.wikipedia.org/wiki/Journal_des_s%C3%A7avans#/media/File:1665_journal_des_scavans_title.jpghttp://en.wikipedia.org/wiki/Philosophical_Transactions_of_the_Royal_Society#/media/File:Philosophical_Transactions_Volume_1_frontispiece.jpg

Le journal des sçavans5 January 1665

Philosophical Transactions6 March 1665

Scientific Articles

Harmon, J.E. & Gross, A.G. (2007). The Scientific Literature. A Guided Tour. Chicago: University of Chicago Press.

• Experiments and descriptions of the natural world• Avoiding “fine speaking”• Various styles of arguing• Qualitative and personal judgements

Proportion of IMRaDadoption in medical journalsNumberof references1900 to 2004

Scientific Articles

Larivière, V., Archambault, É. & Gingras, Y. (2008). Long-‐term variations in the aging of scientific literature: From exponential growth to steady-‐state science (1900-‐2004). Journal of the American Society for Information Science and Technology, 59(2), 288-‐296.Sollaci, L.B. & Pereira, M.G. (2004). The introduction, methods, results, and discussion (IMRAD) structure: a fifty-‐year survey. Journal of the Medical Library Association, 92(3), 364-‐371

• Professionalized and highly specialized• Increased focus on data, graphs, tables and theory• Impersonal, technical and codified• Style guides and gatekeeping• Citations• Introduction, Methods, Results and Discussion

Digital Revolution

arXiv submission statistics from http://arxiv.org/stats/monthly_submissionsLarivière, V., Lozano, G.A. & Gingras, Y. (2014). Are elite journals declining? Journal of the Association for Information Science and Technology, 65(4), 649-‐655.

• Improved access• Acceleration

• Collaboration• Peer review• Distribution of preprints

• Decreasing importance of scientific journal• Journal functions• Diversification of

publication venues• Symbolic capital of

journals unchanged

Submissions to arXiv

Share of top 1% mostcitedpapers

Academic Publishing Market

Larivière, V., Haustein, S., & Mongeon, P. (2015). The oligopoly of academic publishers in the digital era. PLoS ONE, 10(6), e0127502. doi: 10.1371/journal.pone.0127502

• Aggravation of serials crisis

• Elsevier: 30% increase of subscription price

• Profit margins of commercial publishers up to 40%

• Decline of scientific societies as publishers

• >50% of papers owned by five major publishers

Open Access

Archambault, É., Amyot, D., Deschamps, P., Nicol, A., Rebout, L. & Roberge, G. (2013). Proportion of Open Access Peer-‐Reviewed Papers at the European and World Levels 2004-‐2011. Report for the European Commission. http://www.science-‐metrix.com/pdf/SM_EC_OA_Availability_2004-‐2011.pdf

Budapest Open Access Initiative“immediate, free availability on the public internet, permitting any users to read, download, copy, distribute, print, search or link to the full text of these articles”

• Gold and Green• Libre and Gratis• Hybrid

• Elsevier: $500 to 5,000• Springer: $3,000• Wiley: $3,000

Freelyavailable journal papers2004 to 2011

Budapest Open Access Initiative (2002)

Open Science

Kraker, P., Leony, D., Reinhardt, W. & Beham, G. (2011). The case for an open science in technology enhanced learning. International Journal of Technology Enhanced Learning, 3(6), 643-‐654.

“opening up the research process by making all of its outcomes, and the way in which these outcomes were achieved, publicly available on the World Wide Web”

• Open Data• Open Source• Open Methodology• Open Access• Open Peer Review

Krakeret al. (2011, p. 645)

Bibliometrics

Gross, P.L.K. & Gross, E.M. (1927). College libraries and chemical education. Science, 66(1713), 385-‐389.

Library collection management Journals cited in the Journal of the American Chemical Society1926

Bibliometrics

Garfield, E. (1955). Citation indexes for science. A new dimension in documentation through association of ideas. Science, 122, 108-‐111.

Information retrieval•

“It would not be excessive to demand that the thorough scholar check all papers that have cited or criticized such papers, if they could be located quickly. The citation index makes this check practicable.”

• Institute for Scientific Information• Science Citation Index• Source Author Index• Citation Index

Garfield (1955, p. 108)

Bibliometrics

Price, D. J. d. S. (1961). Science Since Babylon. New Haven / London: Yale University Press,Price, D. J. d. S. (1963). Little Science, Big Science. New York: Columbia University Press.

Sociology of scienceDerek J. de Solla PriceScience since Babylon (1961)Little Science – Big Science (1963)

Bibliometrics

Merton, R. K. (1988). The Matthew effect in science, II: Cumulative advantage and the symbolism of intellectual property. Isis, 79, 606–623.

Sociology of scienceRobert K. Merton• Social norms of science

• Communalism• Universalism• Disinterestedness• Organized skepticism

• Matthew effect

“symbolically, [the reference] registers in the enduring archives the intellectual property of the acknowledged source by providing a pellet of peer recognition of the knowledge claim”

Merton (1988, p. 621)

Bibliometrics

Moed, H.F., Burger, W.J.M., Frankfort, J.G, van Raan, A.F.J. (1985). The use of bibliometric data for the measurement of university research performance. Research Policy, 14(3), 131-‐149.

Research evaluation• Performance measurement and policy instrument

“When used properly, bibliometric indicators can provide a ‘monitoring device’ for university research-‐management and science policy. They enable research policy-‐makers to ask relevant questions of researchers on their scientific performance, in order to find explanations of the bibliometric results in terms of factors relevant to policy.”

• Commercialization

Moedet al. (1985, p. 131)

BibliometricsResearch evaluation• Part of hiring, promotion and funding decisions• Dashboard tools

Bibliometrics

Hvistendahl, M. (2013). China’s publication bazaar. Science, 342(6162), 1035-‐1039.van Noorden, R. (2013). Brazilian citation scheme outed: Thomson Reuters suspends journals from its rankings for ‘citation stacking’, Nature, 500(7464), 510-‐511.

Research evaluation• Oversimplification

• Publications = productivity• Citations = impact

• Uninformed use and misuse• Impact factor• h-‐index

• Adverse effects• “Salami” publishing• Honorary authorship• Self-‐citations• Citation cartels

Scholarly metrics

Björneborn, L. & Ingwersen, P. (2004), Toward a basic framework for webometrics. Journal of the American Society for Information Science and Technology, 55(14), 1216–1227.

Definitionsinformetrics

scientometrics

bibliometrics

cybermetrics

webometrics

adaptedfrom: Björneborn& Ingwersen(2004, p. 1217)

Scholarly metrics

Otlet, P. (1934). Traité de documentation: le livre sur le livre, théorie et pratique.Pritchard, P. (1927). Statistical bibliography or bibliometrics? Journal of Documentation, 25, 348-‐349..

Bibliometricsinformetrics

scientometrics

bibliometrics

cybermetrics

webometrics

“La «Bibliometrie» sera la partie définie de la Bibliologie qui s'occupe de la mesure ou quantité appliquée aux livres.”

“the application of mathematics and statistical methods to books and other media of communication”

Pritchard (1969, p. 348)

Otlet (1934, p. 14)

Scholarly metricsAltmetrics


informetrics

scientometrics

bibliometrics

cybermetrics

webometrics altmetrics


Scholarly metrics

Priem, J. (2014). Altmetrics. In B. Cronin & C. R. Sugimoto (Eds.), Beyond bibliometrics: harnessing multidimensional indicators of performance (pp. 263–287). Cambridge, MA: MIT Press. Rousseau, R. & Ye, F. (2013). A multi-‐metric approach for research evaluation. Chinese Science Bulletin, 3288–3290. doi:10.1007/s11434-‐013-‐5939-‐3

Altmetricsinformetrics

scientometrics

bibliometrics

cybermetrics

webometrics altmetrics

“study and use of scholarly impact measures based on activity in online tools and environments”

“a good idea but a bad name”

Rousseau & Ye (2013, p. 3289)

Priem(2014, p. 266)

Scholarly metricsDefinition

altmetrics

informetrics

scientometrics

bibliometrics

cybermetrics

webometrics


Scholarly metrics


Scholarly metrics

Haustein, S., (2016). Grand challenges in altmetrics: heterogeneity, data quality and dependencies. Scientometrics, 108(1), 413–423.

altmetrics

informetrics

scientometrics

bibliometrics

cybermetrics

webometrics

Scholarly metricsActs: viewing, reading, saving, diffusing, mentioning, citing, reusing, modifying, etc.Scholarly documents: papers, books, blog posts, datasets, code, etc.Scholarly agents: researchers, universities, funders, journals, etc.

“[S]cholarly metrics are thus defined as indicators based on recorded events of acts […] related to scholarly documents […] or scholarly agents […].”

Haustein (2016, p. 348)

Altmetrics

Priem, J., Taraborelli, D., Groth, P., & Neylon, C. (2010). Alt-‐metrics: a manifesto. October. Retrieved from http://altmetrics.org/manifesto/

• Information overload“We rely on filters to make sense of the scholarly literature, but the narrow, traditional filters are being swamped. However, the growth of new, online scholarly tools allows us to make new filters; these altmetrics reflect the broad, rapid impact of scholarship in this burgeoning ecosystem.”

• Criticism against current form of research evaluation• Alternative forms of research output• Alternative use and visibility of publications

Priemet al. (2010)

AltmetricsCoverage per platform

Haustein, S., Costas, R., & Larivière, V. (2015). Characterizing social media metrics of scholarly papers: The effect of document properties and collaboration patterns. PLoS ONE, 10(5), e0127830. doi: 10.1371/journal.pone.0120495Zahedi, Z., & Haustein, S. (in preparation). Which document features attract users in Mendeley? An analysis of bibliographic characteristics of Web of Science publications and Mendeley readership counts.

Mathematics &Computer Science

Natural Sciences& Engineering

Life &Earth Sciences

Biomedical &Health Sciences

Social Sciences& Humanities

76,4 %

83,7 %

91,4 %

86,5 %

81,7 %

Men

deley

7,5 %

12,9 %

21,6 %

31,7 %

26,0 %

Twitter

AltmetricsCoverage per discipline


AltmetricsDensity and intensity per platform

Intensity


AltmetricsSpearman correlations with citations

Perfectn

egativecorrelation

Perfectp

ositive correlatio

n

Haustein, S., Larivière, V., Thelwall, M., Amyot, D., & Peters, I. (2014). Tweets vs. Mendeley readers: How do these two social media metrics differ. Information Technology, 56(5), 207–215. doi: 10.1515/itit-‐2014-‐1048Haustein, S., Costas, R., & Larivière, V. (2015). Characterizing social media metrics of scholarly papers: The effect of document properties and collaboration patterns. PLoS ONE, 10(5), e0127830. doi: 10.1371/journal.pone.0120495

AltmetricsSpearman correlations with citationsNSF Subdiscipline General Biomedical Research 2011

Size of data points represents number of Mendeley readers in Twitter graph (left) and number of tweetsin Mendeley graph (right).

Haustein, S., Larivière, V., Thelwall, M., Amyot, D., & Peters, I. (2014). Tweets vs. Mendeley readers: How do these two social media metrics differ. Information Technology, 56(5), 207–215. doi: 10.1515/itit-‐2014-‐1048

AltmetricsDocument types

Haustein, S., Costas, R., & Larivière, V. (2015). Characterizing social media metrics of scholarly papers: The effect of document properties and collaboration patterns. PLoS ONE, 10(5), e0127830. doi: 10.1371/journal.pone.0120495

AltmetricsHighly tweeted

AltmetricsAltmetrics in the wild

Opportunities and Challenges

• Heterogeneity• Time and timing• Audiences and user groups

Altmetrics

Opportunities• Different acts• Diverse motivationsØ Diverse impact

Challenges• Understanding underlying processesØ Determining the meaning of metrics

Heterogeneity of Altmetrics

Heterogeneity

Savingto Mendeley

Mentioningin News

Heterogeneity

Recommendingon F1000

Tweeting

Heterogeneity

Bertin, M., Atanassova, I., Gingras, Y., & Larivière, V. (2015). The invariant distribution of references in scientific articles. Journal of the Association for Information Science and Technology, 67(1), 164-‐177. doi: 10.1002/asi.23367

Distribution of references along the IMRaDstructure

Citing in ajournal article

HeterogeneityActs referring to research objects

Haustein, S., Bowman, T. D., & Costas, R. (2016). Interpreting “altmetrics”: Viewing acts on social media through the lens of citation and social theories. Dans C. R. Sugimoto (dir.), Theories of Informetrics and Scholarly Communication (p. 372–405). Berlin: De Gruyter Mouton. doi: 10.1515/9783110308464-‐022

RESEARCH OBJECT

Time and TimingOpportunities• Detailed life cycle of scientific outputØ Fine-‐grained indicators and adequate benchmarks

Challenges• Versions of research output• Publication dates

Time and TimingJournal article• Submitted manuscript• Revised manuscript• Accepted manuscript• Version of Record• Online publication• Journal issue• Online date• Issue month

Ø Adjusting indicators

3 March 201415 July 2014

21 January2015February2015

Time and Timing

Time and TimingTweets before publication?

Time and TimingWeekday effects on Twitter

weekday ofonline publication:

based on:8,765 Springer papers with

online publication date19,010 tweets received within one year

of online publication date

Audiences and User GroupsOpportunities• Differentiating between types of users• Measuring societal impact

Challenges• Identifying users and user groups• Determining engagement

Audiences and User Groups

Alperin, J. P. (2015). Moving beyond counts: A method for surveying Twitter users. In altmetrics15: 5 years in, what do we know? Amsterdam, The Netherlands. Retrieved from: http://altmetrics.org/wp-‐content/uploads/2015/09/altmetrics15_paper_3.pdfTsou, A., Bowman, T. D., Ghazinejad, A., & Sugimoto, C. R. (2015). Who tweets about science? In Proceedings of the 2015 International Society for Scientometrics and Informetrics (pp. 95–100). Istanbul, Turkey.

Identifying Twitter users• Altmetric.com classification

• Among a random sample of 2,000 accounts tweeting papers, 34% of individuals identified as having PhD

• Of 286 users linking to SciELO articles, 24% employed at university, 23% students, 36% not university affiliated

*based on Altmetric.com data 06/2015

(Tsou, Bowman, Ghazinejad, & Sugimoto, 2010)

(Alperin, 2015)

1

2

3

Audiences and User Groups

Haustein, S., & Costas, R., (2015). Identifying Twitter audiences: who is tweeting about scientific papers? Communication présentée au SIG/MET Workshop, ASIS&T 2015 Annual Meeting, 7 novembre 2015, Saint-‐Louis, MO (USA).

topics andcollectives

academic

personal

Node sizenumber of accountsassociated with term

Node colorcluster affiliation

Terms in Twitter bio

Audiences and User GroupsEngagement with scientific papers on Twitter

Haustein, S., Bowman, T. D., Holmberg, K., Tsou, A., Sugimoto, C. R., & Larivière, V. (2016). Tweets as impact indicators: Examining the implications of automated bot accounts on Twitter. Journal of the Association for Information Science and Technology, 67(1), 232–238. doi: 10.1002/asi.23456

Audiences and User GroupsTwitter arXiv bots

Audiences and User GroupsJournal accounts

Conclusions• Scholarly communication and the reward system

of science are changing• Potential to become more transparent and

diverse• Open Science• Scholarly metrics

• Fundamental difference between posting on social media and academic publishing

• More metrics = more complexity

Outlook

More scientometric studies.

More sociological research.

Avoid adverse effects.

Haustein, S. (2017). The evolution of scholarly communication and the reward system of science

Science

Transcript of Haustein, S. (2017). The evolution of scholarly communication and the reward system of science