Bibliometrics: Bibliometrics: coming ready or notcoming ready or not

CAUL, September 2005CAUL, September 2005Cathrine Harboe-ReeCathrine Harboe-Ree

Information kitInformation kit IntroductionIntroduction Assessing Quantitative Performance Assessing Quantitative Performance

Indicators WorkshopIndicators Workshop– Reading listReading list

Citation Analysis in Research Evaluation, Citation Analysis in Research Evaluation, Henk F MoedHenk F Moed

Managing Australian Research Output for Managing Australian Research Output for Increased Return on Investment: the Role Increased Return on Investment: the Role of Open Access Institutional Repositories, of Open Access Institutional Repositories, Cathrine Harboe-ReeCathrine Harboe-Ree

Research Indicator Databases available Research Indicator Databases available from Thomson Scientific (ISI)from Thomson Scientific (ISI)

IntroductionIntroduction

What is bibliometrics?What is bibliometrics? Why is bibliometrics important?Why is bibliometrics important? What’s bibliometrics got to do with us?What’s bibliometrics got to do with us?

Quantitative Performance Quantitative Performance Indicators WorkshopIndicators Workshop

ARC Linkage projectARC Linkage project– Range of performance indicators to assess Range of performance indicators to assess

research performanceresearch performance– Assessment of each indicator for validity, etcAssessment of each indicator for validity, etc– Evaluation of different performance measures on Evaluation of different performance measures on

institutions, groups and individualsinstitutions, groups and individuals– Applicability of field-specific weights to counter Applicability of field-specific weights to counter

inequalitiesinequalities ANU, UQ, DESTANU, UQ, DEST Chief Investigator – Linda ButlerChief Investigator – Linda Butler 16 May workshop to test various indicators16 May workshop to test various indicators

Quantitative Performance Quantitative Performance Indicators Indicators

Citations to non-source publicationsCitations to non-source publications Journal publications classified by impact quartilesJournal publications classified by impact quartiles Identification of highly cited publicationsIdentification of highly cited publications Book publications weighted by publisher prestigeBook publications weighted by publisher prestige Honours, awards and prizesHonours, awards and prizes Election to learned academies and professional Election to learned academies and professional

academic associationsacademic associations ConferencesConferences Service to journalsService to journals Visiting FellowshipsVisiting Fellowships Office Bearers in learned academies and professional Office Bearers in learned academies and professional

academic associationsacademic associations Copyright Agency Limited (CAL) dataCopyright Agency Limited (CAL) data Membership of grants committeesMembership of grants committees

Quantitative Performance Quantitative Performance Indicators findingsIndicators findings

Number of publications as the sole indicator is not Number of publications as the sole indicator is not valid valid

Some sort of peer review process is desirableSome sort of peer review process is desirable Several indicators should be used in combinationSeveral indicators should be used in combination Multiple indicators should not be aggregated to a Multiple indicators should not be aggregated to a

single numbersingle number Quality measures have to be responsive to their Quality measures have to be responsive to their

fieldsfields Individual or small unit assessment is expensive Individual or small unit assessment is expensive

and inappropriateand inappropriate Journal impact factors of the ISI type are highly Journal impact factors of the ISI type are highly

inappropriate inappropriate Process/technical errors can develop over time, Process/technical errors can develop over time,

invalidating all datainvalidating all data

BibliographyBibliography

Linda Butler’s ARC reference listLinda Butler’s ARC reference list Henk F. MoedHenk F. Moed Citation analysis in research Citation analysis in research

evaluationevaluation

Henk Moed: validity Henk Moed: validity elementselements

FormalFormal OpenOpen Scholarly foundedScholarly founded Supplemented with expert knowledgeSupplemented with expert knowledge Carried out in clear policy contextCarried out in clear policy context Explicit statement of basic notions of Explicit statement of basic notions of

scholarly qualityscholarly quality Enlightening rather than formulaicEnlightening rather than formulaic

Henk Moed: Thomson ISIHenk Moed: Thomson ISI Coverage excellent in physics, chemistry, Coverage excellent in physics, chemistry,

molecular biology, biochemistry, biological molecular biology, biochemistry, biological sciences, clinical medicinesciences, clinical medicine

Good in applied and engineering sciences, Good in applied and engineering sciences, biological sciences, geosciences, biological sciences, geosciences, mathematics, medicine and health related mathematics, medicine and health related social sciencessocial sciences

Moderate in sociology, political science, Moderate in sociology, political science, anthropology, education and humanitiesanthropology, education and humanities

Affected by Affected by – non-source material (books, conference non-source material (books, conference

proceedingsproceedings– Language and national barriersLanguage and national barriers

Henk Moed: findingsHenk Moed: findings Alternative data sources and methodologies should be Alternative data sources and methodologies should be

exploredexplored The expertise of specialists is importantThe expertise of specialists is important There is no single ‘perfect’ indicator of journal There is no single ‘perfect’ indicator of journal

performanceperformance Thomson ISI journal impact factors are not a reliable guideThomson ISI journal impact factors are not a reliable guide Data accuracy is a key problemData accuracy is a key problem The substantive content of work should be taken into The substantive content of work should be taken into

accountaccount The level of aggregation is crucialThe level of aggregation is crucial Both citation analysis and peer review have strengths and Both citation analysis and peer review have strengths and

limitslimits A study of RAE questions its ability to identify ‘top’ A study of RAE questions its ability to identify ‘top’

research departmentsresearch departments Including more sources (ie web) does not necessarily lead Including more sources (ie web) does not necessarily lead

to more valid assessment of scholarly contributionto more valid assessment of scholarly contribution

Open access impactOpen access impact Strong consistent evidence that open access Strong consistent evidence that open access

availability has a significant impact on availability has a significant impact on citation rates (50-250%)citation rates (50-250%)

Citations themselves are not a measure of Citations themselves are not a measure of qualityquality

Scholars value citednessScholars value citedness RQF could consider it moreRQF could consider it more University ranking systems using journal University ranking systems using journal

impact factorsimpact factors– Should use citation factorsShould use citation factors

Tension between ‘quality’ and ‘accessibility’ Tension between ‘quality’ and ‘accessibility’ and “impact”and “impact”

Thomson Scientific Thomson Scientific databasesdatabases

Essential Science IndicatorsEssential Science Indicators Institutional Citation ReportsInstitutional Citation Reports University Science IndicatorsUniversity Science Indicators National Science IndicatorsNational Science Indicators High Impact PapersHigh Impact Papers Journal Citation ReportsJournal Citation Reports Web Citation IndexWeb Citation Index

Thomson Scientific Thomson Scientific databasesdatabases

Linda Butler Linda Butler – Uses raw dataUses raw data– Cleans itCleans it– Maps to RFCD codesMaps to RFCD codes– Cautions against use by people who Cautions against use by people who

don’t understand the datadon’t understand the data– Is concerned that there are not enough Is concerned that there are not enough

people who can interrogate the datapeople who can interrogate the data