Journal ranking as a measure of research

excellence in Australia

“Not everything that can be counted counts, and not everything that counts

can be counted.”

EXCELLENCE IN RESEARCH FOR AUSTRALIA (ERA)

• “objective of providing institutions, researchers, industry and students with a sound, evidence-based means of identifying areas of strength and potential, as well as areas where we need to do better”

“Universities must brace for ERA funding consequences”

BY: JILL ROWBOTHAM From: The Australian February 16, 2011

• "No doubt ERA has been a valuable exercise for universities to have a look at their research performance," Larkins says.

• ….it will be interesting to see what leads the government gives to how this information will be used."

Controversy

• In late 2008, the Australian Research Council (ARC) began a ranking exercise of all journals in which Australian researchers had published in 2003-2008.

• They were allocated a rank of A*, A, B or C.• Not ranked meant a newish journal.

A personal example

http://lamp.infosys.deakin.edu.au/era/

A personal example

http://lamp.infosys.deakin.edu.au/era/

A personal example

http://lamp.infosys.deakin.edu.au/era/

(not on NUS list)

“End of an ERA: journal rankings dropped”

BY:JILL ROWBOTHAM From: The Australian May 30, 2011

• “...rankings were being deployed inappropriately ..., in ways that could produce harmful outcomes, and based on a poor understanding of the actual role of the rankings.”

• The journals lists will still be of great utility and importance, but the removal of the ranks and the provision of the publication profile will ensure they will be used descriptively rather than prescriptively.”

Prof Les FieldsDVC(R) UNSW

• “It was providing incentives to publish in the listed journals rather than in the most appropriate outlets for the disciplines.”

• “There was pressure to move away from publishing books or in books and towards the listed journals since this is what was being measured and captured by the ERA.”

• “The ARC have opted for more responsibility for their expert review panels”

Professor Joseph Lo Bianco, President of the Australian Academy of the Humanities

• “The Academy is aware of several institutions who were directing staff to publish only in top ranked journals, despite the fact that the scholarly monograph remains the pre-eminent form of publication in some disciplines.”

Academy of the Social Sciences in Australia

• “The ranking of publication outlets based on international prestige had threatened to drive productive researchers away from an Australian research focus in favour of research that would be of interest to countries, or regions, where the highest ranked publication outlets are found.”

Professor Bob Williamson AO, Australian Academy of Science

• “… the Australian Academy of Science argued strongly that key areas such as interdisciplinary research and new research were seriously disadvantaged by journal ranking.

• This affected not only areas of science and technology, but also interactions between the sciences and the humanities”

• “The ranking of a journal as A* does not mean every paper in it is first rate, and some very good papers may appear in smaller journals.”

• “People whose work is very relevant to Australian issues rather than internationally, and those in new fields or collaborating between several universities, have been particularly disadvantaged.”

Professor Margaret Sheil, CEO of the

Australian Research Council

• “The journal rankings were just one indicator we looked at in the ERA process.”

• “...a conversation that happens in every grant assessment process. It happens in every promotion committee in the country.

• People say, ‘Where are they getting their work published and is that appropriate for this type of work?’”

• “The committees know, because they are experts, which journals are higher quality or prestigious.

• So, for example, Nature and Science are like the gold standard in science but there are certain areas of science that never get into Nature and Science because they are not of general enough interest.”

• “What they are going to have in the new system is they will have a list of what those top 20 journals are in terms of frequency of publication.

• (Frequency of publication) means how often that unit publishes in a journal. For example, (the unit may publish) 40 papers in Australian Law Review and 10 papers in the Journal of International Law and so on.

• If you are a scientist and you see a unit that has 20 papers in Science and 10 papers in Nature, you do actually know a lot about the quality (of research being produced by that unit).”

So, in measuring research outputs• Publications are very important• Some journals are “better” than others.• However, journal ranking as the primary measure

of excellence can disadvantage:– disciplines that disseminate their findings in

books and monographs;– new / emerging disciplines;– esoteric research;– national / regional research;– interdisciplinary research;– non-traditional research outputs.

Jason Nelson’s art portal http://secrettechnology.com/ has had over 10 million hits since 2007

Non-traditional research outputs (ARC)

• Original Creative Works;• Live Performance of Creative Works;• Recorded/Rendered Creative Works; and• Curated or Produced Substantial Public

Exhibitions and Events.

“a statement identifying the research component of each such research output must be provided”

Recorded/Rendered Creative Works

Recorded/Rendered Creative Works

Recorded/Rendered Creative Works

Recorded/Rendered Creative Works

Recorded/Rendered Creative Works

• “…the research component is contained within the recording/rendering. Simple documentations of live performances of creative works without a research component are not eligible…”

Research Statement for Non-Traditional Research Outputs

• Research Background– Field– Context– Research Question

• Research Contribution– Innovation– New Knowledge

• Research Significance– Evidence of Excellence

AN EXAMPLE OF A VISUAL ARTS RESEARCH STATEMENT

Another “fringe dweller”

Another “fringe dweller”

1,301 publications

12,446 citations

H-index = 49

Paper #1

417 citations, ca 20 per year

Paper #1

417 citations, ca 20 per year

• IF = 1.270• not even on NUS list

Valuable scholarship under recognized?

• “It can scarcely be denied that the supreme goal of all theory is to make the irreducible basic elements as simple and as few as possible without having to surrender the adequate representation of a single datum of experience.”

• “Everything should be made as simple as possible, but no simpler.”