1

Pride and prejudices: Problems with national and international league tables

Dr Paul Greatrix, Registrar, University of Nottingham

2

League Tables

•  Background •  Who wants to know? •  Who are the providers of this valuable

information? •  Mad, bad and dangerous •  National and international dimensions •  Measuring the unmeasurable?

3

Background

They sell papers...

•  ...and, some would suggest, create perverse incentives

4

Who wants to know?

•  The Government •  The State •  The Funding Councils •  Potential Students •  Alumni •  Journalists

•  Parents, teachers, advisors

•  Employers – national and international

•  Overseas sponsors •  Jo Public •  The Universities •  Dangerous obsessives

5

Who’s responsible for providing this valuable data? •  The Times

•  Sunday Times

•  The Guardian

•  Complete University Guide

•  Financial Times

•  THE

•  Shanghai Jiao Tong University

•  QS •  US News and World Report •  HEFCE •  Others... •  and, of course,

Government

6

Invaluable information

•  For all stakeholders •  Intelligent decision-making •  Better than prejudice •  Reflecting the realities of

the market place •  We have a right to know

•  Employers have the right to know

•  We aren’t stupid

7

League tables are a bad thing...

“The silly season that marks the publication of University league tables is nonsensical and illogical. As any New Scientist knows, letters into numbers; quality into quantity won’t go. League tables are simplistic, divisive and undermine the qualitative nature of a University’s work”

8

But we’ll use them anyway!

“Having said that, I’m not ashamed to report that we came a very creditable 79th overall, with my own department rating a particularly good score for research

- and as I remarked to the Dean, you can’t get much better than that.”

9

Dangerous... •  Criteria used do not reflect quality of education •  Historical data •  Variation over time •  Scores are institutional averages – mask strengths •  Distorting effect of weightings and scalings and

data manipulation

•  Many of criteria used are inter-related (ie not independent)

10

... extremely dangerous... •  Apples and elephants and

paperclips •  Perverse incentives •  Hugely political •  Open to manipulation •  Delivered by journalists •  Spurious precision •  Serious consequences for

universities, departments, staff and students

11

The Times 2011

1 Oxford 2 Cambridge 3 LSE 4 Imperial 5 UCL 6 St Andrews 6 Durham 7 UCL 8 Warwick 9 Lancaster 10 Exeter

11 York 12 Bath 13 Bristol 14 Sussex 15 Edinburgh 16 Nottingham 17= Leicester 17= Sheffield 19 Southampton 20 Loughborough

12

Sunday Times 2011

1 Cambridge 2 Oxford 3 Durham 4 LSE 5 Bath 6 St Andrews 7 UCL 8 Warwick 9 Exeter 10 Bristol

11 Loughborough 12 Newcastle 13 Sheffield 14 Imperial 15= Nottingham 15= York 17 King’s 18 Lancaster 19 Sussex 20 Glasgow

13

The Guardian 2011

1 Cambridge 2 Oxford 3 St Andrews 4 LSE 5 UCL 6 Warwick 7 Lancaster 8 Durham 9 Loughborough 10 Imperial

11= Exeter 11= Sussex 13 SOAS 14 Bath 15 York 16 Edinburgh 17 Leicester 18 UEA 19= Nottingham 19= Surrey

Wednesday, 4 April 12 14

Complete University Guide 2011

1 Cambridge 2 Oxford 3 Imperial 4 LSE 5 Durham 6 St Andrews 7 UCL 8 Warwick 9 Lancaster 10 Bath

11 Bristol 12 York 13 Edinburgh 14 Southampton 15 Exeter 16 King’s 17 Nottingham 18 SOAS 19= Loughborough 19= Sussex

15

International value-added

•  HE is now a global business •  Global branding assisted by competitive

ranking •  International benchmarking increasingly

important, especially in research •  Student recruitment is increasingly

international... •  ...mobile students are increasingly

choosy •  It’s all good healthy fun

16

Just as dangerous... •  Again, the criteria used do not reflect

quality of education •  Archaic and irrelevant data •  Major biases to large, English-

speaking, research- and science-intensive universities

•  Institutional scores are extraordinarily broad brush

•  Distorting effect of weightings, scalings and data manipulation

•  Far from comprehensive surveys of peers and employers

17

Times Higher Education World Rankings 2011

4 Oxford 6 Cambridge 8 Imperial College 17 University College London 36 Edinburgh 47 LSE

48 Manchester 56 King’s 66 Bristol 83 Durham 85 St Andrews 99 Sussex

18

1  Cambridge (1) 5 Oxford (6) 6 Imperial (7) 7 UCL (4) 20 Edinburgh (22) 27 King’s (21) 29 Manchester (30) 30 Bristol (27) 50 Warwick (53) 59 Glasgow (77)

64 LSE (80) 67 Birmingham (59) 72 Sheffield (69) 74 Nottingham (73) 75 Southampton (81) 93 Leeds (85) 95 Durham (92) 96 York (88) 97 St Andrews (95)

QS World Rankings 2011

19

5 Cambridge (5)

10 Oxford (10)

20 UCL (21)

24 Imperial (26)

38 Manchester (44)

53 Edinburgh (54)

68 King’s (63)

70 Bristol (66)

85 Nottingham (84)

97 Sheffield (88)

Shanghai Jiao Tong ARWU 2011

20

1  Harvard 2  Tokyo University 3  Keio University 4  HEC (France) 5= Kyoto 5= Oxford 7  Ecole Polytechnique 8  Waseda 9  ENA (France) 10  Seoul National University

11 U of Pennsylvania 12 Columbia 13= Stanford 13= Tohoku U 13= Nottingham 16 MIT 17 Institute for Study of Politics, Paris 18 U St Gallen 19 U Sao Paolo 19 Northwestern U

Another approach: Ecole des Mines de Paris Rankings

21

1 Massachusetts Institute of Technology 2 Stanford University 3 Harvard University 4 University of California Berkeley 5 Cornell University 6 University of Michigan 7 California Institute of Technology 8 University of Minnesota 9 University of Illinois Urbana Champaign 10 University of Texas Austin

Another Approach: Webometrics

22

1 University of Cambridge, World Rank: 28 2 Swiss Federal Institute of Technology WR: 40 3 University of Oxford WR: 42 4 University of Helsinki WR: 58 5 University of Oslo WR: 60 6 University of Edinburgh WR: 65 7 University College London WR: 68 8 Utrecht University WR: 71 9 University of Manchester WR: 82 10 Royal Institute of Technology (Sweden) WR: 84

Webometrics Europe

23

International approaches 1

Shanghai Jiao Tong: Ranking of World Universities –  6 indicators covering: quality of education;

quality of faculty; research output; performance relative to size

–  Includes: Nobel and Fields winners among alumni and staff; highly cited researchers; articles in Nature and Science; articles in citation indices.

24

International approaches 2

25

Other tables

•  ALOHA, PISA, OECD •  China – number of billionaires •  Just research performance: ranking of

scientific papers •  Most conservative, best parties, most interns •  European initiative – U-Multirank •  Islamic universities

.

Guess the League Table!

1)  Loughborough University 2)  University of Sheffield 3)  University of East Anglia 4)  University of Cambridge 5)  University of Dundee 6)  University of Oxford 7)  University of Glasgow 8)  University of Leeds 9)  Aberystwyth University 10) University of Southampton

Which league table?

1)  Leeds Metropolitan University 26.72 2)  University of Liverpool 26.63 3)  University of Wales Institute, Cardiff 26.14 4)  Bath Spa University 24.95 5)  University of Glamorgan 22.26 6)  University of Bath 21.97 7)  University of Brighton 21.98 8)  Buckinghamshire New University 21.59 9)  Newcastle University 21.2 10) De Montfort University 21.0

Which league table?

29

1  University of Glamorgan 10.9 2  University of Wales Institute, Cardiff 10.6 3  University of Brighton 9.5 4  Leeds Metropolitan University 9.4 5  Aberystwyth University 9.1 6  Manchester Metropolitan University 8.4 7  University of Dundee 8.2 8  University of Liverpool 8.1 9  University of Strathclyde 7.7 10  University College London 7.6

Which league table?

30

1 University of Nottingham, UK 2 Northeastern University, US 3 University of Connecticut, US 4 University College Cork, Ireland 5 Linkoping University, Sweden 6 University of California, Berkeley, US 7 University of California, Los Angeles, US 8 Washington University In St. Louis, US 9 University of California Merced, US 10 University of Bath, UK

Which league table?

31

1 Greenwich 2 Sheffield Hallam 3 Kingston 4 Westminster 5 East London 6 Central Lancashire 7 Leeds Met 8 Wolverhampton 9 Coventry 10 Middlesex

Which league table?

32

1 Webb Institute 70.9 2 Carleton College 61.3 3 Princeton University 60.3 4 Middlebury 60.1 5 Amherst 59.5 6 Williams College 57.6 7 Centre College 56.7 8 Indiana Institute of Technology 55.1 9 Davidson College 54.9 10 Thomas Aquinas College 52.5

Which league table?

33

1. Oxford 39 2. Cambridge 21 3. Manchester 8 4=. Imperial College 5 4=. Open University 5 6=. Durham 4 6=. Sussex 4 8=. St. Andrews 3 8=. Birkbeck 3

Which league table?

1)  Nottingham Trent University 2)  University of Gloucestershire 3)  University of Worcester 4)  University of Plymouth 5)  Bournemouth University 6)  University of Greenwich 7)  Bath Spa University 8)  University of East Anglia 9)  University of Hertfordshire 10) Leeds Metropolitan

Which league table?

35

An indicator too far (not covered in the UK tables - yet)

•  Alumni giving •  Academic staff pay •  Research income •  Citations •  Brand impact But international tables will increasingly influence methodologies of UK tables

36

What is to be done •  Publish prolifically and get top RAE grades •  Be Highly Cited •  Get articles in Nature and Science •  Win a Nobel Prize or a Fields Medal •  Make sure your students all get top jobs with big multinationals (and

then win Nobels) •  Plug the institution relentlessly and cultivate peers, Headmasters/

mistresses and employers •  Improve NSS results (and SSRs) - every little helps •  Spend more on Library and IT and everything else to do with teaching

and learning •  Recruit more international students and staff (all with Nobel potential)

37

Conclusions •  They aren’t going to go away

•  The international dimension will become increasingly significant

•  Methodologies - for both national and international tables - are all dubious, at best

•  They can and will be used by many different groups – but can be dangerous in the wrong hands

•  Handle with great care!

38

Finally For a copy of the presentation and further league table commentary and observations, visit:

http://registrarism.wordpress.com/ Also on Twitter: @registrarism