Organisational complexity as a challenge to research assessment: a case study of the University of...
Transcript of Organisational complexity as a challenge to research assessment: a case study of the University of...
![Page 1: Organisational complexity as a challenge to research assessment: a case study of the University of Oxford](https://reader036.fdocuments.net/reader036/viewer/2022062406/55b3c77bbb61ebc0548b460d/html5/thumbnails/1.jpg)
Organizational complexity as a challenge to research assessment
Sally RumseyThe Bodleian LibrariesUniversity of Oxford
A case study of the University of Oxford
![Page 2: Organisational complexity as a challenge to research assessment: a case study of the University of Oxford](https://reader036.fdocuments.net/reader036/viewer/2022062406/55b3c77bbb61ebc0548b460d/html5/thumbnails/2.jpg)
Complexity
A result of
• Size of the institution and the amount of research produced (research intensive)
Coupled with
• Organisational complexity of the institution (administrative structure; governance; etc)
![Page 3: Organisational complexity as a challenge to research assessment: a case study of the University of Oxford](https://reader036.fdocuments.net/reader036/viewer/2022062406/55b3c77bbb61ebc0548b460d/html5/thumbnails/3.jpg)
University of Oxford Facts and figures*
• > 5800 academic and research staff• > 5600 research students• 4 university academic divisions (> 70 departments)
– Humanities: 9 faculties + Ruskin School of Art– Social Sciences: 16 departments– MPLS (Mathematical, Physical & Life Sciences): 12 departments– Medical Sciences: 16 departments– Plus cross disciplinary institutes eg Oxford Martin School and others
• 44 colleges and PPHs (Permanent Private Halls) (separate legal entities to University)
• External research funding £478.3m• Approx 4700 active research awards• Estimated 16,000 journal articles published p.a. • > 120 libraries* 2013 and 2013/14 figures: Annual Review 2013-14
![Page 4: Organisational complexity as a challenge to research assessment: a case study of the University of Oxford](https://reader036.fdocuments.net/reader036/viewer/2022062406/55b3c77bbb61ebc0548b460d/html5/thumbnails/4.jpg)
Democracy
• Wellington Square as the centre of ‘the University’
• Democratic• Decision-making devolved to
the four Academic Divisions and their departments/faculties
• Decision-making can progress slowly
• Example: Symplectic adoption for REF 2014
• REF OA Service design project (for REF 2020) Cppyright Clare Byers
![Page 5: Organisational complexity as a challenge to research assessment: a case study of the University of Oxford](https://reader036.fdocuments.net/reader036/viewer/2022062406/55b3c77bbb61ebc0548b460d/html5/thumbnails/5.jpg)
Colleges
• Separate legal entities to the university
• Dual appointments• College-only
appointed staff• Not on HR system• Submitted to REF
Copyright Jason Partridge
![Page 6: Organisational complexity as a challenge to research assessment: a case study of the University of Oxford](https://reader036.fdocuments.net/reader036/viewer/2022062406/55b3c77bbb61ebc0548b460d/html5/thumbnails/6.jpg)
Governance and infrastructure
• Complex committee structure • Routes and timing of committees• Systems integration (WIP)• RIM data quality• Lack of unique identifiers and controlled vocabularies
across the entire institution (Divisions, Colleges, NHS etc)
• Devolved model • Internal matters also relevant to external integration• Compliance with standards (eg RIOXX)
![Page 7: Organisational complexity as a challenge to research assessment: a case study of the University of Oxford](https://reader036.fdocuments.net/reader036/viewer/2022062406/55b3c77bbb61ebc0548b460d/html5/thumbnails/7.jpg)
Facts and figuresOxford REF 2014 submission
Submitted to 31 Units of Assessment
(UoA)* (out of 36)
c.2400 research staff
submitted
c. 8500 outputs
submitted
Returned 380,000 data
fields
REF 2008 resulted in £1.9 billion research income over 6
year period
Reported 4892.5 doctoral
degrees awarded
Took 2½ years to manage and compile return
* http://www.ref.ac.uk/panels/unitsofassessment/
![Page 8: Organisational complexity as a challenge to research assessment: a case study of the University of Oxford](https://reader036.fdocuments.net/reader036/viewer/2022062406/55b3c77bbb61ebc0548b460d/html5/thumbnails/8.jpg)
REF complexity illustrative caseOII – Oxford Internet Institute
• A department of Social Sciences Division• Multi/cross disciplinary• Submitted under 9 UoAs (inc psychology,
economics, law, geography)• Difficulty in identifying which are the ‘best’
outputs. Best for whom?
![Page 9: Organisational complexity as a challenge to research assessment: a case study of the University of Oxford](https://reader036.fdocuments.net/reader036/viewer/2022062406/55b3c77bbb61ebc0548b460d/html5/thumbnails/9.jpg)
Quality control for REF Case studies
• Quality control was a challenge• 31 UoAs split across 4 academic divisions• Needed consistency across all submitted case
studies across:– Unit– Within the division– Across all 4 divisions
![Page 10: Organisational complexity as a challenge to research assessment: a case study of the University of Oxford](https://reader036.fdocuments.net/reader036/viewer/2022062406/55b3c77bbb61ebc0548b460d/html5/thumbnails/10.jpg)
REF complexity illustrative case: Impact
• Medical sciences – broad areas such as ‘cancer’ or ‘neuroscience’ over a large academic division
• Do not ‘sit’ in any one department• Administrative staff cannot know all the
details to report impact• Impact can be reported back 20 years
previously. Difficult to identify and track back so far – people leave. Generally cite only 10 years
![Page 11: Organisational complexity as a challenge to research assessment: a case study of the University of Oxford](https://reader036.fdocuments.net/reader036/viewer/2022062406/55b3c77bbb61ebc0548b460d/html5/thumbnails/11.jpg)
The effect on assessment
• Locating data required for assessment
• Dealing with a variety of systems
• Lack of single location for each data type
• Data entry/import (quantity)• Consistency• Size of the operation and
numbers of people involved
Copyright Clare Byers
![Page 12: Organisational complexity as a challenge to research assessment: a case study of the University of Oxford](https://reader036.fdocuments.net/reader036/viewer/2022062406/55b3c77bbb61ebc0548b460d/html5/thumbnails/12.jpg)
• REF OA Service Design Project
• Pilot Jan – June 2015• Simple, short message
– For individuals– For support staff
(including directive)• Comms is key• RS/Bod/Divisional staff
![Page 13: Organisational complexity as a challenge to research assessment: a case study of the University of Oxford](https://reader036.fdocuments.net/reader036/viewer/2022062406/55b3c77bbb61ebc0548b460d/html5/thumbnails/13.jpg)
ORCIDs
• How ORCIDs should help– De facto standard– Common across all units of the University– Supports integration externally– To be required for REF 2020?– Ultimately link individuals, publications, funding– Assist automation
• Difficulties eg College staff not on HR/CUD system• Limitations• ORCIDs at Oxford service
![Page 14: Organisational complexity as a challenge to research assessment: a case study of the University of Oxford](https://reader036.fdocuments.net/reader036/viewer/2022062406/55b3c77bbb61ebc0548b460d/html5/thumbnails/14.jpg)
ORCIDs at Oxford service is live
Released 12 May 2015
![Page 15: Organisational complexity as a challenge to research assessment: a case study of the University of Oxford](https://reader036.fdocuments.net/reader036/viewer/2022062406/55b3c77bbb61ebc0548b460d/html5/thumbnails/15.jpg)
Points for discussion
• Data quality• Systems integration• Standards compliance• Common framework
between agencies• Reduce
administrative burden
• Automation to cope with scale Sally Rumsey, the Bodleian Libraries, 2015