http://cerify.ukoln.ac.uk
Mahendra Mahey (UKOLN)Stephanie Taylor (UKOLN)Niamh Brennan (TCD)Kevin Kiely (TCD)
Eva
luat
ion
of
CE
RIF
an
d C
RIS
Imp
ort
an
d E
xpo
rt o
f D
ata
and
Fa
cili
tati
on
of
dat
a ex
chan
ge
Insti
tutio
ns s
ee v
alue
of C
ERIF
and
CRI
S, T
ell t
heir
regi
onal
insti
tutio
ns a
t reg
iona
l an
d na
tiona
l eve
nts.
CERI
F an
d CR
IS E
ngag
emen
t Inc
reas
es
Incr
easi
ng le
vel o
f Eng
agem
ent w
ith a
CRI
S /
CERI
F
Research Information ManagementProcess Mapping and Gap Analysis
Current and past CRIS/CERIF/RIM projects
InCites
Research in View
TCD CRIS TCD CRIS
Thomson CRIS
Research in View
and InCites
Thomson CRIS
Research in View
and InCites
Other, e.g.
PURE?
Other, e.g.
PURE?
Data Surgery
CERIFTCDXML2002
CERIFTCDXML2002
CERIFTCDXML2008
CERIFTCDXML2008Mappings
and Cross Walks
relevant CERIF Schema
Mappings and Cross
Walks relevant
CERIF Schema
PriorityProcess Areas –
‘as is’ and ‘to be’
PriorityProcess Areas –
‘as is’ and ‘to be’
CERIF HealthCheck
CERIF HealthCheck
CERIFy ApproachPeople Centered Socio Cognitive
Reducing duplication
TCD – CERIF People and Publications
User requirements elicitationPrioritization of RIM Processes / Identification,
Mapping, Gap and Stakeholder Analysis
Questionnaires Site Visits
Business Process Analysis‘as is’ and ‘to be’
4 Priority Processes
Data from at least 2
partners
Data from at least 2
partners
http://cerify.ukoln.ac.uk
Mahendra MaheyStephanie TaylorRosemary RussellMichael DayTalat Chaudhri
Niamh BrennanKevin KielyJimmy TangRoisin Croker
InCites Exchange of Data
http://cerify.ukoln.ac.uk
How Is This Used?
http://cerify.ukoln.ac.uk
• RAE requirement• Comparison with other universities (e.g.
Russell Group)• Global comparisons (updated annually)• Citation information generated from this tool
provides a lot of key information
Collection of Data
http://cerify.ukoln.ac.uk
• Two-way process between Thomson Reuters & individual institution
• Requires a lot of work by institution• Institutional data collected in local database of staff
research information
User Issues
http://cerify.ukoln.ac.uk
• “There is a lot of effort involved in understanding the data - there would be much value in standardising the data.”
• E.g. - problems with author ID - duplicates & null fields
User Requirements
http://cerify.ukoln.ac.uk
• “If the data was in a standardised format it would make life much easier!”
• Nightly updates• Internationalised data sets• Relational database with mapping
research output onto staff ID number for returned data
• Better mapping to institutional schools structure
Measures of Esteem
http://cerify.ukoln.ac.uk
How Is This Used?
http://cerify.ukoln.ac.uk
• RAE requirement• Personal review for promotion process• Assessment & benchmarking - internal & external• As key information in the assessment in judging the
overall quality of research outputs• Drafting REF documentation• Part of institution-wide annual planning cycle
Collection of Data
http://cerify.ukoln.ac.uk
• From individual researchers on adhoc basis• Systematic for RAE• Continuous tracking for REF• Ask individual schools to provide this on regular basis
User Issues
http://cerify.ukoln.ac.uk
• Process is ‘woolly’ - difficult to provide meaningful data• Perception varies depending on given point in a career path• Citations alone are not accurate enough• Context is needed to make sense of raw data• Defining this process is very difficult• Researchers can undersell themselves and don’t make
information available
User Requirements
http://cerify.ukoln.ac.uk
• Systematic way of capturing this is required• Bringing a bibliometrics perspective to esteem-related
information • Take into account that one measurement doesn’t fit all• Acknowledge differences in different subjects• When one size doesn’t fit all, find where to plug the gaps in
information • Personalised audit tool to be built into RIM system
Thomson Reuters data exchange – what’s InCites?
Why CERIFy it?• The community expressed interest in this process• Current data is unsatisfactory ; the dataset is a specific view
– Institutional Citation Report is ‘historic’: based on affiliation at the time of authorship. – Does not show Schools, Research Centres etc. (WoS API will not provide this either).– Author, institution name variants (no automatic DAI in WoS/InCites)– Missing papers (aside from general coverage issues)– But.. institutions want the metrics + institutional structure in InCites + to embed data in local CRIS/IR:
Sample
• Current exchange process is unsatisfactory: non-standard schema, heavy demands on institution, disappointing results
• Opportunity to create a CERIF data model for exchange of people, institutional affiliations, publication data & metrics
• Opportunity to demonstrate a 360⁰ data exchange using CERIF• If successful, it should help to show value and help to engage institutions with CERIF• Model can be re-used for other similar exchange processes
Process 1: InCites ‘as is’ – Based on Queen’s University Belfast
Process 1: InCites ‘to be’ – Generic (based on CERIFy Data Surgery)
Thomson Reuters
MIS/CRIS developers
Research Office / Library
DATA EXCHANGE PARTNERS
Business Intelligence Officers/Institutional Analysts
1. Online export of CERIF-based data to TR from RIM/CRIS via web services (nightly updates)
5. Reporting on Research 8.1 Internal Reporting 8.2 External Reporting
[6] 5 TR send report to institution on results of matching process (including matched Author ID)
3 Institutional Data automatically matched with TR records for authors & publications
2. TR import and update deltas
4. InCites 'Institutional Profile' established and updated automatically
[7] TR data integrated with CRIS data (if required) Note: may use TR API instead or alongside
EXTE
RNAL
EXTE
RNAL
Data drilldown
Creation of CERIF Data Model for ‘InCites’ Exchange
Over to Kevin…
Where we are – results• CERIF Data model• Data exchange next week!• Includes: extended ‘publication types’ list• Multiple identifiers (Researcher ID; Institutional ID, HESA ID,
Publication ID etc• Full metrics = standardisation
Next stepsQUB
Thomson Reuters are working on automating the process using Web Services…
Notes on Esteem & REF
“‘Esteem’ is no longer included as a distinct element in the assessment” 67. All information provided by HEIs in submissions to the 2014 REF must be capable of verification [note: can CERIF help
provide the reference or source of data?] Under ‘Definition of Impact for the REF’: “a. Impacts on research or the advancement of academic knowledge within the higher education sector (whether in the UK
or internationally) are excluded. (The submitted unit’s contribution to academic research and knowledge is assessed within the ‘outputs’ and ‘environment’ elements of REF.)”
Part 3 Section 5: Environment template (REF5) 183. Information is required about the research environment for each submitting unit relating to the period 1 January 2008
to 31 July 2013. Each submission must include a single completed REF5 form, consisting of the following sections: • Overview. • Research strategy. • People, including: – Staffing strategy and staff development. – Research students. • Income, infrastructure and facilities. • Collaboration and contribution to the discipline.
From: Assessment framework and guidance on submissions, HEFCE, July 2011 http://www.hefce.ac.uk/research/ref/pubs/2011/02_11/02_11.pdf
REF5 ‘Environment’:•Maximum score reduced from 30%(RAE 2008) to 15% (REF 2014)•Indicators of Esteem = Collaboration & Contribution to the Discipline
Research Monitoring Team
HoD/DDR
DATA EXCHANGE PARTNERS
Deans
Academic
2. Details of form transferred to RAD
6. Access electronic copy deposited within RAD SharePoint site Departmental folder
3. Pdf report produced
4. Hardcopies printed for Research Monitoring meetings
1 Completes ‘Other Activities’ annual form on Research Activity Database (RAD) SharePoint interface
5. Electronic copy deposited within RAD SharePoint site Departmental document folder
7. Hardcopies printed for Research Monitoring meetings
Process 1: Esteem ‘as is’ – Based on Aberystwyth University
Research Monitoring Team
HoD/DDR
Deans
4. Reports generated by HoDs, Deans etc.
3. CRIS / RIM system data is queried and reported by Research Monitoring Team
6. Data Exported to external agencies as required
Academic
2. Academic updates CRIS/RIM with other elements of Measures of Esteem
MIS/Library
1. CRIS/RIMacademic profile is automatically populated with elements of 'Measures of Esteem'
External agencies/Funders
5. Reports generated by HoDs, Deans etc.
Process 1: Esteem ‘to be’ – CERIFy Data Surgery
Indicators of Esteem:based on CERIFy DataSurgery, current practiceIn partner institutions + a number of external sources
Indicators of Esteem - the extended version:based on original CERIFy IE, With additions to publications & outputs
Please see spreadsheet for latest list CERIFy Indicators ofEsteem
OrgUnitProjectPersonPubl.Etc.
Link tables
cfOrgUnitId etc.cfEsteemIndicatorId
cfClassIDcfClassSchemeId
cfStartDatecfEndDate
Link tables
cfOrgUnitId etc.cfEsteemIndicatorId
cfClassIDcfClassSchemeId
cfStartDatecfEndDate
Link tables
cfOrgUnitId etc.cfEsteemIndicatorId
cfClassIDcfClassSchemeId
cfStartDatecfEndDate
Link tables
cfOrgUnitId etc.cfEsteemIndicatorId
cfClassIDcfClassSchemeId
cfStartDatecfEndDate
OrgUnitProjectPersonPubl.Etc.
Esteem Indicatorlanguage table
Link table
cfEsteemMeasureId cfEsteemIndicatorId
cfStartDatecfEndDate
Esteem Measure table
cfEsteemMeasureId cfEsteemMeasureKind
cfCountIntegercfValueFloatingPoint
cfValueJudgemental NumericcfCountIntegerChange
cfValueFloatingPointChangecfValueJudgementalChange
cfValueJudgementalText
Esteem Indicator table
CERIFy Esteem Indicators Model – based on MICE (Impact) Note: current focus for Esteem is more on Indicators rather than on Measures
NB: this adaptation has yet to be finally validated by Brigitte
Esteem Indicators mapped to UK RAE 2008, Australian ERA
Esteem Indicators for REF5: Contribution to Discipline/Collaborations
The Issue with Esteem Indicators [as with Impact]• The processes associated with the capture and reporting of Esteem Indicators
can benefit more if there is an effort to standardise at least some of the data required.
• Most of the data is not currently available from sources other than narratives or reports supplied by members of academic staff
• To reduce time and effort & assist with verification:- Where data can be imported from external sources it should.- ‘Authority-controlled’, taxonomies and definitions lists should be agreed and
built into the systemNOTE: the RAE 2008 submissions under ‘Research Environment, Measures of Esteem’ provide an extremely valuable source of information on the type of information provided by institutions in this area. This body of knowledge is not available for other REF-relevant areas such as ‘Impact’. The RAE 2008 ‘Measures of Esteem’ is a particularly rich source of data from the disciplinary viewpoint. The analysis of these ‘texts’ has great potential to assist with developing the type of ‘authority lists’/taxonomies described above. The following slides show the results from analysing the RAE 2008 University of Bath ‘Measures of Esteem’ submission in the field of Pharmacy. This text was analysed using Many Eyes and Open Calais. The results of using Open Calais for semantic analysis and entity extraction even for this small sample are very positive.
Acknowledgements• The CERIFY Team & Partners• Brigitte Jorge, EuroCRIS• Thomson Reuters InCites CERIFy team, Philadelphia• Bo Alroe, Atira /UK Pure User Group (Esteem Indicators)• http://cerify.ukoln.ac.uk
Top Related