A Study of the Model and Methodology for A Study of the Model and Methodology for

Institute Evaluation in CASInstitute Evaluation in CAS

Guoliang YANG, Xiaoxuan LI

Institute of Policy and Management ， CAS

Management Innovation and Evaluation Research Center, CAS

ContentsContents

1.1. Brief introduction of CAS and Evaluation Practice Brief introduction of CAS and Evaluation Practice

of institutes in CAS of institutes in CAS

2.2. Three-hierarchy Evaluation ModelThree-hierarchy Evaluation Model

3.3. Conclusions and DiscussionsConclusions and Discussions

Brief Introduction of CAS and Institute Evaluation Practice in Brief Introduction of CAS and Institute Evaluation Practice in CASCAS

Brief introduction of CASBrief introduction of CAS

• Chinese Academy of Sciences (CAS) was founded in Beijing on November 1, 1949. Consisting of the Academic Divisions and various subordinate institutions, it is the lead national academic institution in natural sciences, a major advisory body to the government on science and technology related issues, and a national comprehensive research and development center in natural sciences and high technology in China.

• The objective of CAS is to develop the base for scientific research, for training scientific talent and for incubating high-tech industries in China; to become a national scientific think tank and to evolve into a national research institution that boasts “first-class achievements, first-class efficiency, first-class management and first-class talent.”

CASCAS

Academic DivisionsAcademic Divisions CAS HeadquartersCAS Headquarters

Main InfrastructureMain Infrastructure

－ － 17 Large-scale Scientific 17 Large-scale Scientific

Research FacilitiesResearch Facilities

－ － 7 National Labs7 National Labs

－ － 5 Field Stations Networks5 Field Stations Networks

－ － 36 National Engineering 36 National Engineering

CentersCenters

－ － 273 Technology Transfer 273 Technology Transfer

CentersCenters

－ － 317 Journals 317 Journals

－ － 46 National Associations and 46 National Associations and

Societies Societies

–Division of Mathematics and Division of Mathematics and PhysicsPhysics

–Division of ChemistryDivision of Chemistry–Division of Life Sciences and Division of Life Sciences and

MedicineMedicine–Division of Earth Sciences Division of Earth Sciences –Division of Information Technical Division of Information Technical

SciencesSciences–Division of Technological SciencesDivision of Technological Sciences

Members 709Members 709

Foreign Members 53 Foreign Members 53

Committee for Consultation Committee for Consultation

and Review and Review

Committee on Scientific EthicsCommittee on Scientific Ethics

Committee for Science Committee for Science

Popularization and PublicationPopularization and Publication

Institutions Directly Institutions Directly under CASunder CAS

－－ 94 Research Institutes94 Research Institutes

－－ 2 Universities and Schools2 Universities and Schools

－－ 2 Supporting Units 2 Supporting Units

－－ 3 Botanical Gardens 3 Botanical Gardens

－－ 12 Branches12 Branches

－－ 2 Press and Publication 2 Press and Publication

CompaniesCompanies

－－ 1 Assets Management 1 Assets Management

CompanyCompany

－－ 22 Holding Enterprises 22 Holding Enterprises

Distributed in 22 provinces Distributed in 22 provinces and cities over China.and cities over China.

Overview of CASOverview of CAS

• In 1998, with the approval of the Chinese government, the In 1998, with the approval of the Chinese government, the CAS launched the Knowledge Innovation Project (KIP).CAS launched the Knowledge Innovation Project (KIP).

• In the process of the KIP, in recognition of the national In the process of the KIP, in recognition of the national strategic requirements and the world trend in science and strategic requirements and the world trend in science and technology, CAS has made the most profound and technology, CAS has made the most profound and extensive adjustments in its disciplinary deployment and extensive adjustments in its disciplinary deployment and organizational structure since its founding in 1949.organizational structure since its founding in 1949.

Knowledge Innovation Project (KIP)Knowledge Innovation Project (KIP)

Focus on quantitative evaluationFocus on quantitative evaluationFocus on quantitative evaluationFocus on quantitative evaluation Qualitative evaluation Qualitative evaluation Qualitative evaluation Qualitative evaluation

Ranking of institutesRanking of institutesRanking of institutesRanking of institutes Classify by categoriesClassify by categoriesClassify by categoriesClassify by categories

AnnuallyAnnuallyAnnuallyAnnuallyfive year evaluation, annually analysis with five year evaluation, annually analysis with

midterm adjustmentmidterm adjustmentfive year evaluation, annually analysis with five year evaluation, annually analysis with

midterm adjustmentmidterm adjustment

Independently Independently quantitative quantitative evaluation and peer-reviewevaluation and peer-reviewIndependently Independently quantitative quantitative evaluation and peer-reviewevaluation and peer-review

Integration of quantitative quality assessment and Integration of quantitative quality assessment and peer-reviewpeer-review

Integration of quantitative quality assessment and Integration of quantitative quality assessment and peer-reviewpeer-review

Combination of self-evaluation & group Combination of self-evaluation & group discussion and evaluationdiscussion and evaluation

Combination of self-evaluation & group Combination of self-evaluation & group discussion and evaluationdiscussion and evaluationsimple external evaluationsimple external evaluationsimple external evaluationsimple external evaluation

Combination of performance evaluation with Combination of performance evaluation with process management and future developmentprocess management and future developmentCombination of performance evaluation with Combination of performance evaluation with process management and future developmentprocess management and future developmentPerformance evaluation onlyPerformance evaluation onlyPerformance evaluation onlyPerformance evaluation only

Development of Evaluation Concepts and Methods in CASDevelopment of Evaluation Concepts and Methods in CAS

Evaluation Process

Communication review

Peer review

on-site review

Quantitative Analysis

Self-evaluation

Decision Process

Decisionamong group

Finaldecision

Framework of Comprehensive Evaluation (2005)Framework of Comprehensive Evaluation (2005)Framework of Comprehensive Evaluation (2005)Framework of Comprehensive Evaluation (2005)

Period: 5 years

2001 － 2005

2006 － 2010

Self-Evaluation:Self-Evaluation: to find out problems to find out problems and disadvantagesand disadvantages

Communication Review:Communication Review: Assessment on both the Assessment on both the quality of research and quality of research and managementmanagement

Peer Review: Peer Review: Emphasize Emphasize on output qualityon output quality

Quantitative Analysis:Quantitative Analysis: Output performance Output performance and managementand management

On-site Review:On-site Review: To To realize diagnosis and realize diagnosis and guidance of the CAS guidance of the CAS headquarter to institutesheadquarter to institutes

Evaluation ProcessEvaluation ProcessEvaluation ProcessEvaluation Process

S&T outcomes

Management Mechanism

Development Strategy

Planning for next 5 years

Providing evidence for the approval of 5-year Providing evidence for the approval of 5-year

strategic plan strategic plan

Database for status and development analyzingDatabase for status and development analyzing

Related with resources allocation Related with resources allocation

Related with institutes directors’ salaryRelated with institutes directors’ salary

Number of InstitutesNumber of Institutes OutstandingOutstanding GoodGood AcceptableAcceptable

Basic ResearchBasic Research 55 88 11

High-TechHigh-Tech 99 1818 22

Sustainable DevelopmentSustainable Development 1212 2323 22

Evaluation Result and its Application Evaluation Result and its Application (Comprehensive evaluation, 2005)(Comprehensive evaluation, 2005)

ResultResult

ApplicationApplication

IndicatorsIndicatorsIndicatorsIndicators

Ability of developing the S&T productivity

Ability of System Renovating

Ability of leading innovation

Ability of S&T Transferring the outcome

Excellent scientists

R&D inputsR&D inputs

Research infrastructure

S&T OutputsS&T Outputs

Institutional criteriaInstitutional criteria

Operational managementOperational management

Cooperation internally & Cooperation internally & externallyexternally

Strategic decisionStrategic decision

Key S&T projects undertakenKey S&T projects undertaken

Pre-research of the cutting-age Pre-research of the cutting-age field field

Innovative CultureInnovative Culture

dissemination of sciencedissemination of science

Technology DiffusionTechnology Diffusion

International ExchangeInternational Exchange

International CooperationInternational Cooperation

Ability of Absorbing international resources

Based on 24 fundamental indicatorsBased on 24 fundamental indicators Based on 24 fundamental indicatorsBased on 24 fundamental indicators

Quantitative MonitoringQuantitative Monitoring 1 Publications: papers, works

2 Academic talks

3 Awards

4 Patents

5 Copy right of software

6 New medicine, pesticide, new veterinary drugs , new fertilizer , new breed

7 Standards establishment

8 Consultation report

9 Technology transferring

10 Economic benefits of stock-owned enterprise

11 Social and economic benefits of technology transferring

12 graduates education

13 Postdoctoral training and academic visitors

14 Training and science popularization

15 Construction and maintenance large scale research equipment/platform

16 Significant innovation contribution

17 Project

18 Excellent research leaders

19 Infrastructure and capacity

20 Funds

21 Achievement of S&T goal

22 Policy implement

23 Building of innovation culture

24 Education

Adopted the calculation method of GDP indexAdopted the calculation method of GDP index

Capable of Lateral and longitudinal comparisonCapable of Lateral and longitudinal comparison

Lateral comparison provides CAS with information of Lateral comparison provides CAS with information of

institutes’ statusinstitutes’ status

longitudinal comparison enables institutes’ self-monitoringlongitudinal comparison enables institutes’ self-monitoring

Database supportive to analyzing overall development of CASDatabase supportive to analyzing overall development of CAS

Quantitative MonitoringQuantitative Monitoring

m

iij

jj

x

wm＝P

1

100

n

jjiji px＝z

1

Formula

Indicators for S&T Management and Policy GuidanceIndicators for S&T Management and Policy GuidanceIndicators for S&T Management and Policy GuidanceIndicators for S&T Management and Policy Guidance

1.1. Execution of Strategic planExecution of Strategic plan

2.2. S&T layout and adjustmentS&T layout and adjustment

3.3. Projects advising and undertakingProjects advising and undertaking

4.4. Human resourcesHuman resources

Qualitative monitoring on essential management elements of institutesQualitative monitoring on essential management elements of institutes

Promote communication among institutes with similar research fieldsPromote communication among institutes with similar research fields

Managerial Control of institutesManagerial Control of institutes

5.5. Management innovationManagement innovation

6.6. S&T facilityS&T facility

7.7. Innovative cultureInnovative culture

8.8. S&T outputs and their effectsS&T outputs and their effects

Annual self-Annual self-evaluation form of evaluation form of

the institutesthe institutes

8 aspects8 aspects

31 indicators31 indicators

Qualitative MonitoringQualitative Monitoring

Qualitative MonitoringQualitative Monitoring

Quantitative MonitoringQuantitative Monitoring

Monitoring ResultMonitoring Result

Status Report for each Institute

Three-hierarchy Evaluation ModelThree-hierarchy Evaluation Model

Yearly quantitative and qualitative monitoring

Peer review

At the bottom hierarchy are indicators that can reflect common characters of the institutes, including annual quantitative & qualitative monitoring. This part is the improvement of former monitoring essentially.

The second hierarchy are the key indicators and benchmarks, which reflects the characters of certain category of institute. It can be used in both diagnosing evaluation for individual institute and comparison between institutes of the same category. Newly developed indicators reflect individual characteristics of each institute.

The top hierarchy is qualitative evaluation by experts review, and it is also a hierarchy of individual evaluation. Using MPG and RIKEN as references, Each institute will have an advisory committee to provide suggestions for strategies and layouts periodically. This layer is individual for each institute.

Quantitative Qualitative

Three-hierarchy Evaluation ModelThree-hierarchy Evaluation Model

Key indicators

Key indicators of Pilot InstitutesKey indicators of Pilot Institutes

Common indicators:1 、 S&T Talents2 、 funding per FTE3 、 Awards

Institute of Institute of High Energy High Energy PhysicsPhysics

Institute of Institute of Computing Computing TechnologyTechnology

Cold and Arid Cold and Arid Regions Regions Environmental Environmental & Engineering & Engineering Research Research InstituteInstitute

Shanghai Shanghai Institute of Institute of Technical Technical PhysicsPhysics

Dalian Dalian Institute of Institute of Chemical Chemical PhysicsPhysics

Institute of Institute of MicrobioloMicrobiologygy

Institute of Institute of Physics Physics

Invited Speak

High-quality papers

Undertaking important tasks

Outcome Transfer

Building andrunning of large

facilities

High-qualitypapers

Platform of

resource

and data

Outcome transfer

High-qualitypapers

Undertaking Important tasks

Platform of

resource

and data

High-quality

papers

Outcome

transfer

Engineering Application and Demonstration

Benchmarks

Based on analysis of the international research institutions and

experimental analysis of the characteristics of pilot institutes.

The setting of benchmark: there are four methods in our research:

—— the more the better;

——different standards based on the world’s criteria;

——based on the development situation of institutes;

——based on the experience.

Conclusions and DiscussionsConclusions and Discussions

Conclusions and Discussions(1)Conclusions and Discussions(1)

Conclusions

• Key indicators and benchmarks regarding different types of

institutes were proposed, including key quantitative indicators and

benchmarks as well as key qualitative indicators and anchoring

method. Thereby, three hierarchy evaluation model has been set

up.

• The new model put more emphases on the character of evaluation

by category and individual features of institutes.

• In summary, this model has developed Quality-orientated

Comprehensive Evaluation system in methodology.

Conclusions and Discussions(2)Conclusions and Discussions(2)

Discussions• Institutes Classification: Including how to classify the institutes into

different categories? What’s the proper scale for the category?

Whether the demonstration of the category features allows the

institute to choose different types evaluation scheme by themselves?

• Key Indicators and Benchmarks: Should the benchmarks be set up

on the basis of the individual features of the institute or should

universal benchmarks be adopted in the institutes of the same

category?

• If key indicators and benchmarks be used in evaluation, how to

reorganize management system at CAS level to meet the

requirements of this evaluation method? Who can approve the key

indicators and their benchmarks?

ThanksThanks

Contact information:

www.casipm.ac.cn

glyang@casipm.ac.cn