Kil-Woo Lee, Yu-Cheong Chon, Jung-Min Song€¦ · ‒ Introduction to the Analytic Hi erarchy...
Transcript of Kil-Woo Lee, Yu-Cheong Chon, Jung-Min Song€¦ · ‒ Introduction to the Analytic Hi erarchy...
PICMET ’13 Conference
Kil-Woo Lee, Yu-Cheong Chon, Jung-Min Song
Korea Institute of S&T Evaluation and Planning (KISTEP)
PICMET ’13 Conference
The objective of this study is to examine the relative importance of the attributes of national R&D output index. Relative importance data were collected from AHP(Analytic Hierarchy Process) survey of experts who have responsibility of R&D output management in universities and institutes located in Seoul and Dae-jeon cities in Korea. Five attributes were basic research output, application development research output, economic outcome, social outcome, and human-resource outcome. We found that economic outcome was the attribute that affected researcher preference. The findings of this study could be applied as a basic information in terms of decision making process for the efficient management of national R&D performance.
1872
2013 Proceedings of PICMET '13: Technology Management for Emerging Technologies.
PICMET ’13 Conference
Contents
Problem Description
Solution Procedure: AHP
Computational Results
3/27
PICMET ’13 Conference
[Negative Balance of Technology Trade]
1995 1999 2004 2009
1,835 million $
2,493 million $2,731 million $
4,856 million $
[Research Productivity & Sum of Tech. Transfer Revenue]
1.48%
4.32%
151million $
2,280million $
: Sum of Technology Transfer Revenue
:Research Productivity
2010 2010 2009
• Lower Application of R&D Performance in Korea
Utilizations and diffusions of research performance are
emerged as important issues
Research Background
SolutionProblem Result
• Policy of Technology Transfer and Commercialization
-『 Acton theperformanceevaluationandmanagementofnational researchanddevelopmentprojects』
-『 2nd Basic Research Plan of Research Performance management and Application (2011~2015) 』
4/27
1873
2013 Proceedings of PICMET '13: Technology Management for Emerging Technologies.
PICMET ’13 Conference
Performance Management
System
Research for R&D Performance Management System
Research for R&D Performance Analysis Method
Need for Analyzing Relative Importance of R&D Performance
Index
Expansion of R&D Performance Transfer &
Commercialization
Maximize Effective of R&D
Performance Management
Performance Evaluation
Management& Application
of R&D Performance
Existing Proposed Research
Necessities of Research
Necessities of Research
Improvement Strategy Policy IssueImprovement Strategy
Improvement Strategy PolicyPolicy IssueIssue
SolutionProblem Result
R&D Performance Management System
Response Strategy System Centered to Issue
Understand Level from Analyzation
Policy Environments
TechnologySupply
Technology Demand
Transfer Infra
5/27
PICMET ’13 Conference SolutionProblem Result
Efficiency
Effectiveness
Enforceability
Political Acceptability
Equity
Compatibility
FlexibilityMAUT
Multi-Attribute Utility Theory
AHPAnalytic Hierarchy Process
PROMETHEEPreference Ranking Organization
METHod for Enrichment Evaluation
Reason for Choosing AHP
Criteria for Policy Assessment
Objective of This Research
• Structuralize and analyze the problems
• Measure relative importance or preference of
each factors
• Gather the opinions of the hands-on workers
Multi-criteria Decision Making
6/27
1874
2013 Proceedings of PICMET '13: Technology Management for Emerging Technologies.
PICMET ’13 Conference
Literature Review
AHP Survey ‒ Introduction to the Analytic Hierarchy Process (T. L. Satty, 1990)‒ Propose a reliable AHP method using Delphi technique, which aims to incorporate the relative
expertise of panel and to increase their consensus (Song, et al. 1999)
Definition of R&D Performance‒ Propose main types of evaluation studies (A. Airaghi, et al. 1999)‒ Cross tabulation of important impact dimensions of public research and technological
development spending is suggested that depicts different classes of effects with regard to time and worlds (L. Georghiou, et al. 2001)
Application to other areas‒ Development of Evaluation Model for Knowledge Management Performance (Jung, et al, 2010)‒ Evaluation Items in Technology Transfer (Lee, et al. 2011) ‒ Information Security (Sung, et al. 2010)
SolutionProblem Result
7/27
PICMET ’13 Conference
Contents
Problem Description
Solution Procedure: AHP
Computational Results
8/27
1875
2013 Proceedings of PICMET '13: Technology Management for Emerging Technologies.
PICMET ’13 Conference
A
A
B
B
C
C
9 8 7 6 5 4 3 2 1 2 3 4 5 6 7 8 9
Increasing Importance to the Left Increasing Importance to the Right
SolutionProblem Result
To Investigate the relative importance between the decision - making factors‒ Possible to analyze the multi-criteria decision-making‒ Applicable to all areas which need to decision-making
AHP survey process‒ Setting Decision-making Hierarchy ‒ Collecting data through Pair-wise Comparison‒ Estimating of Relative Importance between the decision - making factors‒ Driving overall ranking by summing weight between the decision - making factors
9/27
PICMET ’13 Conference
Conceptual Model
Step 1Step 1
Step 2Step 2
Step 3Step 3
Step 4Step 4
Exploratory
Solid
Evaluation studies Time
ShortMedium
LongWorld
ScienceEconomy
Society
Research Performance
Economical Effect Ripple EffectEconomical
Effect
Output Impact
Research Performance
Output
Impact
Economical Impact
Workforce Impact
Basic Research Performance
Application Research Performance
Social Impact
SolutionProblem Result
10/27
1876
2013 Proceedings of PICMET '13: Technology Management for Emerging Technologies.
PICMET ’13 Conference SolutionProblem Result
– Georghiou, L., Rigby, J. and Cameron, H., “Assessing the Socio-economic Impacts of the Framework Programme, University of Manchester, pp. 169-172, 2001.
Definition Example
Solid Impact Studies
More characteristic evaluation approaches, take place ex post and are concerned with a full range of effects
Legal impact, Economic Impact, et al.
Exploratory Impact Studies
Examine potential effects ex ante but have a more economic and quantitative focus
Paper, Patent, Industrial property, et al.
- Airaghi A., Busch N.E., Georghiou L., Kuhlmann S, Ledoux M.J., van Raan A.F.J. and Viana Baptista J., “Options and Limits for Assessing the Socio-Economic Impact of European RTD Programmes,” ETAN, Commission of the European Communities, 1999.
Outcome Impact
Short-term Medium-term Short-term Medium-term
Scientific and Technological
New/improved techno logical know-how Increased productivity Increased
competitiveness
Economic New/improved products, services and processes
Increased sales and market share
New markets; development of SMEs
Societal Job creation better infrastructure, healthcare, new regulation
Increased prosperity satisfaction equality
Stable, cohesive society with highly participative
citizen
11/27
PICMET ’13 Conference
제 1세대(~1995)
제 1세대(~1995)
SolutionProblem Result
Effe
ct o
f Im
pact
Range of Performance
Direct Research Performance
Direct Economical Effect
Indirect Economical Effect
Indirect Ripple Effect
Smaller Bigger
Expl
orat
ory
Solid
Performance from R&D(ex. Patent, Paper)
Economical Profit from Technology Transfer & Commercialization
Economical Ripple Effect (ex. Value-added of Enterprise, Positive Balance of Technology Trade)
Scientific, Technological, Social, and Cultural Ripple Effect(Without Economical Ripple Effect)
12/27
1877
2013 Proceedings of PICMET '13: Technology Management for Emerging Technologies.
PICMET ’13 Conference
Index for Research
Performance Management
Direct Effect
Ripple Effect
Economical Impact
WorkforceImpact
Basic Research Performance
Applicable Research
Performance
Social Impact
SolutionProblem Result
13/27
PICMET ’13 Conference SolutionProblem Result
14/27
1878
2013 Proceedings of PICMET '13: Technology Management for Emerging Technologies.
PICMET ’13 Conference
Direct Effect
Ripple Effect
Economical Impact
WorkforceImpact
Basic Research Performance
Application Research
Performance
SocialImpact
Level 1 Level 2
Science Citation Index
Highly cited scientific
articles(top 1%)
Nature science cell
The number of Highly Cited Researchers
Nobel prizes
Patent applications per thousand total
researchers
Number of "triadic" patent
famillies
Highly cited patent applications to the U.S. PO(top 1%)
Highly cited patent applications to the
EPO(top 1%)
Number of patent with co-
applicants
Value added of innovation-intensive
manufacturing sectors
High-tech exports(% of manufactured exports)
Share of high technology in total exports (%)
Technology balance of payments
The World Top 200 universities
Global 1000 leading companies
Venture capital investment
GERD as a percentage of GDP
Graduation rates at doctorate level
Graduation rates at doctorate level
Science and engineering graduates at doctorate
level
Science and engineering graduates at doctorate
level
Proportion of the national field of science and technology degree
Proportion of the national field of science and technology degree
Scientific and technical personnel ratio of
business professionals
Scientific and technical personnel ratio of
business professionals
SolutionProblem Result
15/27
PICMET ’13 Conference
Level 1Level 2
Factor Reference
Basic Research Performance
Science Citation Index COSTII, Web of Science
Highly cited scientific articles(top 1%) ESI DB
Nature science cell Web of Science
The number of Highly Cited Researchers Thomson Reuters ISIHIGHLYCITED.COM
Nobel prizes IMD World Competitiveness
ApplicationResearch
Performance
Patent applications per thousand total researchers WIPO(World Intellectual Property Organization)
Number of "triadic" patent famillies OECD Main Science and Technology Indicators
Highly cited patent applications to the U.S. PO(top 1%) National R & D performance analysis reports
Highly cited patent applications to the EPO(top 1%) OECD STI Scoreboard
Number of patent with co-applicants COSTII, OCED iLibrary
Economical Impact
Value added of innovation-intensive manufacturing sectors EU Innovation Scoreboard, OECD STI Scoreboard
High-tech exports(% of manufactured exports) COSTII, World Bank Science & Technology
Share of high technology in total exports (%) COSTII, OECD Science, Technology and Industry Outlook
Technology balance of payments COSTII, OECD Main Science and Technology Indicators
SocialImpact
The World Top 200 universities COSTII, Times Higher Education-QS World University Rankings
global 1000 leading companies COSTII, Forbes, Global 2000 Leading Companies
Venture capital investment COSTII, OECD Science, Technology and Industry Outlook
GERD as a percentage of GDP COSTII, OECD STI Scoreboard
WorkforceImpact
Graduation rates at doctorate level Science and Engineering Index, OECD STI Scoreboard
Science and engineering graduates at doctorate levelEU Innovation Scoreboard, Science and Engineering Index,
OECD STI Scoreboard
Proportion of the national field of science and technology degree COSTII, Scienceand Engineering Index, OECDSTI Scoreboard
Scientific and technical personnel ratio of business professionals EU Innovation Scoreboard, COSTII, OECD STI Scoreboard
SolutionProblem Result
16/27
1879
2013 Proceedings of PICMET '13: Technology Management for Emerging Technologies.
PICMET ’13 Conference
Pair-wise Comparison
Questionnaire Configuration
SolutionProblem Result
AHP survey for research performance management indicators- Literature review and exports' advice related in research performance management
and application
- Respondents represent points as relative importance for each index
Period: 2011. 10. 15. ~ 2011. 12. 1. (for 6 weeks) Interview R&D Performance management representatives in Korean
research institutions
Develop matrix of pairwise ratios to estimate of relative weights
SurveyBelong Number
University 6
Government Research Institute 13
Enterprise Institute 2
Techno Park 4
Sum 25
- These members are experts of Korea
R&D performance management at each
research centers
17/27
PICMET ’13 Conference
Contents
Problem Description
Solution Procedure: AHP
Computational Results
18/27
1880
2013 Proceedings of PICMET '13: Technology Management for Emerging Technologies.
PICMET ’13 Conference
17.3
14.6
29.5
18.2
20.4
0 5 10 15 20 25 30 35
Workforce Impact
Social Impact
Economical Impact
Application Research
Performance
Basic Research Performance
Relative Importance(%)
SolutionProblem Result
An Indicator of the Consistency of the Respondents( Consistency Ratio <0.1 ) → Can be trusted
19/27
PICMET ’13 Conference
11.5
11.8
20.3
27.1
29.3
0 5 10 15 20 25 30 35
Science Citation Index
Nobel prizes
Science Citation Index
The number of Highly Cited
Researchers
Highly cited scientific articles(top
1%)
Relative Importance(%)
SolutionProblem Result
An Indicator of the Consistency of the Respondents( Consistency Ratio <0.1 ) → Can be trusted
20/27
1881
2013 Proceedings of PICMET '13: Technology Management for Emerging Technologies.
PICMET ’13 Conference
13.8
18.1
19.8
23.6
24.7
0 5 10 15 20 25 30
Number of patent with co-
applicants
Patent applications per thousand
total researchers
Highly cited patent applications to
the USPO(top 1%)
Highly cited patent applications to
the EPO(top 1%)
Number of "triadic" patent families
Relative Importance(%)
SolutionProblem Result
An Indicator of the Consistency of the Respondents( Consistency Ratio <0.1 ) → Can be trusted
21/27
PICMET ’13 Conference
22.4
23.3
27.0
27.3
0 5 10 15 20 25 30
Value added of innovation-
intensive manufacturing sectors
Technology balance of payments
Share of high technology in total
exports (%)
High-tech exports(% of
manufactured exports)
Relative Importance(%)
SolutionProblem Result
An Indicator of the Consistency of the Respondents( Consistency Ratio <0.1 ) → Can be trusted
22/27
1882
2013 Proceedings of PICMET '13: Technology Management for Emerging Technologies.
PICMET ’13 Conference
17.9
21.1
23.2
37.8
0 5 10 15 20 25 30 35 40
The World Top 200 universities
Venture capital investment
global 1000 leading companies
GERD as a percentage of GDP
Relative Importance(%)
SolutionProblem Result
An Indicator of the Consistency of the Respondents( Consistency Ratio <0.1 ) → Can be trusted
23/27
PICMET ’13 Conference
11.9
16.0
34.8
37.3
0 5 10 15 20 25 30 35 40
Graduation rates at doctorate
level
Proportion of the national field of
science and technology degree
Science and engineering
graduates at doctorate level
Scientific and technical personnel
ratio of business professionals
Relative Importance(%)
SolutionProblem Result
An Indicator of the Consistency of the Respondents( Consistency Ratio <0.1 ) → Can be trusted
24/27
1883
2013 Proceedings of PICMET '13: Technology Management for Emerging Technologies.
PICMET ’13 Conference
Level 1 Level 2 Priority
Factor Importance(%) Factor Importance(%) Importance(%) Ranking
Basic
Research
Performance
20.4
Science Citation Index 11.5 2.35 21
Highly cited scientific articles(top 1%) 29.3 5.98 7
Nature science cell 20.3 4.14 12
The number of Highly Cited Researchers 27.1 5.53 8
Nobel prizes 11.8 2.41 20
Application
Research
Performance
18.2
Patent applications per thousand total researchers 18.1 3.29 15
Number of "triadic" patent famillies 24.7 4.50 10
Highly cited patent applications to the U.S. PO(top 1%) 23.6 4.30 11
Highly cited patent applications to the EPO(top 1%) 19.8 3.60 13
Number of patent with co-applicants 13.8 2.51 19
Economical
Impact29.5
Value added of innovation-intensive manufacturing sectors 22.4 6.61 4
High-tech exports(% of manufactured exports) 27.3 8.05 1
Share of high technology in total exports (%) 27.0 7.97 2
Technology balance of payments 23.3 6.87 3
Social Impact 14.6
The World Top 200 universities 17.9 2.61 18
global 1000 leading companies 23.2 3.39 14
Venture capital investment 21.1 3.08 16
GERD as a percentage of GDP 37.8 5.52 9
Workforce
Impact17.3
Graduation rates at doctorate level 11.9 2.06 22
Science and engineering graduates at doctorate level 34.8 6.02 6
Proportion of the national field of science and technology degree 16.0 2.77 17
Scientific and technical personnel ratio of business professionals 37.3 6.45 5
SolutionProblem Result
25/27
PICMET ’13 Conference
Result Summary
Further Research
Conclude
SolutionProblem Result
26/27
1884
2013 Proceedings of PICMET '13: Technology Management for Emerging Technologies.
PICMET ’13 Conference
Reference
27/27
PICMET ’13 Conference
Thank you!
Email : [email protected]
1885
2013 Proceedings of PICMET '13: Technology Management for Emerging Technologies.