Post on 18-Dec-2015
S. Kuhlmann, CGEE International Seminar, Rio de Janeiro 2007 page 1
Rationales and evolution of public 'knowledge policies'
in the context of their evaluation
Seminário Internacional CGEE“AVALIAÇÃO DE POLÍTICAS DE CIÊNCIA, TECNOLOGIA E INOVAÇÃO”
- Diálogo entre Experiências Internacionais e BrasileirasRio de Janeiro, 3-5 December 2007
Professor Stefan KuhlmannUniversity of Twente, The Netherlands
S. Kuhlmann, CGEE International Seminar, Rio de Janeiro 2007 page 2
‘Knowledge policy’ issues Growing dependence of socio-economic development and welfare on sustainable
‘knowledge base’, in particular science, technology, innovation – and education
Internationalisation of industrial research, not caring about national borders
International search for creative, highly skilled people; mobility of top researchers
Knowledge production: (re-) discovery of a “mode 1 – mode 2 transition" (M. Gibbons et al. 1994); advanced technology and innovation: "fusion” of heterogeneous trajectories (Kodama 1995)
Emergent`, generic S/T (e.g. Nano) withm many promises: ‘New actors’, NGO, ELSA , ‘users’ matter
Public research and innovation policy: push for efficiency, evidence and evaluation; ‘bureaucratised’ semi-industrial R&D (Ziman, 2001, 82)
Search for ‘intelligent’ policy designs: ‘Systemic’ policy instruments; concern about ‘policy mix’ and ‘governance’
Multi-level arenas and governance; [EU: large number of policy actors on national, regional, and transnational levels; re-shuffling of institutional research landscape]
S. Kuhlmann, CGEE International Seminar, Rio de Janeiro 2007 page 3
1. ‘Innovation system’ concept; evolution of policy instruments and rationales
2. Detour: ‘One size does not fit all’
3. Weaknesses of present research and innovation policy
4. Stakeholders‘ diverging interests, policy arena, governance
5. Role of ‘Strategic Intelligence’
6. Options and limitations of ‘impact’ evaluation
7. Evaluation methodologies
Overview: 7 steps
S. Kuhlmann, CGEE International Seminar, Rio de Janeiro 2007 page 4
1. ‘Innovation system’ concept; evolution of policy instruments and rationales
S. Kuhlmann, CGEE International Seminar, Rio de Janeiro 2007 page 5
The innovation system model: a heuristic tool
The potential reachof public policies ...
Framework ConditionsFinancial environment; taxation and incentives; propensity to innovation
and entrepreneurship; mobility ...
Education and Research System
Professional education and
training
Higher education and research
Public sector research
Industrial System
Large companies
Mature SMEs
New, technology- based firms
IntermediariesResearchinstitutesBrokers
Consumers (final demand)Producers (intermediate demand)
Demand
Banking, venture capital
IPR and information
Innovation and business support
Standards and norms
Infrastructure
PoliticalSystem
Government
R&I policies
Governance
Source: Kuhlmann & Arnold 2001
Co-evolution
S. Kuhlmann, CGEE International Seminar, Rio de Janeiro 2007 page 6
Innovation system (IS) - a heuristic
IS = analytical heuristic (NOT normative perspective)!
IS = “biotopes” of such institutions engaged in scientific research, the accumulation and diffusion of knowledge, which educate and train the working population, develop technology, produce innovative products and processes, and regulate and distribute them.
IS extend over schools, universities, research institutions (education and science system), industrial enterprises (economic system), the politico-administrative and intermediary authorities (political system) as well as the formal and informal networks of the actors of these institutions.
One can conceptualise national, regional, sectoral, and technological IS.
Each IS is different. Sustainable innovation systems develop their special profiles and strengths only slowly, in the course of decades or centuries. They are based on evolving exchange relationships among the institutions of science and technology, industry and the political system (= co-evolution).
S. Kuhlmann, CGEE International Seminar, Rio de Janeiro 2007 page 7
Researchandinnovationpolicyinstruments
Instruments in a narrow sense Instruments in a broader sense
1. Institutional funding National Research Centres Research Councils ( in Germany DFG;
Max Planck Society) Applied Research and Techn. Deve-
lopment Organisations (in Germanye.g. Fraunhofer Society)
Universities and other Higher Educa-tion Institutions
Others
2. Financial incentives Indirect promotion programmes (e.g.
CIM) Technology promotion programmes
(cooperative R&D projects) Risk capital
3. Other infrastructure and technologytransfer mechanisms
Information and consultancy for SMEs Demonstration centres Cooperation, networks, people
4. Public demand and procu-rement
5. Systemic measures Long-term visions; technology foresight Technology assessment Regulation, Standards, IPR
6. (Continuing) education; training
7. Public policy Competition policy (De-) Regulation Public stimulation of private demand
S. Kuhlmann, CGEE International Seminar, Rio de Janeiro 2007 page 8
Rationales of research and innovation policies Mission rationale, e.g. defence; energy production and conservation, medicine and
public health, space, and agriculture; S/T for "public goods" as an aim of public investments (OECD 1995), e.g. “sustainable development”
Market failure rationale (externalities; indivisibilities; risks)
Co-operative policy rationale, e.g. "Specific Programs" under the EU Framework Programs, or the "Verbundforschungsförderung" (co-operative R&D between public sector institutes, universities, and industry) in Germany
“System failures” rationale, need for structural change in the innovation system, e.g. government initiatives aiming at overcoming sclerotic institutions and procedures e.g. in the academic research system
Counter rationale ”government failures”, e.g. institutional inertia, lack of reliable information (on efficiency and impacts of policies etc.), lack of continuity and long-term perspective, red-tape procedures, rivalry of bureaucracies
Note: Actual policymaking only seldom follows such rationales !
S. Kuhlmann, CGEE International Seminar, Rio de Janeiro 2007 page 9
IS ‘success factors’ and political support
Factor Public policyEntrepreneurial activities
Corporate governance; insolvency legislation; education
Knowledge creation Funding of basic and applied research; (higher) education and training
Knowledge diffusion through networks
Support for R&D and innovation networks (industry, academia, et al.) and clusters; multi-actor programmes; support for knowledge infrastructures (e.g. patent data bases)
Guidance of the search Science and technology foresight exercises; communication platforms/fora for industry, academia, societal organisations and public policy
Market formation Regulatory frameworks for technical standards and norms; ethical regulation; Intellectual property rights (IPR); et al.
Resources mobilization Thematic or sectoral profiling of public investment in science, R&D, and education
Creation of legitimacy/counteract resistance to change
S/T foresight exercises; communication platforms/fora; maintaining policy networks (e.g. multilevel cooperation across regions, nations and trans national levels); fostering institutional adaptation and change
Source: Hekkert et al. 2006
S. Kuhlmann, CGEE International Seminar, Rio de Janeiro 2007 page 10
1. ‘Innovation system’ concept; evolution of policy instruments and rationales
2. Detour: ‘One size does not fit all’
S. Kuhlmann, CGEE International Seminar, Rio de Janeiro 2007 page 11
Detour (1): ‘One size does not fit all’The PRIME ERA Dynamics concept (work in progress)
NSI too static (building on specialisation of ‘national’ or ‘sectoral’ systems) – difficulty to cope with complex dynamics in knowledge prduction and global socio-economic change
Core hypothesis: different ‘search regimes’ in knowledge production correspond to different institutional settings and policies = evolving ‘configurations’.
Implications for policy design: different knowledge dynamics appearing in different ‘configurations’ will evolve with different policy mixes.
Aim of project: Identification of limited set of ‘ideal-type’ knowledge configurations and characteristics of related institutions and policy-mixes – helping to design policy concepts in a prospective manner (foresight)
S. Kuhlmann, CGEE International Seminar, Rio de Janeiro 2007 page 12
Knowledge configurations correspond to
Specific characteristics of different knowledge dynamics
Techno-industrial dynamics market characteristics,
sectors
user behaviour and expectations
Institutional landscapes Regulation
Character of policy arenas and agency
Public policy initiatives and traditions Historical path dependency
[Degree of ‘Europeanisation’]
The involved actors, their ambition, strategy and power.
Detour (2): Knowledge dynamics and institutional “configurations”
S. Kuhlmann, CGEE International Seminar, Rio de Janeiro 2007 page 13
Knowledge dynamics have three main ‘aspects’*:
Growth = capacity to survive and/or prosper within the same institutional and organisational setting. Indicators: publications, patents, exports
Convergence = modalities of knowledge flows, and in particular opposing ‘individual’ vs. ‘distributed knowledge’ and the collaboration patterns
Complementarities = Technical complementarities = role of large shared infrastructures or equipment (critical
infrastructures)
Cognitive complementarities = collaboration patterns (bilateral vs. multilateral e.g. networks and clusters); critical mass, competences to be assembled to develop a relevant ‘research production unit’
Institutional complementarities = heterogeneous collaboration for efficient productive settings (e.g. strong relationship between clinicians and biologists in biotechnology); frequency of industry-university collaborations
(* Building on Bonaccorsi 2005ff; see also the rich body of literature in innovation economics, sociology of science and in science and technology studies (STS).)
Detour (3): Knowledge dynamics and institutional “configurations”
S. Kuhlmann, CGEE International Seminar, Rio de Janeiro 2007 page 15
Detour (4): ‘Anecdotal’ examples of knowledge dynamics
Source: Laredo, 2006
Dominant science-led knowledge field
Physics Computer science / TI Molecular biology
Dynamics / crystallisation (Cognitive complementarities)
Large objects or technical systems
Distributed PI (patent pools…)
Science bases / individual PI & transfer/licences
Trajectory (degree of convergence)
Early selection of a dominant design / cumulative improvements
Adoption of standards and design
Competition between paradigms
Critical infrastructures (technical complementarities)
Specific very large equipments
Generic infrastructures (broadband networks…)
No entry barrier
Coordination mode (driving institutional complementarities)
National large programmes (product oriented)
Technological programmes Strong industry-university relations
Networks & clusters (bottom-up)
Main industrial actors National champions (specialising in “public” infrastructures/ services)
MNF (oriented toward mass markets) / specialised NTBF (B to B)
Start-up & venture capital (in early phases) (concentration around large established firms in wider diffusion)
‘representative’ industries Nuclear energy, space, civil aeronautics, fixed numerical telecoms
Information technology, mobile communications
biotechnology
S. Kuhlmann, CGEE International Seminar, Rio de Janeiro 2007 page 16
1. ‘Innovation system’ concept; evolution of policy instruments and rationales
2. Detour: ‘One size does not fit all’
3. Weaknesses of present research and innovation policy
S. Kuhlmann, CGEE International Seminar, Rio de Janeiro 2007 page 17
Weaknesses of research and innovation policy(in EU and else) Lack of Governance mechanisms allowing for more ‘systemic’ orchestration
between diverse knowledge and innovation related policy domains (see LEG 2007)
High degree of departmentalisation, sectoralisation of the political administration, and low inter-departmental exchange and co-operation
Heterogeneous, un-linked arenas: often ‘corporatist negotiation deadlocks’
Failing attempts at restructuring responsibilities in government because of institutional inertia
Dominance of ‘linear model’ of innovation in policy approaches
‘Innovation policy’ run in a very specific, narrow field focusing on introduction of new technologies in SMEs, IPR or VC issues etc.
S. Kuhlmann, CGEE International Seminar, Rio de Janeiro 2007 page 18
Need for ‘systemic’ policy instruments
Re-shaping of innovation and research systems
facilitating the construction (Neue Kombinationen) and deconstruction of subsystems, preventing of lock-in
supporting prime movers
ensuring that all relevant actors are involved
Stimulating demand articulation, strategy and vision development
Building cross-linking platforms and "new spaces" for learning and experimenting
Providing and exploiting an infrastructure for distributed "strategic intelligence" (building on technology assessment, foresight, evaluation, benchmarking etc.)
S. Kuhlmann, CGEE International Seminar, Rio de Janeiro 2007 page 19
1. ‘Innovation system’ concept; evolution of policy instruments and rationales
2. Detour: ‘One size does not fit all’
3. Weaknesses of present research and innovation policy
4. Stakeholders‘ diverging interests, policy arena, governance
S. Kuhlmann, CGEE International Seminar, Rio de Janeiro 2007 page 20
National
research
ministry
National
research
ministry
Othernationalministrie
s
Othernationalministrie
s
Regional
govern-ments
Regional
govern-ments
National
parlia-ment
National
parlia-ment
EUCom-
mission
EUCom-
mission
Multi-nationalcompani
es
Multi-nationalcompani
es
SMEasso-
ciations
SMEasso-
ciations
Industrial
asso-ciations
Industrial
asso-ciations
Uni-versities
Uni-versities
Nationalresearchcenters
Nationalresearchcenters
Research
councils
Research
councils
Contractresearchinstitute
s
Contractresearchinstitute
s
Consumer
groups
Consumer
groups
Environ-ment
groups
Environ-ment
groups
Public research and innovation policy actors’ arena – a heuristic
StrategicIntelligence
Organised actors: differing interests, values, and power; bounded rationality
Competition for impact and resources
No dominant player?
Contested policies
Search for (some) alignment and policy learning - otherwise ‘exit’
‘Enlightenment’ through ‘Strategic Intelligence’
S. Kuhlmann, CGEE International Seminar, Rio de Janeiro 2007 page 22
1. ‘Innovation system’ concept; evolution of policy instruments and rationales
2. Detour: ‘One size does not fit all’
3. Weaknesses of present research and innovation policy
4. Stakeholders‘ diverging interests, policy arena, governance
5. Role of ‘Strategic Intelligence’
S. Kuhlmann, CGEE International Seminar, Rio de Janeiro 2007 page 23
Policy Learning
Concept of bounded rationality in policy learning processes
Providing insights in the limited but relevant impact on the quality of decision-making for “rationalisation”-based policies (see Braun/Benninghoff referring to Heclo, Hall, Olsen/Peters)
Concept of single and double loop learning (Argyris/Schön 1978)
first-order (single loop) learning helps “to keep organizational performance within the range set by organizational norms. The norms themselves […] remain unchanged”;
Second-order (double loop) learning concerns “incompatible organizational norms by setting new priorities and weightings of norms, or by restructuring the norms themselves together with associated strategies and assumptions” .
S. Kuhlmann, CGEE International Seminar, Rio de Janeiro 2007 page 24
Strategic intelligence and policy learning
Strategic intelligence is ...
... a set of sources of information - often distributed and heterogeneous
explorative/empirical as well as analytical (theoretical, heuristic, methodological) tools
well known strategic intelligence tools are evaluation studies, performance measurement, benchmarking initiatives, foresight exercises, or technology assessment (TA)
employed to produce “multi-perspective” insight in the actual or potential costs and effects of public or private policy and management, to be 'injected' and 'digested' in political arenas
facilitating policy learning
S. Kuhlmann, CGEE International Seminar, Rio de Janeiro 2007 page 25
Analysing the dynamics of research and innovation systems
Analyses & methods:Evaluation (ex post, monitoring, ex ante)Delphi, scenarios, TAPolicy-analysisInstitutional analysisStatist.-econometrical analysesNetwork analysis
Indicators:corporate dataSectorial techno-eco- -nomic performanceBibliometricsRegulatory data (e.g. norms, standards)
Sectors,technologies:
retrospectively, prospectively
Innovationprocesses:
micro, meso,macro
Actors:
Companies,Science,
Policymakers
S. Kuhlmann, CGEE International Seminar, Rio de Janeiro 2007 page 26
‘Forum’ and ‘Strategic Intelligence’
S. Kuhlmann, CGEE International Seminar, Rio de Janeiro 2007 page 27
1. ‘Innovation system’ concept; evolution of policy instruments and rationales
2. Detour: ‘One size does not fit all’
3. Weaknesses of present research and innovation policy
4. Stakeholders‘ diverging interests, policy arena, governance
5. Role of ‘Strategic Intelligence’
6. Options and limitations of ‘impact’ evaluation
S. Kuhlmann, CGEE International Seminar, Rio de Janeiro 2007 page 28
R&D policy ‘evaluation’ may be defined as ...
“... methodology-based analysis and assessment of the appropriateness of S/T policy assumptions and targets, of the related measures and their impacts, and of the goal attainment.”
(cf. Kuhlmann/Holland 1995a, 199; Kuhlmann/Meyer-Krahmer 1995, 3pp)
S. Kuhlmann, CGEE International Seminar, Rio de Janeiro 2007 page 29
Typical R&D evaluation questions (Arnold/Guy 1997, 72)
Appropriateness: Was it the right thing to do?
Economy: Has it worked out cheaper than we expected?
Effectiveness: Has it lived up to the expectations?
Efficiency: What’s the return on investment (ROI)?
Efficacy: How does the ROI compare with expectations?
Process efficiency: Is it working well?
Quality: How good are the outputs?
Impact: What has happened as a result of it?
Additionality: What has happened over and above what would have happened anyway?
Displacement: What hasn’t happened which would have happened in its absence?
Process Improvement: How can we do it better?
Strategy: What should we do next?
S. Kuhlmann, CGEE International Seminar, Rio de Janeiro 2007 page 30
Impact dimensions of public R&D spending
Direct impacts Indirect impacts Main domains ofimpact of public
spending Short-term Long-term Short-term Long-term
Science(“Wissenschaft”) Typical impacts
scientificfindings
knowledge improved teaching
industrial spill-overs
Economy andsociety
Typical impacts
improvedtechnology
improvedtechnicalknow-how
increasedproductivity
improvedcompetitive-
ness
Policy
Typical impacts
improvedunder-
standing
problem-solving
increasedproblem awa-
reness
increasedgeneral satis-
faction
S. Kuhlmann, CGEE International Seminar, Rio de Janeiro 2007 page 31
Funded project
Other policies
Programme orinstitution
Scope and limitations of impact measurement of public R&D Market dynamics
Change ofbehaviour
Improvedknow-howbase
New product orprocedure
Company or institute
Sector orregion
Nationaleconomy
S. Kuhlmann, CGEE International Seminar, Rio de Janeiro 2007 page 32
Searching for impacts of policy ...
Evaluation identification of impact of (public) action
scientific, technological, economic, societal, political, ...
past/future, direct/indirect, intended/non-intended, ...
Condition: Model of input/output relation, of cause/effect, of actors and beneficiaries ...
“Impact” a rational construction of more or less complexity
S. Kuhlmann, CGEE International Seminar, Rio de Janeiro 2007 page 33
Summative and formative evaluation Summative Evaluation
systematic, indicator based
mainly ex post - or interim - measurement and assessment of the performance of programmes (including projects)
to assess the programme design, implementation management and the leverage of funding and to learn for future approaches
Formative Evaluation systematic consulting, moderating, assessing activities
seeking to assist policy makers, programme managers and programme participants
throughout the whole life cycle of funding programmes
to make all actors involved learn and (re-)adjust
and thus contribute to the overall success (and/or improvement and/or termination) of programmes and funded structures and to learn for future approaches.
S. Kuhlmann, CGEE International Seminar, Rio de Janeiro 2007 page 34
1. ‘Innovation system’ concept; evolution of policy instruments and rationales
2. Detour: ‘One size does not fit all’
3. Weaknesses of present research and innovation policy
4. Stakeholders‘ diverging interests, policy arena, governance
5. Role of ‘Strategic Intelligence’
6. Options and limitations of “impact” evaluation
7. Evaluation methodologies
S. Kuhlmann, CGEE International Seminar, Rio de Janeiro 2007 page 35
Evolution of research systems and evaluation practices
Example:German research system
S. Kuhlmann, CGEE International Seminar, Rio de Janeiro 2007 page 36
Grants for open-ended research
Contracts and mission-oriented R&D programs
Ex-ante assessment (appraisal)
Peer review of proposals
Is prerogative of customer or sponsor, often ad-hoc and in-house appraisal
Ex-post assessment (evaluation)
Only through track record as criterion in appraisal of later proposals
Expert panels (with support) evaluate output, goal attainment
R&D world until the 1970s
S. Kuhlmann, CGEE International Seminar, Rio de Janeiro 2007 page 37
Grants for open-ended research
Grants and programs for strategic research
Contracts, mission-oriented R&D programs
Ex-ante assessment: appraisal
Peer review of proposals
- Peer and user review of scientific and societal quality
- Relevance for priority themes (cf. foresight)
- New actors (e.g. patient associations)
Remains prerogative of customer or sponsor, often ad-hoc and in-house appraisal
Ex-post assessment: evaluation
Track-record
(returns as focus on excellence)
- Expert panels estimate strategic value
- Methods to trace uptake, impact
Expert panels evaluate output, goal attainment
R&D world from the 1980s
onward
S. Kuhlmann, CGEE International Seminar, Rio de Janeiro 2007 page 38
Variety of R&D evaluation methods: a “metrix”?
Evaluation mission ex ante (based on
foresight?): strategic options?
monitoring, real time: management, fine tuning
ex post: learning, legitimisation
summative
formative
Evaluation data official statistics (R&D, patents)
bibliometrics
questionnaire-based surveys
interviews
case studies
Evaluative methodology peer review, peer panels
input/output; cost/benefit; before/after (descriptive statistics; econometrics)
comparison groups
benchmarking
network analysis
foresight
technology assessment
Specify mixdepending onpolicy issue -- no general
metrix!
See also "Evaluation Toolbox": http://epub.jrc.es/docs/EUR-20382-EN.pdf
S. Kuhlmann, CGEE International Seminar, Rio de Janeiro 2007 page 39
Evaluation methods, quantitative and qualitative Quantitative: Statistical data analysis
Innovation Surveys: basic data describe the innovation process, using descriptive statistics
Benchmarking: comparisons based on a relevant set of indicators across entities
Quantitative: Modelling methodologies Macroeconomic modelling and simulation: broader socioeconomic impact of policy
interventions
Microeconometric modelling: effects of policy intervention at the level of individuals or firms
Productivity analysis: impact of R&D on productivity growth at different levels data aggregation
Comparison group approach: effect on participants using statistical sophisticated techniques
Qualitative and semi-quantitative methodologies Interviews and case studies: direct observation of naturally occurring events to
investigate behaviours in their indigenous social setting
Cost-benefit analysis: economic efficiency by appraising economic and social effects
Expert panels/peer review: scientific output relying on the perception of peer scientists
Network analysis: structure of cooperation relationships and consequences for individuals and their social connections into networks
Foresight/ technology assessment: identification of potential mismatches in the strategic efficiency of projects and programmes
Source: Fahrenkrog et al., RTD Evaluation Toolbox, http://epub.jrc.es/evaluationtoolbox/start.swf
S. Kuhlmann, CGEE International Seminar, Rio de Janeiro 2007 page 40
A professional evaluator should:
Focus on decision context
Understand the stakes
Use relevant methods (and use them well)
Understand the nature of the object (here: R&D and its institutions)
Understand the evolving context (here: knowledge, research & innovation system)
S. Kuhlmann, CGEE International Seminar, Rio de Janeiro 2007 page 41
Specifics of R&D evaluation (1)
R&D is open-ended (success can be re-defined after the fact)
Connection between R&D and effects is non-linear and indirect
Even in application-oriented research it may take 10 years or more for impact to be realized (and attribution then becomes tenuous)
But evaluation is required earlier …
S. Kuhlmann, CGEE International Seminar, Rio de Janeiro 2007 page 42
Specifics of R&D evaluation (2)
R&D functions in a larger whole, thus its productivity and effects depend on what happens there
no simple RoI (return on investment) approaches
Knowledge, Research and Innovation System (KRIS)
links with internationally defined scientific fields and domains of application
S. Kuhlmann, CGEE International Seminar, Rio de Janeiro 2007 page 43
Specifics of R&D evaluation (3)
Content & domain are important
distantiated measures (e.g. number of publications, patents) do not capture enough
judgment of domain experts is necessary (but has limitations as well, cf. peer review)
balance the two – responsibility of the evaluator!
S. Kuhlmann, CGEE International Seminar, Rio de Janeiro 2007 page 44
There is not one evaluative ‘truth’:
Actual R&D policies relate to variety (and competition) of targets, envisaged effects and underlying rationales and assumptions variety of promotion instruments, overlapping, competing ...
Involved actors (policy, industry, science) pursue heterogeneous, partly conflicting interests, assumptions, expectations success criteria differ
Ever more R&D policy interventions aim at multiple purposes and heterogeneous actors (e.g. the set of “socio-economic” targets and clients) increased complexity and interbreeding of input-output-outcome relationships
Specifics of R&D evaluation (4)
S. Kuhlmann, CGEE International Seminar, Rio de Janeiro 2007 page 45
Specifics of R&D evaluation (5)
Standard (quantitative) analyses often not applicable
limited number of cases (sometimes N=1, and evaluator has to mobilize experience/insight in similar cases)
skewed distributions (of productivity, of impact, of uptake in innovation) – so one cannot use sampling
S. Kuhlmann, CGEE International Seminar, Rio de Janeiro 2007 page 46
U r s a c h e u n d W i r k u n g. - Vor der Wirkung glaubt man an andere Ursachen als nach der Wirkung.Friedrich Nietzsche, Die fröhliche Wissenschaft, Aphorismus 217, 1882
Cause and impact. - Before any impact you believe in other causes than after the impact.
S. Kuhlmann, CGEE International Seminar, Rio de Janeiro 2007 page 47
Prof. Dr. Stefan KuhlmannUniversity of Twente Chair Foundations of Science, Technology and Society School of Management and Governance Institute of Governance Studies (IGS)Enschede, The Netherlandse-mail: s.kuhlmann@utwente.nl
Thanks for your attention !!
Further info and contactSmits, R. / Kuhlmann, S. (2004): The rise of systemic instruments in innovation policy. In: Int. J. Foresight and Innovation Policy (IJFIP), Vol. 1, Nos. 1/2, 2004, 4-32
Shapira, Ph., Kuhlmann, S. (eds.) (2003): Learning from Science and Technology Policy Evaluation: Experiences from the United States and Europe, Cheltenham (E. Elgar)
IPTS (ed.) (2002): RTD Evaluation Tool Box. - Assessing the Socio-Economic Impact of RTD-Policies Brussels/Luxembourg (EUROPEAN COMMISSION, IPTS Technical Report Series, EUR 20382 EN)
International R&D Evaluation Course (4 days) at University of Twente, see: http://www.mb.utwente.nl/stehps/news/rd_2007.doc/