Presentation

Version 9.2.2003

VTT TECHNOLOGY STUDIES

Evaluating the societal impacts ofPublic research organisations:

A (belated) paradigm shift in the making

Kaisa Lähteenmäki-Smith and Kirsi HyytinenVTT Technology Studies

Version 9.2.2003


Presentation

• The project on societal impact evaluation: research questions, methods, participating organisations and their differentiated working environments, steering processes and R&D ambitions

• Evaluation model for societal impacts of publicly funded R&D organisations

• Some key findings and conclusions and their implications for the evaluation model

Version 9.2.2003


Description of the project: questions

• How are publicly funded R&D organisations adjusting their evaluation practices to the emerging ‘management by results’-based accountability demands set by the government?

• How can different evaluation practices and methods be used for this purpose in assessing the societal impacts of R&D activities?

• What are the similarities and differences, as well as lessons to be drawn between policy sectors (best practices, challenges, bottlenecks…)?

• What are the implications of impact assessment on administrative practices, including steering processes, in respect of governmental actors and public research organisations?

Version 9.2.2003


Description of the project: organisations

• MULTI-SECTOR TECHNOLOGY R&D INSTITUTE (VTT) is national research institute in technology working in close interface with both the business sector and policy-making.

• DEFENCE SECTOR R&D INSTITUTE: The Finnish Defence Forces Technical Research Centre

• RESEARCH INSTITUTE IN THE AGRICULTURE AND FOOD SECTOR, MTT Agrifood Research Finland

• TWO POLYTECHNICS – one WITH ITS SUBSTANTIVE FOCUS IN THE AREA OF BUSINESS MANAGEMENT, ANOTHER MULTIDISCIPLINARY POLYTECHNIC

Version 9.2.2003


Motivations of impact assessment

Internal and external motives :

Change in management and administrative cultureDemand for the accountability and result-driven planningSpread of a service-oriented culture – customer orientationDemand for transparency and accountability

Compatibility and integration of policy objectivesSelection of sectoral objectives and consolidation of sector objectives with inter-connected sectors Selection and compatibility of horizontal goals

Demand for organisational developmentIdentification of demands and multiple development challengesSupporting strategic planning – contributes to steering and priority-setting, allows for operational freedom in developing core competences and activities Prioritizing the activities

Version 9.2.2003


National approach to accountability

• National and international dimensions of effectiveness• Customer-orientation: are the needs of the clients met?• Future orientation: are the visions and strategic targets set

meeting the needs of the changing nature of society and of the policy environment?

• Cost effectiveness• Socio-economic and societal dimension: Have the main

external costs and effects been taken into consideration?

Version 9.2.2003


THE EMERGENT FRAMEWORK OF

IMPACT EVALUATION

Version 9.2.2003


Key concepts 1/3- output, outcome, impact,

effectiveness

The preservation of healthy oat grain compounds in the baking process

The healthy food product

Lower cholesterol levels

Improved national health longer life-expectancy economic effects

Output

Outcome

Impact

Effectiveness

Version 9.2.2003


Kohdealueenulkopuolella

T&k-vaikutukset

Key concepts 2/3- impact typology

IMPACTS

ANTICIPATED

In target area In target area

Useful

Detrimental

Useful

Detrimental

UNANTICIPATED

Outside target area Outside target area

Other impacts

R&Dimpacts

Otherimpacts

R&Dimpacts

( Mickwitz 2004)

Version 9.2.2003


Key concepts 3/3-impacts through time

IMMEDIATE•networking• improved R&D

efficiency• patent applications• publications• prototypes

INTERMEDIATE•partnership-based cooperation• new/ improved products, services and processes•company growth•improved company competitiveness•cost savings•higher employment•strengthened expertise•technology transfer•standards, norms•support for decision making•public discourse

ULTIMATE• improved industry

competitiveness• higher investment• better safety• improved quality of life• promotion of regional

development• improved awareness

-1 0 1 2 4 6 8 10 15 YEARS, OR MORE

IMPACTS

Benefits to economy and

society

Benefits to project participants

(Tassey 2003)

Version 9.2.2003


Dimensions of impacts and final beneficiaries

1. Economy, technology and commercialisation

2. Knowledge, expertise and human capital

3. Networking and social capital

4. Decision making and public discourse

5. Social and physical environment

Own organisation

Customer

Society

Public R&D-organisation

Version 9.2.2003


Evaluation model: some criteria

• Relevance• Quantifiable nature• Reliability• Easy availability• Dynamics of change and trends• Strategic feedback loop

Version 9.2.2003


How to get information about impacts and effectiveness

through evaluation

Ex ante evaluationPreparation phase

Execution phase

Concluding phase

Ex post evaluation

Stakeholder and client ’inputs’ into evaluation cycle•Follow-up studies

•Client satisfaction surveys

•Interactive discussion fora

Mid-term evaluation

Version 9.2.2003


Impact assessment as a learning and steering tool

Rationales for public support

MOTIVESMarket failure

Transaction costsRisks and uncertainty

Policy statements

OBJECTIVESPublic safety

New enterprises and workplaces

Welfare services

Policy implementation

ORGANIZATIONAL CAPACITY

as the means(for doing

’the rights things’)

Transparency, permanence, flexibility, predictability, legitimacy, justness

Using the knowledge about changes amongst

the stakeholders

ADDED VALUE OF R&D

To the organisation To the customer

For society

Definition of the new rationales

Allocation of the resources

Prioritizing

Definition on the strategy

Feedback

Presentation

Documents

Transcript of Presentation