NZSSES/ICSER Conference Defining and Evaluating ‘Science for Sustainability’ AIRP-SD Project...

NZSSES/ICSER Conference

Defining and Evaluating Defining and Evaluating ‘Science for Sustainability’‘Science for Sustainability’

AIRP-SD Project

Paul Weaver & Leo [email protected]

[email protected]


AIRP-SD GoalsAIRP-SD Goals

Develop an approach for ex ante, concomitant and ex post evaluation of sustainability-oriented RTD programmes

Describe innovative RTD approaches Identify good and bad practices Assess transferability of practice Establish programme design guidelines


Toward an evaluation architectureToward an evaluation architecture

Lack of theory and practice suggests need to develop both through an iterative, heuristic procedure – begin with tentative hypotheses about the nature

of the research processes that might support SD; test/refine these through the evaluation process itself

Need for externally-specified evaluation criteria Depends on establishing a conceptual model of

sustainability-oriented RTD


AIRP-SD Procedure

Establish conceptual model of sustainability research

Select case study research programmes/projects

Describe research outcomes, designs, contexts

Evaluate outcomes, design, context separately

Relate research outcomes to design

Relate design to context

Relate outcomes to context Revise conceptual m

odel

Recommendations for research management and design

Identify good practice


Derive conceptual model from:Derive conceptual model from:

the nature and concept of sustainable development

the nature of the socio-ecological systems of interest

the nature of SD challenges to science and society


SD challengesSD challenges

Orientation and ambition - paradigm change - transition - prescriptive - normative - vision-led

Complexity of socio-ecological systems of interest, indeterminacy, irreversibility, risk, precaution

Multiple stakeholders, high stakes, uncertainty, non-market values

Need to communicate with non-scientists


Evaluation objects (outcomes)Evaluation objects (outcomes)

Outcomes

Contribution to strategic orientation

Contribution to solution set (quantity/quality)

Contribution to strategic

management

Contribution to social capital

Contribution to scientific capital and capacities

Contribution to influencing society


Evaluation objects (D&P)Evaluation objects (D&P)

Design and Process

Management and finance model

Procedure for initiating/defining programme

Research model (methods, tools,

procedures)

Procedure for handling risk and uncertainty

Procedure and basis for decision making

Procedure for communication & dissemination


Evaluation objects (context)Evaluation objects (context)

Context

Established social capital

Established RTD management capital

Established scientific/research

capital

Extent & urgency of unsustainability or vulnerability

Understanding & awareness of (un) sustainability across society


Sustainability principlesSustainability principles

Global and systemic approaches Knowledge sharing and mutual learning Broad participation Environmental heritage Precaution / Action to prevent risk Resilience


Sustainability principles (cont.)Sustainability principles (cont.)

Transparency and justification of decision making

Appropriate scale-issue relation Inter-generational equity Intra-generational equity


AIRP-SD working hypothesesAIRP-SD working hypotheses

Hypothesis I

A wide set of generic research outcomes can be identified and evaluated on a set of externally specified sustainability goals and criteria

Hypothesis II

Research designs and processes that conform strongly with the principles of sustainable development are likely to contribute to strong research outcomes

Hypothesis III

Contexts where sustainability problems or awareness is high are likely to facilitate SD research. Strong scientific/social capital in relation to SD challenges will facilitate SD research and strong outcomes


Hypothesis IVHypothesis IV

CONTEXT

Research design and process

Research outcomes

Research context, design/process and outcomes are connected in a loop


Nine case studiesNine case studies

ALR ATSD CUTS ECO-CYCLE HABIFORUM MAHRE SEP STD ZERI


Fact finding versus evaluationFact finding versus evaluation

Separate tasks Data collection – factual information – What?

How? Who? Evaluation – analytical, judgmental – Whether?

How well? Evaluative questions shape the mindset and

background for fact-finding


Methodological issuesMethodological issues

Restricted time-frame for analysis; some outcomes yet to emerge: interim outcomes, process-orientation

Ambiguities; e.g., financing model: detailed specification and differentiation

Subjectivity: consistent definitions, inter-subjectivity and cross-checking

Depth vs breadth: encourage evaluation to enable validation in due course


Multi-level approach (outcomes)Multi-level approach (outcomes)

N ove ltyG en era lisab ility

A ffo rd ab ilitye tc .

N ew p rod u c tion sys tem sN ew su ffic ien cy so lu tion s

e tc .

R an g e o f so lu tion s

C om p atib ilityIn teg ra tion

B a lan ce , e tc .

G lob a l issu esE coeffic ien cyE q u ity, e tc .

Q u a lity o f so lu tion s

C om p atib ilityD irec tion a lly re liab le

A ccep tedA d ap tab le , e tc

F u tu re vis ion sM etap h ors o f life q u a lity

In d ica to rs

S tra teg ic reo rien ta tionan d p a th fin d in g

S ys tem s-leve lP rocess o rien ted

C on tin g en c iesB arrie rs /b o tt len ecks

S cen ariosTech n o log y assessm en t

A c tion p lan s , e tc .

S tra teg ic m an ag em en t

LEVEL III

LEVEL II

LEVEL I

OUTCOM ES


Evaluation of each objectEvaluation of each object

Evaluate each outcome using SD goals Evaluate each D&P feature using SD principles Evaluate each context feature using SD

challenges Strength (strong, weak, neutral) Direction (positive, negative)


Evaluation of interrelationsEvaluation of interrelations

Evaluate influence of each D&P feature on outcomes

Evaluate influence of each D&P feature on each other (synergies)

Evaluate influence of context (context dependency, transferability) of each D&P feature

Evaluate any RTD induced change in context


Data analysisData analysis


Results: good practicesResults: good practices

Initiation– stakeholder inventory/analysis– broad participation– re-conceptualisation, scope/goals

Finance– co-funding arrangements– dedicated funds for re-conceptualisation,

stakeholder participation, communication– phased funding arrangements


Results: good practices (cont.)Results: good practices (cont.)

Management– internal learning (iterative, adaptive)– openness, transparency, accountability– continuous monitoring and quality assurance– re-defined set of success criteria

Research– vision-led, system-based, long-term oriented,

inter/trans-disciplinary, iterative– multi-functional approaches/tools


ConclusionsConclusions

Importance of context Importance of structure and process Importance of goal setting Importance of SD principles Synergy among design features - need for

thorough integration Feedback - practice makes perfect


Defining and Evaluating Defining and Evaluating ‘Science for Sustainability’‘Science for Sustainability’

AIRP-SD Project

Paul Weaver & Leo [email protected]

[email protected]

NZSSES/ICSER Conference Defining and Evaluating ‘Science for Sustainability’ AIRP-SD Project...

Documents

Transcript of NZSSES/ICSER Conference Defining and Evaluating ‘Science for Sustainability’ AIRP-SD Project...