TCA VENet | project evaluation

Gerd Beidernikl

Center for Education and Economy, Research & Consulting

Graz, Austria

1st project meeting

11. September 2005, Cyprus

2

the goals of this session

To introduce myself and the ZBW

To develop a common understanding WHY we are evaluating our work and HOW we are going to do it

To present the (preliminary) evaluation concept and to get feedback on it

To make sure the words and concepts used are familiar to everybody

To develop a common understanding of how to develop criteria that could be used as criteria for success

Overview on the upcoming steps

3

Introduction

4

my background

Profession: Sociology, further training in the field of evaluation and quality management

Researcher in the „Center for Education and Economy“, Graz, Austria

5 full time employees

Public and private clients

My main working areas: labour market research, research on vocational education and training, evaluation, statistics and surveys

Ca. 20 evaluation projects in the last 4 years

5

project references on EU level

Evaluation of the ongoing Leonardo project „inclusive training for disability care workers“

Self-evaluation of the Equal project „girls crack IT“

Evaluation of the Styrian Territorial Employment Pacts

Evaluation of the article VI project „GO BEST“

Project management of an ongoing Leonardo project „online consulting tool for IT professions“

Member of a proposal development group in the 6th framework programme, priority 7

Reasearch activities for the CEDEFOP: reports on the Austrian training system, comperative country studies, administrating the ERO database (European Research Overview)

6

workflow of this presentation

Evaluation

What does it mean?

What can it do for us?

What do we need?

Concept

Log FrameElements of tender

Gender Mainstreaming

Defining the Log FrameIndicators

Terms of Reference

Next steps

Timeline

At this meeting

After the meeting

Feedback

Feedback

7

Evaluation

>>There are as many different definitions of „evaluation“ as there are evaluators.<<

8

what is “evaluation”

>>Evaluation is the systematic investigation of the merit or worth of an object or process for the purpose of reducing uncertainty in decision making.<< (Mertens, 1998)

3 types of evaluation – regarding the point in time

Ex-ante or predictive evaluation concept

Acompanying or formative evaluation processes

Ex-post or summary evaluation results

Different context, different focus

Generating and securing knowledge

Controlling

Legitimizing actions

Dialogue with stakeholders, etc…..

9

why evaluation? what’s the benefit?

An evaluation offers a systematic way of approaching a topic benefits for project planning

Gives neutral feedback for all project partners

Informs about performance, efficiancy, effectiveness and quality of the project

Makes the project results visible marketing, proof

Possibility to learn in the runtime of a project

Possibility to counteract in case of anticipated problems

Secure Knowledge

Identify best practice and transferable models

10

what this evaluation should be

Support for the transnational cooperation

Answering the question: Do we deliver what we are supposed to deliver?

Neutral reflexion of what is going on

Suggestions for changes and improvement

Ensuring positive project development by giving periodic input

Fostering the coperation and communication

11

what an / this evaluation should not be

A judgement of good and bad, of right or wrong

Evaluators are no project outsiders, they are project partners

Evaluators are no pitbull terriers, lurking for any mistake

Evaluation ≠ project controlling

No analysis of cost efficiancy

No evaluation of the benefits for the target groups

12

What are your expectations?

13

Evaluation concept

14

logical framework

One commonly used apporach: logical framework = LogFrame

The LogFrame is the starting point of the evaluation

Displays the main logic of the evaluation subject

A skeleton of the project that has to be evaluated

Represents the causal relations between different levels of objectives, actions and results

15

LogFrame

GoalsNeeds Ressources / Input

Processes / Mechanisms

Effects / results

Overall objectives

Indikators

Specific objectives

Expected Products

Impact

Outcome

Output

Performance

Relevance

Effectiveness

Efficiency

Secondary effects - unexpected

sustainability

Context effects

Structure

Concept

16

preconditions

There are commonly shared, defined and agreed goals

Goals have to be measureable

There are defined ressources

There are defined processes

The skeleton of the log frame has to be filled based on information in the kick off stage of the project.

Should be kept in mind in every working session of this meeting.

Will be worked out by the evaluation team after the meeting in cooperation with the project management

17

call for tenders

The call for tender asked for

Evaluation of the agreed procedures, tasks and responsibilities formal evaluation

Evaluation of the results and the processes to reach these results content evaluation

Participation at every meeting

Monitoring of the Gender Mainstreaming Process

Periodically reporting

18

main elements of the tender

Evaluation TCA VENet

Formal Evaluation Content Evaluation

Concept Evaluation

Structural Evaluation

Gender Mainstreaming

Process Evaluation

Result Evaluation

VENet ModelEuropean Mainsteaming

Spport SystemInternet Platform

Final Activity

Gender Mainstreaming – cross-sectional topic

19

formal evaluation - concept

Concept analysis

Assessment of the codified project structure and plan

Assessment of management tools

Assessment of communication tools

Data

project documents (e.g. proposal)

meetings

protocols (e.g. steering group)

interviews

20

formal evaluation - structure

Structural analysisFocus: transnational cooperation als system of interacting

organizations resp. individuals

Assessment of flow of communication

Assessment of the cooperation patterns

DataHard facts: contribution on formal level (reports, website,..)

Soft facts:Social network survey (x2, email)

Satisfaction survey (x2, email)

Event questionnaire (x5, at every project meeting)

Interviews

Documents, Internet Platform (?),….

21

Social network analysis (SNA)

SNA – research method that focuses on the relation (communication, cooperation, exchange of information,…) between any given set of actors.

A SNA survey asks the partners to rate their own network behaviour according to different criterias (frequeny of interaction, channels of interaction, topics of interaction,…).

The answers on these questions are producing a picture of the overall network. All individual answers are displayed on the network level.

Benefit

You get SNA plots of the network structure

Key fiugures on well and not well established elements

22

network plots

23

Gender Mainstreaming

GM are the efforts we make to ensure that gender aspects and gender equality is considered in all parts and stages of the project (working plan, objectives, actions,…).

A cross sectional topic

GM aspects are considered in every part of the evaluation

and the evaluation itself

3 main GM approaches

Equal opportunities approach neutral treatment

Diversity approach awareness of the different needs

Affirmative action approach focus on women

24

Gender Mainstreaming

Formal Evaluation Content Evaluation- Gender sensitive language- Roles and functions- GM as a topic at the meetings- GM as cross sectional topic in the project- Gender aspects in documents- Treatment of GM in general- GM experience of the partners-…

- Is GM considered by the partners in

- the analysis of underling

problems

- the definition of goals and

target groups

- the design of the processes

- the implementation

- Do the products reflect GM principles?

- Design / content

- Dissemination

- Benefit

Data: protocols, interviews, survey questions, products,…

25

Content evaluation

Two levels

Processes

Results

Main questions

Have the intended results been reached (quantity and quality)?

Have the processes to reach these results been carried out as intended?

Did the partners contribute as scheduled?

Are there any positive or negative deviations? If so, why?

Which deductions for the following project work can be made?

26

Content evaluation

For each of the five main products

VENet model, internet platform,…..

Data

Documents

Interviews

The products themselves

Check lists resp. progress reports for the project (in cooperation with the project management)

Questions in the email survey

27

deliverables of the evaluation

Short reports before each meeting

Distributed electronicly (internet platform)

Participation at every transnational meeting (1 w.d.)

Presentation of evaluation results and workshop on the findings (1-2,5h)

1 mid-term report

1 final report

28

Feedback on the evaluation model What is your impression?

Does the evaluation provide the information we need?GM in our project?

How do we take the evaluation results up?

29

Defining the LogFrame

30

Defintion

By now the LogFrame is empty

Will be based on the transnational project plan with defined goals, milestones and responsibilities

Next Step: Filling the LogFrame with information

Results from this meeting (e.g. working plan,…)

Documents

In cooperation with transnational coordination

The rough LogFrame will increase in its density as the project goes on

31

Definition

The defined aims of the project are crucial for the evaluation!

Important: If you set up any aims, goals and objectives:

be SMART

S…specific: general aims must be broken down

M…measurable: no untouchable aims

A…acceptance: agreement among all partners

R…realistic: no trivial aims, no unrealistic visions

T…timely fixed and with fixed responsibilities

32

Definition

The LogFrame is the catalogue of the evaluation.

Beneath this we will set up the so called Terms of Reference

Kind of evaluation handbook, the practice version of the tender

Puts down the procedures of the evaluation

Commonly shared evaluation principles

Will beset up by the ZBW

33

Next Steps

34

At this meeting

Consider the LogFrame allready when working out objectives (SMART), products and processes

Evaluation grows as the project grows.

Some short interviews on the 2nd resp. 3rd day of the meeting – transnational coordinators

A concluding event questionaire in the last session of the meeting

Define one main contact person regarding the evaluation in each partner organization

35

After the meeting

ZBW will work out the LogFrame in cooperation with the transnational coordination

Set up the Terms of Reference

Set up dates for the next evaluation steps

Inform the partners about the evaluation

36

Timeline

37

Your responsibilities!

To commit and take active part in the evaluation process

To fill in questionnaires carefully

To submit documents and information when asked for

To return you contribution to the evaluation in time – respect deadlines

To evaluate the evaluation! Feedback is welcome!!

38

Contact information

Thank you!Let‘s get it on!

Mag. Gerd BeiderniklZentrum für Bildung und Wirtschaft

Joanneumring 5/4, 8010 Graz, Austria (valid 1st October 2005)

Mail: gerd.beidernikl@zbw.atPhone: +43 / 316 / 72 17 44 -13

Fax: +43 / 316 / 72 17 44 -21Mobile: +32 / 650 / 46 21 061