Some words on evaluating (technological?) interventions

(EE of EEE?)

Henk SligteKohnstamm Institute for Educational Research

University of AmsterdamHenk.Sligte@UvA.NL

http://kohnstamminstituut.uva.nl/htm/english.htm

Evaluation

• Evidence based policy and practice: effect (impact) evaluation what works?

• The other side of the medal: explanatory evaluation: what works for whom in what circumstances and why? Finer granulanity

• Assignment of the Dutch Ministry of Education to develop a method to complement impact evaluation

• Based on policy interventions, can it work for (technological) innovations, for EEE?

Effect (Impact) evaluation

• Through (quasi)experiments demonstration of causal relation between intervention and found effects– Experiment: Controlled Randomized Trials– Quasi: natural experiment– Difference-in-difference: compare with business as

usual (the whole population minus the experimental)

Pretest-posttest modelCompare with a similar control or reference

groupsEffects can exclusively be ascribed to intervention

BUT:

• The rationale (the why) for these effects remains unknown: black box

• Explanatory evaluation focuses specifically on this and can further validate the results of an impact assessment (effect evaluation)

• Questions addressed:– How and why does an intervention work?– If there is no or a smaller effect occurs than

expected, why is that the case?– Which unwanted / unintended effects occur

(also unexpected positive)?

Intervention

• VM2– School drop-out in first year vocational

education enormous– Intervention: combine systems of preparatory

and normal vocational education– Effects: compare experimental group with

population as a whole on numbers of dropouts– Conclusion: less dropout… “in xx percentpoint

of the cases a significant lower chance of dropout”…

– How do find more differential effects and how to understand them better?

Explanatory evaluation (EE)

• Pawson, R. & Tilley, N. (1997). Realistic Evaluation. London: SAGE Publications

• What works for whom in what circumstances and why?

• Ex ante (before), ex durante (during), ex post (after)

• Central to answering this question is the reconstruction of the policy (or program) theory

• The policy theory is the sum of assumptions or hypotheses that explain how the policy should work These can be represented in statements like:

• If (intervention) ... Then ... (Outcome), and in some cases also conditions But (take care that…)

EE: contexts

• In the reconstruction of the theory, it is important to have an eye for the conditions and factors that play a role in a variety of contexts.

• The context (or conditions) can be physical, but also social, for example, certain groups or characteristics of target groups (an intervention works only for motivated students).

• Attention to circumstances helps explain why an intervention works better in one case than in the other case.

• The intervention works through certain mechanisms, mechanisms that make "things work" and can be seen as the driving force behind an intervention.

EE

• Creation of concrete causal schemes: Context-Mechanism-Outcome schemes– Problem mechanism– Intervention that generates Change Mechanism(s)

EE: field studies

• Test the theory, test the hypotheses• Enter the field studies. The researcher

must the different groups of actors in the picture.

• Consider what information to whom can be achieved, what should be asked from whom: who knows what?

• Formulate relevant questions for each (type of) informant to answer and think about the most appropriate method for each type involved persons.

Field studies

• Reconstruction and comparison with the policy theory takes place in a learning dialogue at different levels.

• Types (levels) of respondents:– Policy implementers (e.g. project leaders)– Intermediaries (e.g. teachers)– Target group (e.g. students, elderly, etc)

• Researchers: open attitude but focus on understanding and consensual knowledge (do we agree that this is what really happened?)

EE: Field studies

• Each actor from his own role in the operation of the policy can reflect on assumed mechanisms and give an explanation of (un) planned and (un) desired effects.

• The question to the audience itself is– whether the assumptions indeed work for them– are assumptions about their behavior correct,– what side effects they face.

• Cyclical process: stop rules...

EE: Critical evaluation

• Here the explanations are found why the policy intervention has or has not the desired effects, the goal of an explanatory evaluation.

• If the effects of an intervention are not as positive as expected, the researcher may ask the following questions: – Is the theory or policy measure(s) plausible?– Was the theory sufficiently differentiated? – Was implementation successful?– Were necessary conditions met? – What should be modified?

• Learning effects for policy makers…

Effects of EEE??

• What is the intervention? Is it similar to policy interventions?

• How to measure effects? Pre-test/post-test?

• Compare with reference/control groups?• What do you measure?

– Knowledge, skills, attitudes, higher order skills (reflection, learn to learn, learn to see activities as learning), motivation?

– Other, new things?

EE of EEE?

• What is the theory behind EEE?• Reconstruct the assumptions• Distinguish assumptions at levels:

– The ideas, the theories– The structures and systems developed and

used– The anticipated effects on various groups of

actors• Do field research• Evaluate whether your assumptions are

shared with various actors: test the hypotheses

Generic interview scheme

• THEN – NOW – LATER• Contexts-Mechanisms-Outcomes• Start with NOW• What Outcomes realised? Differentiate (level

of ideas-theories, structures-systems, behaviour)

• THEN: What Processes (mechanisms) caused the Outcomes?

• What crucial success & failure factors?• LATER: what new/adjusted outcomes?• How to achieve the outcomes?