1

Examination Impact: Monitoring how teachers

cope with change

Dianne WallTania Horák

Lancaster University UK

2

Background

• Next Generation TOEFL (NGT) to be launched September 2005 (in certain sites only)

• Two impact studies commissioned by Educational Testing Service (producer of TOEFL)

3

Our impact study

• Where?Central and Eastern Europe

• Who?TOEFL preparation course providers

4

Phase 1 Phase 2 Phase 3

Baseline Transition Impact

Jan 03 to Oct 04 Oct 04 to Mar 06 After introduction of NGT

-Description of current TOEFL preparation classes-Class observations, interviews with teachers, DoS and students.-10 sites in 6 countries

-Tracking study -’Interviews’ via Internet

-Description of NGT preparation classes -Class observations, interviews with teachers, DoS and students.- same sites

5

Theoretical underpinnings 1) Henrichsen (1989) – Hybrid Model

of the Diffusion / Implementation Process

2) Messick (1996) - ‘evidential link’

6

Henrichsen’s model

Antecedents Process Consequences

7

Process • The Receivers• The Innovation• Characteristics of Communication• The Resource System • The User System • Inter-elemental factors

8

How our study fits the model:

Antecedents Process Consequences

Baseline Transition Impact

9

Messick’s ‘evidential link’

Antecedents Process Consequences

‘evidential link’

10

Phase 2• Who? 6 teacher participants from

Phase 1 • Where? 5 countries: Bulgaria, Croatia,

Lithuania, Poland, Slovakia.• When? data collection period: January

- May 05• How? Computer Mediated

Communication• What? investigation of pps’ awareness

and understanding of and reactions to NGT, to help us understand how they are preparing for the arrival of NGT

11

What?• Interviews based on monthly

tracking questions • Completion of tasks designed to

familiarise participants with NGT • Interviews discussing the task

responses

12

Data collection activities Interview 1

Task sentto pps

Pps return taskResearchers

discuss responses

Interview 2

13

The Tasks• January - Features of TOEFL preparation

classes

• February - Awareness of current and future TOEFL

• March - Focus on the new writing tasks

• April - Focus on the new speaking tasks

• May - Prediction of actual content and methodology of pps’ NGT preparation classes

14

Findings

• Understanding of NGT

• Materials

• Teacher Support

15

Understanding of NGT

Teachers seem clear about the format of NGT but less clear about the actual constructs.

16

Understanding of NGT

‘I don't think the matter of accent was really addressed, though maybe those who grade the tests have an implicit understanding of how accent plays a role and will score accordingly. In preparing students for tests, it would help to have the issue of accents and how they affect scoring addressed directly in the rubric.’

17

Understanding of NGT

Conclusion

Awareness-raising requires more than providing examples of the test.

18

Materials

Teachers rely heavily on published materials.

19

Materials ‘I can’t even dare think of something like that but I suppose I’ll have to improvise. The other possibility is to reschedule the [NGT] classes to the moment when there will be materials.’

20

Materials

ConclusionsImpact will probably be affected by the textbook writers Impact of the test may well be hard to distinguish from impact of the textbook

21

Teacher Support

The test provider has supplied various forms of teacher support but they are not necessarily accessible to the teachers who need them.

22

Teacher Support

‘I just heard it mention by [the Director] … She also said it is very expensive …. I cannot afford, the center cannot afford.’

23

Teacher SupportConclusion

Careful consideration must be paid to the logistics of teacher support if it is to be effective and reach those who need it, when they need it.

24

Professionalism• Test Provider - by commissioning

impact studies

• TOEFL teachers – by participating in our study

25

For further information • About the project:d.wall@lancaster.ac.uk t.horak@lancaster.ac.uk

• About NGT:http://toeflpractice.ets.org

26

27

Tracking questions• Have you learnt anything about New TOEFL

which you didn’t know last month? If so, what?•  Is the new exam being discussed in your

institution (e.g. by other teachers/ management)? If so, what are the main issues being discussed? 

• Have students asked about the new TOEFL? If so, what have they asked about?

• Are you worried/ concerned about anything concerning the new TOEFL? If so, what?

• Is there anything else of interest concerning new TOEFL which has happened this month?