Testing English as a Foreign Language

Two EFL-Tests used in Germany

Sebastian Kluitmann

Philologische Fakultt

Albert-Ludwigs-Universitt Freiburg

Name: Sebastian Kluitmann

Anschrift: Staufener Strae 33

79115 Freiburg im Breisgau

Erklrung zur Wissenschaftlichen Arbeit

Hiermit versichere ich, dass ich die vorliegende Arbeit selbststndig und nur mit

den angegebenen Hilfsmitteln angefertigt habe und dass alle Stellen, die dem

Wortlaut oder dem Sinne nach anderen Werken entnommen sind, durch Angabe

der Quellen als Entlehnungen kenntlich gemacht worden sind.

Datum: 15.04.2008

Unterschrift:

Testing English as a Foreign Language

Two EFL-Tests used in Germany

Wissenschaftliche Arbeit

zur

Erlangung des ersten Staatsexamens

fr das Lehramt an Gymnasien

der Philologischen Fakultt der

Albert-Ludwigs-Universitt

Freiburg im Breisgau

vorgelegt von

Sebastian Kluitmann

geboren in

Freiburg im Breisgau

April 2008

Anglistik

Acknowledgements

This paper would not have been possible without the help of many people.

I would like to thank Prof. Dr. Bernd Kortmann for supervising this thesis. Thanks are also

due to Ralf Meurer of Sprachenmarkt.de for prompting the idea for this paper as well as

making part three possible. I am indebted to Dr. Glenn Fulcher, who was kind enough to

share his work with me. My thanks go to Sheila Gordon-Schrder for providing valuable

feedback on various parts of the manuscript. I would also like to thank Ellie Purkis, who

kindly proofread the manuscript.

Last but not least I would like to thank all family and friends for general support and

encouragement.

Table of Contents

i

PREFACE 1

INTRODUCTION THE HISTORY OF LANGUAGE TESTING 4

1 LANGUAGE TESTING 8

1.1 What is a language test? 8

1.2 Reliability and validity 10 1.2.1 Reliability 10 Classical True Score theory 11 G-theory 16 Item response theory 18 1.2.2 Validity 22 Construct validity 23 Content validity 24 Concurrent validity 26 Predictive Validity 27 Face validity 28

1.3 Washback 29

1.4 Impact 31

1.5 Ethics and standards test uses 33

1.6 Test and item types 35 1.6.1 Reading Comprehension 35 1.6.2 Listening Comprehension 38 1.6.3 Writing 39 1.6.4 Speaking 41 1.6.5 Grammar 42

1.7 The CEF 43

2 ETS TOEIC AND THE KMK-ZERTIFIKAT 46

2.1 The TOEIC-Test 47 2.1.1 The TOEIC history and (cl)aims 47 Critical reception 48 2.1.2 Description and evaluation of the TOEIC-tests design 56 Format 56 Development 57 2.1.3 Description and evaluation of the TOEIC-tests item types 61 Reading Comprehension 61 Listening Comprehension 62

Table of Contents

ii

2.2 The KMK-Zertifikat 64 2.2.1 The KMK-Zertifikat - history and (cl)aims 64 2.2.2 Description and evaluation of the KMK-Zertifikats design 66 Format 66 Development 66 2.2.3 Description and evaluation of the KMK-Zertifikats item-types 68 Reading Comprehension 68 Listening Comprehension 70 Writing 73 Speaking 75

2.3 Conclusion: summary perspectives 76 2.3.1 Summary 76 2.3.2 Perspectives 79

3 SURVEY 80

3.1 Method 80

3.2 Hypotheses 81

3.3 Analysis 82

BIBLIOGRAPHY 90

Preface

1

Preface

In our globalised world, being able to speak one or more foreign languages is a

prerequisite, as employers on a national as well as on an international scale pay

attention to the foreign language skills of their future employees (cf. Morfeld

2003: 385, Sommer 2005: 3 and Bauer/Toepfer 2004: 20), focusing mostly on

English. For English is still the undisputed lingua franca of the modern work

force; even despite the European Councils attempts to further the diversity of

languages. Actually, the European Council acknowledges that in their own

institution, there have always been limits on multilingualism [...which] are

dictated by both practical considerations and budgetary constraints, in the

interests of keeping operating expenditure down (COE 2007). Needless to say

that Global Players, internationally and multi-culturally organised viz. oriented

companies do not think differently. This is why for millions of learners around

the world the ability to communicate in English is the passport to economic

prosperity, social mobility and educational advancement (Fulcher 2007). So, to

increase their chances on the job-market, they devote both time and money to

having their English language skills assessed and attested.

In the following paper, I will scrutinise and compare two tests of English as a

Foreign Language: ETS TOEIC-test on the one hand and the German Ministry of

Educations KMK-Fremdsprachenzertifikat. For in the case of the TOEIC-test,

there has been relatively little independent research (cf. Cunningham 2002: 1)

despite its apparent popularity in fact, with more than 4 million test-takers

worldwide each year, it is the most widely used test for English as a Foreign

Language. The situation for the KMK-Zertifikat is even more remarkable.

Although it has been offered since 1998 and is passed off as being internationally

recognised, measures to ensure the appropriateness of its design and results as

well as the consistent application of standards have only recently been

implemented (cf. Dill et. al. 2005: 1). Furthermore, apart from the

aforementioned study, which was conducted by the developers of the test

Preface

2

themselves, there has been no evaluation of the KMK-Zertifikat whatsoever. The

decision to choose the TOEIC-test and the KMK-Zertifikat was thus motivated by

the evident lack of independent research regarding the respective tests as well as

a keen interest in the comparison of perhaps differing test designs and test

methods.

In the first chapter Language Testing, I will provide an introductory summary

of the current state of research by investigating language testing in general and

the intricacies and problems involved, touching issues such as the different sorts

of test design/method, the phenomenon of washback, reliability and validity as

well as more ethical considerations questioning the use of tests or even the very

standards they are based on. Once this groundwork is laid, I will begin with the

actual comparison of the two tests in chapter two ETS TOEIC and the KMK-

Zertifikat, matching them against the criteria developed in chapter one. Chapter

three Survey concludes with a survey of language testing at school and the two

tests popularity based on an empirical study conducted from September to

November 2006. 230 grammar schools (Gymnasien) in Baden-Wrttemberg were

contacted and presented with a questionnaire concerning the significance of

foreign languages for the particular school, the interest in and present use of

language tests as well as the popularity of major tests of English as a foreign

language. The 142 (61.74 %) questionnaires that were sent back provide the

foundation of the third chapter. Apart from the obvious question of how familiar

the teachers are with the two tests, I will examine whether there is a correlation

between certain basic background conditions such as the quantity of pupils, the

location of the school or the implementation of foreign languages in the school

profile and, say, the availability of external certification at this school or the

teachers degree of familiarity with different tests. In any way, the teachers can be

seen as playing an important role in promoting certain tests, as pupils are likely

to ask them which test they should take for their specific purposes.

Preface

3

However, an area which might merit further research is the question of the

popularity of various different tests with employers, viz. recruiters, as existing

information tends to focus either on the acceptability of tests in a particular, often

academic, context1, or is restricted to a particular test itself2. Thus, at the moment,

a comparison of the tests in this respect is impossible. First tentative

investigations in this area can be detected in the TOEIC marketing material on the

one hand and Wagners study as cited by Dill ( Dill et al. 2005: 5/6) on the

other. The respective findings will be covered in more detail in chapter 2.

1 Consider e.g. universities criteria for admission to certain programmes: some accept the

TOEFL only, some demand that future students take the IELTS, others are more liberal

and allow various different tests as well.

2 Although studies may reveal that in their ads, a certain number of companies refer to a

particular tests results to describe the applicants desired English language abilities, it

has yet to be shown that candidates with similar skills, yet a different certificate proving

them would not be considered.

Introduction the history of language testing

4

Introduction the history of language testing

The history of testing can be traced back a long way. Play is one of the basic

phenomena constituting the human condition (cf. Fink 1995: 356 ff) and in play,

humans compete against each other and test their abilities. Game and play reveal

the positive aspect of testing from th