DFID/ESRC RES-167-25-0263

Measuring student subject learning: in whose language(s)?Guoxing Yu, Pauline Rea-Dickins, Zuleikha Khamis*, Oksana Afitska, Haji Mwevura*, Shumbana Said*, Amour Khamis*, Abdulla Mohammed*

University of Bristol, *State University of Zanzibar

Student Performance in National Examinations: the Dynamics of Language (SPINE, 2007-2010)

Measuring student subject learning: in whose language(s)?Guoxing Yu, Pauline Rea-Dickins, Zuleikha Khamis*, Oksana Afitska, Haji Mwevura*, Shumbana Said*, Amour Khamis*, Abdulla Mohammed*

University of Bristol, *State University of Zanzibar

Student Performance in National Examinations: the Dynamics of Language (SPINE, 2007-2010)

1. 1 Introduction1. 1 Introduction

On a school wall in Kiswahili

Language of examinations

Impacts on society and educational policies:

national and international

Impacts on individuals:performance and progress

2. Focus of Study 11. What evidence is there that language of examination

constitutes a determining factor in the demonstration of subject knowledge of students whose first language differs from the language of examination

2. What assessment accommodations (including linguistic accommodations) are capable of addressing such linguistic unfairness in examinations?

3. What are the social and political impact of language of examination in Sub-Saharan African contexts?

2. Focus of Study 11. What evidence is there that language of examination

constitutes a determining factor in the demonstration of subject knowledge of students whose first language differs from the language of examination

2. What assessment accommodations (including linguistic accommodations) are capable of addressing such linguistic unfairness in examinations?

3. What are the social and political impact of language of examination in Sub-Saharan African contexts?

StudentsTest performance

Teaching andlearning

4. Examples of test itemsMaths

1. Calculate the lateral angles in the figure below.

2. Given that a2-b2=(a+b)(a-b). Evaluate (204)2-(196)2

3. In a certain hotel of 30 customers, 17 enjoy Cocacola, 19 enjoy Fanta and 2 don’t enjoy either. Fill in the venn diagram below and use it to find how many enjoy: i) Cocacola only, ii) Fanta only and iii) both Cocacola and Fanta?

4. The combined ages of Juma and Asha are 10 years. The difference of Asha’s age from twice Juma’s age is 8 years. Find the ages of each one.

Chemistry:

1. Name the process that could be used to obtain the named compound in the following mixture a) coconut oil from water, b) iodine and sand, c) salt from salt solution

Biology:

• Give a brief explanation on what will happen on the following: A locust not suffocating when its head is immersed in water while the remaining part of body is outside.

4. Examples of test itemsMaths

1. Calculate the lateral angles in the figure below.

2. Given that a2-b2=(a+b)(a-b). Evaluate (204)2-(196)2

3. In a certain hotel of 30 customers, 17 enjoy Cocacola, 19 enjoy Fanta and 2 don’t enjoy either. Fill in the venn diagram below and use it to find how many enjoy: i) Cocacola only, ii) Fanta only and iii) both Cocacola and Fanta?

4. The combined ages of Juma and Asha are 10 years. The difference of Asha’s age from twice Juma’s age is 8 years. Find the ages of each one.

Chemistry:

1. Name the process that could be used to obtain the named compound in the following mixture a) coconut oil from water, b) iodine and sand, c) salt from salt solution

Biology:

• Give a brief explanation on what will happen on the following: A locust not suffocating when its head is immersed in water while the remaining part of body is outside.

1.2 Context• NCLB Act of 2001 (USA) and Every Child Matters green

paper (2003, UK)

• World Declaration on Education for All and Framework for Action (UNESCO, 1990)

• For students in SSA, English is an exoglossic language and their teachers are learners of English too and are not necessarily competent users of English.

• … that poor countries often operate expensive and often complex language policies, whereas rich countries usually operate simple and relatively cheap language policies . (Williams 2006:1987)

1.2 Context• NCLB Act of 2001 (USA) and Every Child Matters green

paper (2003, UK)

• World Declaration on Education for All and Framework for Action (UNESCO, 1990)

• For students in SSA, English is an exoglossic language and their teachers are learners of English too and are not necessarily competent users of English.

• … that poor countries often operate expensive and often complex language policies, whereas rich countries usually operate simple and relatively cheap language policies . (Williams 2006:1987)

3. Further questions1. To what extent does the language background of test

takers affect their performance in content-based assessment delivered in English?

2. To what extent do test takers’ English language abilities (esp. reading and writing skills) affect their performance in such assessments?

3. To what extent is test takers’ performance affected by language complexity and linguistic demands of test items?

4. How might test accommodations (e.g. modifying linguistic demands of items) affect performance in content-based assessments?

3. Further questions1. To what extent does the language background of test

takers affect their performance in content-based assessment delivered in English?

2. To what extent do test takers’ English language abilities (esp. reading and writing skills) affect their performance in such assessments?

3. To what extent is test takers’ performance affected by language complexity and linguistic demands of test items?

4. How might test accommodations (e.g. modifying linguistic demands of items) affect performance in content-based assessments?

5. Impact on individuals in tests

1. AERA/APA/NCME 1999: 91

For all test takers, any test that employs language is, in part, a measure of their language skills. This is of particular concern for test takers whose first language is

not the language of the test.

2. Effects of language background/ability on test performance

3. Then, can TEST ACCOMMODATIONS (e.g. modification of linguistic complexity of items, use of native language, see Butler and Stevens 1997, 2001) provide a quick fix as the most direct and immediate policy remedy to address the complex issues surrounding “language in examinations”?

4. Effects of accommodations on performance:

(a) Modifications of linguistic demands of items

Studies using linguistic modifications for EAL learners in content-based assessment have drawn on conceptually from findings of psychological studies where a changing in the wording or structure of a test item has been observed to affect students (mainly L1 learners) performance, however, in practice, the effects of linguistic modifications on test performance on EAL learners appear much more complex, interacting with students language abilities, type of items and subject areas, etc.

(b) Using L1, or dual-language or side-by-side bilingual tests, e.g.

Duncan et al. (2005) and Abedi et al. (2006) both found dual-language maths tests did not affect performance, though preferred by test takers (Duncan et al. 2005).

Eisemon et al. (1993): the language of assessment, French or Kirundi, profoundly influenced the measurement of achievement in most subjects tested. And the most able students were most affected by being measured through French.

5. Impact on individuals in tests

1. AERA/APA/NCME 1999: 91

For all test takers, any test that employs language is, in part, a measure of their language skills. This is of particular concern for test takers whose first language is

not the language of the test.

2. Effects of language background/ability on test performance

3. Then, can TEST ACCOMMODATIONS (e.g. modification of linguistic complexity of items, use of native language, see Butler and Stevens 1997, 2001) provide a quick fix as the most direct and immediate policy remedy to address the complex issues surrounding “language in examinations”?

4. Effects of accommodations on performance:

(a) Modifications of linguistic demands of items

Studies using linguistic modifications for EAL learners in content-based assessment have drawn on conceptually from findings of psychological studies where a changing in the wording or structure of a test item has been observed to affect students (mainly L1 learners) performance, however, in practice, the effects of linguistic modifications on test performance on EAL learners appear much more complex, interacting with students language abilities, type of items and subject areas, etc.

(b) Using L1, or dual-language or side-by-side bilingual tests, e.g.

Duncan et al. (2005) and Abedi et al. (2006) both found dual-language maths tests did not affect performance, though preferred by test takers (Duncan et al. 2005).

Eisemon et al. (1993): the language of assessment, French or Kirundi, profoundly influenced the measurement of achievement in most subjects tested. And the most able students were most affected by being measured through French.

Interviews with teacher (video) and learners (audio) on the use of L1 and L2 in science learning

Interviews with teacher (video) and learners (audio) on the use of L1 and L2 in science learning

7. Social and political impacts1. The opportunity to use examinations as a lever for change

(Kellaghan & Greaney 1992, 2004) in monitoring and improving education quality in SSA are often missed, misused or even abused, leading to a “serious waste of scarce educational resources” (Kellaghan & Greaney 2004: 13), and

2. rather seriously, raises issues of social and individual inequality with discrimination “against minorities, rural populations, girls, and students whose first language differs from that of the examination” (ibid. 7)

3. and raises at least three questions in relation to educational policies:

i) Whether the problems of underachievement are related to a language of instruction that is different from the students’ home language as well as their teachers’?

ii) Whose language(s) should be used as a medium of instruction to meet the targets of quality Education for All?

iii) Which language(s) should be used for formal and high stakes examinations?

7. Social and political impacts1. The opportunity to use examinations as a lever for change

(Kellaghan & Greaney 1992, 2004) in monitoring and improving education quality in SSA are often missed, misused or even abused, leading to a “serious waste of scarce educational resources” (Kellaghan & Greaney 2004: 13), and

2. rather seriously, raises issues of social and individual inequality with discrimination “against minorities, rural populations, girls, and students whose first language differs from that of the examination” (ibid. 7)

3. and raises at least three questions in relation to educational policies:

i) Whether the problems of underachievement are related to a language of instruction that is different from the students’ home language as well as their teachers’?

ii) Whose language(s) should be used as a medium of instruction to meet the targets of quality Education for All?

iii) Which language(s) should be used for formal and high stakes examinations?

6. Impact on teaching and learningThree aspects of classroom learning and language use:

i) Code switching and mixing and discourse features of teacher talk, e.g. ‘safe talk’ (Hornberger & Chick (2001) accommodating students’ as well as teachers’ low language ability, translation as coping strategy (Brock-Utne & Holmarsdottir 2004).

ii) Classroom pedagogy: a limited range of pedagogic activities to engage learners in subject and language learning, within environments with very limited resources available.

iii) Learner classroom language: lack of opportunities for participation in class e.g., due to the unfamiliarity with the language of the instruction.

6. Impact on teaching and learningThree aspects of classroom learning and language use:

i) Code switching and mixing and discourse features of teacher talk, e.g. ‘safe talk’ (Hornberger & Chick (2001) accommodating students’ as well as teachers’ low language ability, translation as coping strategy (Brock-Utne & Holmarsdottir 2004).

ii) Classroom pedagogy: a limited range of pedagogic activities to engage learners in subject and language learning, within environments with very limited resources available.

iii) Learner classroom language: lack of opportunities for participation in class e.g., due to the unfamiliarity with the language of the instruction.

8. Some conclusions on the use of assessment accommodations

1. There are significant conceptual, methodological and political challenges of applying accommodations in a reliable, valid and feasible manner.

2. The inconsistency of research findings may be partly due to the complex nature and the interactions of accommodations with numerous other factors such as subject areas, student characteristics and the ways in which accommodations are implemented.

3. It is crucial to look beyond accommodations: to analyze the impact of instruction-embedded factors and the ways in which learners examination performance may be shaped by the nature of language use in teaching and learning in order to gain better understandings of the interrelationships between the language of assessment and the language of instruction.

8. Some conclusions on the use of assessment accommodations

1. There are significant conceptual, methodological and political challenges of applying accommodations in a reliable, valid and feasible manner.

2. The inconsistency of research findings may be partly due to the complex nature and the interactions of accommodations with numerous other factors such as subject areas, student characteristics and the ways in which accommodations are implemented.

3. It is crucial to look beyond accommodations: to analyze the impact of instruction-embedded factors and the ways in which learners examination performance may be shaped by the nature of language use in teaching and learning in order to gain better understandings of the interrelationships between the language of assessment and the language of instruction.