Functional English skills in Ethiopia, India and

Vietnam: comparing English ability and use

amongst 15-year olds in three countries

Gayatri Vaidya (Educational Initiatives)

Rhiannon Moore (Young Lives, University of Oxford)

UKFIET Conference 2017

Young Lives school survey overview

Functional English & its importance

Development of repeated measures

cross-country English tests

Findings:

A common scale across three countries:

how much progress do children make?

What functional English skills do children

have?

What are the characteristics of children

with higher or lower levels of English?

Discussion and next steps

PRESENTATION OUTLINE

YOUNG LIVES & LINKED SCHOOL SURVEYS

Longitudinal survey of children,

their households, schools and

communities running for 15 years

in 4 countries

Young Lives school surveys:

introduced in 2010 with a sub-

sample of YL children and peers

2016-17 school surveys: school

effectiveness in Ethiopia, India

and Vietnam

• Ethiopia: upper primary (Grades 7-8)

• India: lower secondary (Grade 9)

• Vietnam: upper secondary (Grade 10)

School effectiveness design:

Student performance in terms of progress (rather

than cross-sectional measure)

Teaching and learning processes affecting progress

The ‘value-added’ of one year of school

To do this, we administered:

Cognitive tests at beginning and end of one school

year:

Functional English

Maths

Transferable Skills (end of year only)

Background instruments and psychosocial measures

to contextualise learning progress

SCHOOL SURVEY DESIGN

The 2016-17 school survey is the first time Young Lives has tested English across multiple countries, due to:

• Policy demand

• Global relevance

• Transferable skill

Conceptualising our area of interest as functional English:

“application of […] skills in purposeful contexts and scenarios that reflect real-life situations” (OFQUAL 2011: 10)

Different purposes for learning English – what does success look like?

Conceptualising ‘functional’ EnglishWHY TEST FUNCTIONAL ENGLISH?

Perceptions

of English

Exposure to

English

… as an

aspirational

language

… as a

language of

social

mobility

… as an

equaliser…

exacerbating

existing

inequalities

… as a

neutral

language

… as a

global

language

Medium of

instruction School

subject

Teacher

proficiency

Private

tuition

Urban/ruralFamily and

community

ENGLISH: PERCEPTION & EXPOSURE

SKILL STRUCTURE & BLUEPRINT DESIGN

Sounds & Letters

Words & meanings

Sentence structure & comprehension

Reading and comprehension

Weightage to different level of Language acquisition

Focus on ‘usability’ and ‘working understanding’ than

standardness

Test items focussed on basic knowledge of words, ability to derive

meaning and being able to understand simple dialogues

Reading- Comprehension included simple stories, Scientific text &

posters

CROSS-COUNTRY TEST DESIGN The assessments were

designed for Wave 1 &

Wave 2 for all three

countries

Depending on the pilot

results, items were

selected for each

country

There were common

items within the wave

and between the wave

to create comparative

data points

The item selection

ensured presence of

items at different

difficulty levels

WAVE 1: SKILL-WISE PERFORMANCE

% Performance on each skill for

each nation was recorded, which

gave indicative numbers for

overall levels.

Since the tools were not identical,

these scores are not comparable

between nations.

0

10

20

30

40

50

60

70

80

S1 S2 S3 S4

0

10

20

30

40

50

60

70

S1 S2 S3 S4

0

10

20

30

40

50

60

70

80

S1 S2 S3 S4

S1 Letters, Sounds & Spellings

S2 Words: Meanings & contexts

S3 Sentence: Structure & understanding

S4 Reading and Comprehension

Eth

iopia

India

Vie

tnam

0

.002

.004

.006

.008

Den

sity

250 500 750English Score

YL India W1 YL India W2

YL Vietnam W1 YL Vietnam W2

YL Ethiopia W1 YL Ethiopia W2

English Performance by YL Country and Wave A common scale for

cross-country and

cross-wave anchor

items was created

using IRT

The scale was

transformed to have

a mean of 500 (in

Wave 1) and a

standard deviation

of 100

In one school year

students in our

sample made an

average of 13 points

of progress on this

common scale

But we also saw variation in scores and

progress across and within the three

countries

CROSS-COUNTRY ANALYSIS – A COMMON SCALE

0

.002.004.006.008

Density

250

500

750

En

glis

h S

core

YL

Eth

iop

ia W

1Y

L Ind

ia W

1Y

L V

ietn

am

W1

0

.002

.004

.006

.008

Den

sity

250 500 750English Score

YL Ethiopia W1 YL India W1 YL Vietnam W1

Level 1: Basic User

Level 2: Competent User

Level 3: Advanced User

SCALE ANCHORING

0

.002.004.006.008

Density

250

500

750

En

glis

h S

core

YL

Eth

iop

ia W

1Y

L Ind

ia W

1Y

L V

ietn

am

W1

0

.002

.004

.006

.008

Den

sity

250 500 750English Score

YL Ethiopia W1 YL India W1 YL Vietnam W1

Level 1 (basic user):

At this level, students can:

• Identify everyday vocabulary

(e.g. bag, banana);

• Recognise familiar antonyms

(e.g. start/stop);

• Comprehend short, simple

sentences and complete simple

dialogues with appropriate

questions or answers.

SCALE ANCHORING: BASIC USER

SCALE ANCHORING: COMPETENT USER0

.002.004.006.008

Density

250

500

750

En

glis

h S

core

YL

Eth

iop

ia W

1Y

L Ind

ia W

1Y

L V

ietn

am

W1

0

.002

.004

.006

.008

Den

sity

250 500 750English Score

YL Ethiopia W1 YL India W1 YL Vietnam W1

Level 2 (competent user):

At this level, students can:

• Use correct question words

(e.g. where, when);

• Complete simple sentences

using appropriate verbs and

adjectives (e.g. lazy, slow,

eat);

• Read short, simple texts and

identify specific information

which can be retrieved directly

from the text.

SCALE ANCHORING: ADVANCED USER0

.002.004.006.008

Density

250

500

750

En

glish S

core

YL

Eth

iop

ia W

1Y

L Ind

ia W

1Y

L V

ietn

am

W1

0

.002

.004

.006

.008

Den

sity

250 500 750English Score

YL Ethiopia W1 YL India W1 YL Vietnam W1

Level 3 (advanced user):

At this level, students can:

• Recognise less commonly used

vocabulary (e.g. whisper)

• Identify the meaning of

unfamiliar words from the

context in which they are used

(e.g. scarce, precise);

• Read more complex texts on

abstract themes and make

inferences about purpose and

intent based on the entire text.

WHO ARE THE STUDENTS AT EACH LEVEL?

020

40

60

80

10

0

Perc

en

t

Level 1 Level 2 Level 3

Bottom tercile (poorest)

Middle tercile

Top tercile (least poor)

020

40

60

80

10

0

Perc

en

t

Level 1 Level 2 Level 3

Neither parent can read Only mother can read

Only father can read Both parents can read

020

40

60

80

10

0

Perc

en

t

Level 1 Level 2 Level 3

Rural

Urban

Clear differences in the

background characteristics of

students at each level

Students at level 1 (basic users of

English) are more likely to be in

rural areas, from poorer

households and have illiterate

parents than those at other levels

(across all three countries)

Findings suggest that functional English

skills are unevenly distributed

Both where children are (which country)

and who they are (background, wealth,

locality) affects their functional English

skills

This has clear implications for equality of

future opportunities, given the increasing

importance of English– suggests English

skills may act as a ‘divider’ rather than an

‘equaliser’?

Next steps:

Exploring potential for linking items to the CEFR

scale to further the knowledge on ‘can-do’

indicators at different levels

‘Value-added’ analysis of the English test data to

understand more about the schools and classes

where children learn more

Exploration of the student, school and teacher

level factors relating to progress in English

across three very different country contexts

DISCUSSION AND NEXT STEPS

Thank you!

Any questions or comments?