USAID SINDH READING PROGRAM EARLY GRADE READING …The tasks for both the Grade 3 and Grade 5 EGMAs...

USAID SINDH READING PROGRAM

EARLY GRADE READING AND MATHEMATICS ASSESSMENT BASELINE REPORT

SINDH PROVINCE, SUKKUR DISTRICT

Contract No. AID-391-C-14-00001

Submission Date: 14 October 2015

This document was produced for review by the United States Agency for International Development. It was

prepared by Chemonics International, Inc. and School-to-School International.

EARLY GRADE READING AND MATHEMATICS ASSESSMENT BASELINE STUDY REPORT SINDH PROVINCE, SUKKUR DISTRICT

Contracted under Order No. AID-391-C-14-00001

DISCLAIMER The authors’ views expressed in this document do not necessarily reflect the views of the United States Agency for International Development or the United States Government.

SRP BASELINE REPORT, SINDH PROVINCE, SUKKUR DISTRICT-USAID SINDH

OCTOBER 2015 i

ACKNOWLEDGEMENTS

The School-to-School International and Chemonics teams would like to thank USAID/Pakistan for their support. We would also like to recognize the stellar faculty and staff from the Sindh Education and Literacy Department, the Bureau of Curriculum, the Provincial Education Assessment Centre (PEACE) Sindh, and the Provincial Institute of Teacher Education (PITE) Sindh. Finally, this endeavor would not have been possible without the dedication of the Master Trainers, Quality Control Officers, data collection partners, subject experts and administrators who lent their continuous support throughout the assessment activity. The following persons played a major role in the process: Mark Lynd (School-to-School International) Aftab Khushk (School-to-School International) Hetal Thukral (School-to-School International) Beth Fincham (School-to-School International) Gaëlle Simon (School-to-School International)


OCTOBER 2015 ii

CONTENTS

Contents ........................................................................................................................................................................ ii

Acronyms ..................................................................................................................................................................... vi

Executive Summary .................................................................................................................................................... 1

Chapter 1: Introduction ............................................................................................................................................ 10

Chapter 2: Methodology ........................................................................................................................................... 14

Chapter 3: EGRA Results – Sukkur District ........................................................................................................ 22

Chapter 4: EGMA Results – Sukkur District ....................................................................................................... 36

Chapter 5: Contextual Variables and EGRA and EGMA Performance .......................................................... 43

Annexes ....................................................................................................................................................................... 48


OCTOBER 2015 iii

List of Tables and Figures

Table 1: Baseline Schools in Sukkur ....................................................................................................................... 15

Table 2: Baseline Participants in Sukkur ................................................................................................................ 15

Table 3: EGRA Test Structure ................................................................................................................................ 16

Table 4: EGMA Test Structure ............................................................................................................................... 17

Table 5: Enumerators training by District ............................................................................................................. 19

Table 6: Reliability Estimates ................................................................................................................................... 21

Table 7: EGRA Percent Correct Scores by Grade and Task .............................................................................. 23

Table 8: Average Phonics and Reading Fluency by Grade ................................................................................. 24

Table 9: Average Phonics and Reading Fluency by Grade and Gender ........................................................... 27

Table 10: Fluency Rates by PRP Thresholds And Grade ................................................................................... 28

Table 11: Grade 3 Reading Fluency and Reading Comprehension ................................................................... 28

Table 12: Grade 5 Reading Fluency and Reading Comprehension (Grade 2-level Text) .............................. 29

Table 13: Grade 5 Reading Fluency and Reading Comprehension (Grade 4-level Text) .............................. 30

Table 14: EGRA Proportion of Students with Zero-scores by Grade and Task ............................................ 31

Table 15: EGRA Proportion of Students with Zero-scores by Grade, Gender and Task ............................ 32

Table 16: EGMA Percent Correct Scores by Grade and Task ........................................................................... 37

Table 17: Mathematics Fluency Rates by Grade ................................................................................................... 37

Table 18: Mathematics Computation Fluency Rate Means by Grade and Gender ......................................... 40

Table 19: EGMA Proportion of Students with Zero-scores by Grade and Task ........................................... 40

Table 20: EGMA Proportion of Zero-scores by Grade, Gender and Task ..................................................... 41

Table 21: Contextual variables from Student Questionnaire Associated with EGRA or EGMA Performance ...................................................................................................................................................... 44

Table 22: Contextual variables from Teacher and Head Teacher Questionnaire Associated with EGRA or EGMA Performance ........................................................................................................................................ 45

Table 23: Sukkur Reliability Estimates ................................................................................................................... 48

Table 24: EGRA Tasks Statistics for Sukkur ........................................................................................................ 48

Table 25: EGMA Tasks Statistics for Sukkur ....................................................................................................... 49

Table 26: EGRA Grade 3 Score Ranges and Calculations .................................................................................. 50

Table 27: EGRA Grade 5 Score Ranges and Calculations .................................................................................. 51

Table 28: Example of EGRA Percent Correct and Summary Scores ............................................................... 51

Table 29: Example of EGRA Timed Task Scores ............................................................................................... 51

Table 30: EGMA Grade 3 Score Ranges and Calculations ................................................................................. 52

Table 31: EGMA Grade 5 Score Ranges and Calculations ................................................................................. 52

Table 32: Example of EGMA Percent Correct and Summary Scores .............................................................. 53

Table 33: Example of EGMA Timed Task Scores .............................................................................................. 53

Table 34: Complete EGRA Item Statistics by Grade for Sukkur ...................................................................... 54

Table 35: Complete EGMA Item Statistics by Grade for Sukkur ..................................................................... 55

Table 36: Distribution of Students By Primary Language Spoken At Home .................................................. 59

Table 37: Distribution of Students Speaking the Same Language at Home as the Test .............................. 59

Table 38: Average EGRA/EGMA Scores by Reading Material at Home ....................................................... 60

Table 39: Percentage of students by Stories Read Aloud to Student at Home................................................ 60

Table 40: Average EGRA/EGMA Scores by Practice Reading Stories Aloud to Someone at Home........ 61

Table 41: Average EGRA/EGMA Scores by Silent Reading at Home ............................................................ 61

Table 42: Average EGRA/EGMA Scores by Read Quran at Home ............................................................... 61

Table 43: Average EGRA/EGMA Scores by Study or Complete Classwork With Classmates .................. 62


OCTOBER 2015 iv

Table 44: Average EGRA/EGMA Scores by students Missing School Days During the Previous Week 62

Table 45: Average EGRA/EGMA Scores by Parent’s Knowledge of School Performance ........................ 62

Table 46: Distribution of students by students Reporting Working Before or After School ....................... 63

Table 47: Average EGRA/EGMA Scores by Students Who Watch Television at Home ............................ 63

Table 48: Average EGRA/EGMA Scores by Students Listening to the Radio at Home ............................. 63

Table 49: Distribution of students by Computer Access at Home ................................................................... 64

Table 50: Distribution of students by Type of Vehicles at Home ..................................................................... 64

Table 51: Distribution of Teachers’ Academic Qualifications ........................................................................... 64

Table 52: Distribution of Teachers’ Professional Qualifications ....................................................................... 65

Table 53: Average EGRA/EGMA Scores by Teaching Multi-grade Classes .................................................. 65

Table 54: Average EGRA/EGMA Scores by Teachers Using Books Other Than Textbooks in the Classroom .......................................................................................................................................................... 65

Table 55: Distribution of Classroom Tools used by teachers ............................................................................ 66

Table 56: Distribution of Teaching Practices ........................................................................................................ 66

Table 57: Methods by which Teachers’ Udate their Knowledge of Teaching ................................................. 67

Table 58: Distribution of Head Teachers’ Academic Qualifications ................................................................. 67

Table 59: Distribution of Head Teachers’ Professional Qualifications ............................................................ 67

Table 60: Distribution of Head Teachers Teaching Classes ............................................................................... 68

Table 61: Distribution of Head Teachers Who Received Training for Implementing Reading Programs . 68

Table 62: Distribution of Head Teachers Who Received Training for Implementing Mathematics Programs ............................................................................................................................................................ 68

Table 63: Distribution of Schools having a Timetable ........................................................................................ 68

Table 64: Average EGRA/EGMA Scores by School timetable ........................................................................ 69

Table 65: Average EGRA/EGMA Scores by Including Reading in the timetable ......................................... 69

Table 66: Schools by Including Math in the timetable ........................................................................................ 69

Table 67: Distribution of Head Teachers Supporting Reading Instruction ..................................................... 70

Table 68: Distribution of Head Teachers Supporting Mathematics Instruction ............................................. 70

Table 69: Average EGRA/EGMA Scores by Head Teacher’s Knowledge About Student Progress ......... 70

Table 70: Average EGRA/EGMA Scores by Head Teachers’ ObservinG teachers’ use of Lesson Plans 71

Table 71: Distribution of The Person Responsible for Observing Teachers in the Classroom ................... 71

Table 72: Distribution of SMC Decision Making Authority/Responsibility ................................................... 71

Table 73: Distribution of Facility Improvements ................................................................................................. 72

Figure 1: Grade 3 EGRA: Percent Correct Scores by Task .................................................................................. 3

Figure 2: Grade 3 EGRA: Proportion of Zero-Score Students by Task ............................................................ 4

Figure 3: Grade 5 EGRA: Percent Correct Scores by Task .................................................................................. 4

Figure 4: Grade 5 EGRA: Proportion of Zero-Score Students by Task ............................................................ 4

Figure 5: Grade 3 EGMA: Percent Correct Scores by Task ................................................................................. 5

Figure 6: Grade 3 EGMA: Proportion of Zero-Score Students by Task ........................................................... 6

Figure 7: Grade 5 EGMA: Percent Correct Scores by Task ................................................................................. 6

Figure 8: Grade 5 EGMA: Proportion of Zero-Score Students by Task ........................................................... 6

Figure 9: Evaluation Design ..................................................................................................................................... 12

Figure 10: Distribution of EGRA Scores ............................................................................................................. 22

Figure 11: Grade 3 Scores by Task and Gender ................................................................................................... 26

Figure 12: Grade 5 Scores by Task and Gender ................................................................................................... 27

Figure 13: Grade 3 Reading Performance levels and Reading Comprehension Intervals ............................. 29

Figure 14: Grade 5 Reading Performance levels and Comprehension Intervals (Grade 2-level text) ......... 30

Figure 16: EGRA Summary scores by District ..................................................................................................... 33


OCTOBER 2015 v

Figure 17: Distribution of EGMA Scores ............................................................................................................ 36

Figure 18: Grade 3 EGMA Scores by Gender ..................................................................................................... 39

Figure 19: Grade 5 EGMA Scores by Gender ..................................................................................................... 39

Figure 20: Explanation of Box Plots ...................................................................................................................... 56

Figure 21: Phonics and Reading-Rate Fluency Box Plots for Grade 3 ............................................................. 57

Figure 22: Phonics and Reading-Rate Fluency Box Plots for Grade 5 ............................................................. 57

Figure 23: Mathematics Fact-Rate Fluency Box Plots for Grade 3 Figure 24: Mathematics Fact-Rate Fluency Box Plots for Grade 5 ....................................................................................................................... 58


OCTOBER 2015 vi

ACRONYMS

AEO Area/Assistant Education Officers

BoC Bureau of Curriculum

CWPM

EDO

Correct Words Per Minute

Education District Officers EGMA Early Grade Mathematics Assessment

EGRA Early Grade Reading Assessment

GoS Government of Sindh

I-SAPS Institute of Social and Policy Studies

MDC Management and Development Center

MT Master Trainer

PEACE Provincial Education Assessment Centre

PITE Provincial Institute of Teacher Education

PMIU Program Monitoring and Implementation Unit

PRP Pakistan Reading Project

PTC Primary Teaching Certificate

QCO Quality Control Officer

SBEP Sindh Basic Education Project

SEMIS Sindh Education Management Information System

SMC School Management Committee

SRP Sindh Reading Program

STBB Sindh Textbook Board

STS School-to-School International

USAID

VTT

United States Agency for International Development

Voice Tel Tech


OCTOBER 2015 1

EXECUTIVE SUMMARY

Introduction

The Sindh Reading Program (SRP) is a collaborative initiative of the Government of Sindh (GoS) and

USAID/Pakistan that aims to help 410,400 students in Sindh province acquire the foundational reading

and mathematics skills in the early grades to set them up for success in later grades (the project’s primary

emphasis is reading). To this end, SRP’s interventions include teacher training to improve early grade

reading instruction and assessment practices; the provision of additional reading materials; and activities

to enhance parent and caregiver engagement in early literacy for students in Grades 2 and 4. 1 SRP’s

interventions are being implemented across eight districts in the province of Sindh in Pakistan: Dadu,

Jacobabad, Kashmore, Khairpur, Larkana, Kambar-Shahdadkot, Sukkur, and five towns in Karachi. By

2018 (year 5 of the program), one of SRP’s key indicators is to improve reading for at least 200,887 Grade

2 students.2

To determine the extent to which students’ reading skills improve as a result of participating in SRP

interventions, an evaluation was designed to assess students’ reading skills at three time-points in the

project’s life cycle – at baseline (i.e., prior to the start of SRP interventions), at midline (i.e., at the halfway

point of the SRP project in 2016) and at endline (i.e., at the conclusion of SRP interventions in 2018).3

This report provides results of the baseline conducted in September 2014; midline and endline results will

be compared to this baseline in order to measure the impact of SRP interventions. The baseline draws on

data collected using the Early Grade Reading Assessment (EGRA) in order to assess students’ reading

skills, as well as the Early Grade Mathematics Assessment (EGMA) in order to assess their mathematics

skills. The baseline also included interviews with students, teachers and Head Teachers in order to capture

contextual variables that may be associated with strong reading and mathematics outcomes.4

This report summarizes the results of the baseline conducted in Sukkur district. It describes the baseline

methodology; results for EGRA and EGMA, and teacher and head teacher questionnaires; and

recommendations for improving SRP’s interventions.

Methodology

Sampling. The EGRA and EGMA assessments were administered in September 2014 to 12,793 students in 560 schools across eight districts of Sindh Province that were participating in SRP: Dadu, Jacobabad, Kashmore, Khairpur, Larkana, Kambar-Shahdadkot, Sukkur – assessed in Sindhi-language - and five towns in Karachi (which is treated as a district for this study, assessed in Urdu-language). To select students for participation in the baseline student assessments, a sample of students were selected using a stratified random sampling approach, making it possible to generalize the results of this baseline to all

1 In its original design, along with a focus on improving reading skills among students in the early grades, SRP also included interventions to improve student’s numeracy skills. This design was subsequently revised to retain only the core indicator of improving reading skills in the early grades; however, this revision came after baseline data had already been collected. As such, baseline data on both students’ literacy and numeracy were collected and analyzed; at midline and endline, data on literacy only will be collected. 2 These targets reflect revisions as of April 2015. 3 The evaluation utilizes a non-experimental pre-test/post-test design, in which students participating in the intervention are assessed prior to the start of the interventions (pre-test) and following the conclusion of the interventions (post-test); additionally, a mid-point measure is also collected. This design was appropriate for this study since an adequate comparison group (i.e., students in schools not participating in SRP interventions, but similar otherwise) was not feasible. A comparison group is necessary in order to establish impact of the interventions (in other words, to be able to attribute all the changes from pre-test to post-test among participating students to the SRP interventions). 4 Student questions were related to language spoken at home, availability of reading materials at home, study habits, access to communication technology and vehicles at home. Teacher and head teacher questions were related to qualifications, attendance at trainings, scheduling, teacher support, head teacher knowledge of student performance, teacher observation procedures, School Management Council (SMC) authority and facility improvements.


OCTOBER 2015 2

students in SRP Sindhi-language schools in the seven districts and all students in SRP Urdu-language schools in the five towns participating in SRP in Karachi City. In Sukkur, the assessment tools were successfully administered in 70 schools to 1,583 students (94 percent of the goal), 65 grade 3 and 65 grade 5 teachers (93 percent of the goal), and 70 head teachers (100 percent). The percent of teachers sampled was slightly lower than the sampling rate of head teachers because some teachers teach both third and fifth grade. It should be noted that the results from this baseline are considered a measure of learning at the end of Grades 2 and 4, even though the baseline was conducted for Grades 3 and 5 students. The decision to conduct the baseline with this population was made because of the requirement to conduct the baseline before the startup of project activities in Year 1. This timing presents a complication – the effects of “summer learning loss” – i.e., that students typically forget some of what they have learned in a given school year over the summer. The extent of this loss cannot be estimated and the results presented here may in fact be underestimating the actual performance of students prior to the break. In other words, assessments conducted after the summer break will reflect a short-term decline in student achievement. Note that SRP midline and endline assessments will be conducted at the same point in the school year in order to ensure comparability of results across time points. Instrument Development. The baseline EGRA and EGMA tools were developed by SRP to measure core reading and mathematics skills targeted by SRP in its teacher training, instructional materials, and parental involvement interventions. Six types of reading skills were tested with the EGRA: phonemic awareness, phonics, vocabulary, fluency (tested, as appropriate, using passages at two different reading levels: Grade 2 and Grade 4), reading comprehension (Grade 2 and Grade 4-level text) and listening comprehension. Seven types of mathematics skills were tested with the EGMA: number identification, number discrimination, word problems, missing number, computation and multiplication. With both tools, fluency rates were calculated for tasks that were timed. Students were given a set of tasks to perform for each of these skills. EGRA and EGMA tools were developed in “test adaptation workshops,” then piloted and revised prior to baseline data collection. Data Collection. Baseline data were collected from September 2 to September 26, 2014. Students were tested on paper formats for the baseline; subsequent data collections will be conducted using tablets. Data were analyzed by SRP staff, and the results were reported by district and language (Sindhi and Urdu). Quality of Assessment Tools

The quality of the EGRA and EGMA tools was measured by examining their internal consistency, item difficulty, and item-total correlations. 5 Based on the results of the pilot, tasks were revised to ensure high internal consistency, appropriate levels of difficulty (allowing sufficient space to capture growth over the life of the project), and adequate item discrimination.

Analyses of test quality showed that the EGRA and EGMA baseline instruments were of high-quality and can be used as a reference for the assessments that will be developed for use at midline and at endline.6

5 Internal consistency (Cronbach’s alpha) indicates whether the results on the selected EGRA and EGMA tasks are a reliable estimate of the expected score for a student on reading and numeracy, respectively; (2) item difficulty (p-values) indicates the proportion of students who were able to answer an item correctly – this statistic is also important to discriminate between the performance of test-takers; and (3) item-total correlations (or item discrimination), indicates the extent to which an item correlates between the question score and the overall assessment and discriminates between stronger and weaker students (students who score high on the test overall should also have answered individual items correctly). 6 As with the baseline instruments, the midline and endline assessments will also need to be piloted to ensure equivalent levels of difficulty to baseline, thus ensuring comparability of measures over time.


OCTOBER 2015 3

Results

This section describes the students’ baseline results for each EGRA and EGMA task in Sukkur district. Three kinds of results are presented: (1) average percent correct; (2) proportion of zero-score students (students who did not get any of the items on the task correct); and (3) comparisons by gender.7

EGRA Results

The EGRA assessment consisted of six tasks for Grade 3 students and eight tasks for Grade 5 students. The tasks for both the Grade 3 and Grade 5 EGRAs were the same, assessed at the Grade 2 level, with the exception of two additional tasks on the Grade 5 EGRA: fluency and reading comprehension using a story at a Grade 4 reading level. 8 Note that results are reported as Grade 3 and 5, though as noted above, these measures represent student reading skills at the end of Grades 2 and 4. Figure 1 shows that at baseline, students in Grade 3 had the highest scores in Vocabulary, followed by Listening Comprehension; the lowest scores were observed in Phonemic Awareness. Furthermore, as shown in Figure 2, 82.2 percent of Grade 3 students had zero-scores in Phonemic Awareness – that is, these students could not correctly identify the phoneme (smallest unit of sound) at the end of the word. (Zero-scores provide critical warning signs to a system, highlighting areas needing significant, targeted instruction.) Across all tasks, the average percent correct was 31.1 percent. Promisingly, the highest scores – in the areas of Vocabulary and Listening Comprehension – are indicators of prior knowledge and precursors to a student’s future success as a reader.

FIGURE 1: GRADE 3 EGRA: PERCENT CORRECT SCORES BY TASK

7 Rural versus urban location was analyzed in districts with sufficient numbers of schools in each group. In most districts, including Jacobabad, the percentage of urban schools was too low (9%) to allow for meaningful comparisons. 8 Phonemic Awareness (Phoneme Isolation task): On this untimed task, students were asked to identify a phoneme – the smallest unit of sound in a word – at the end of 10 words; Phonics (Non-Word Reading or Decoding task): On this timed task, students were presented with 50 non-words (words created for this exercise) and asked to read the word using decoding skills (if readers know how to decode, they can read a word even when it’s not a real word) in one minute; Vocabulary (Expressive Vocabulary task): On this untimed task, students were presented with 10 pictures and asked to identify what they saw; Fluency (Passage Reading task): On this timed task, students were given one minute to read a passage consisting of 60 words- Grade 3 students were tested using a passage at a Grade 2 reading level and Grade 5 students were tested using both Grade 2-level text and Grade 5-level texts; Reading Comprehension (Reading Comprehension task): On this untimed task, students were asked comprehension questions based on the same 60-word passage they read for the fluency task; Listening Comprehension (Listening Comprehension task): On this untimed task, students were first read a story, then asked three questions to assess their understanding of the story’s meaning.

6.8%

17.8%

65.0%

29.2%

14.7%

52.8%

Average: 31.1%

PhonemicAwareness

Phonics Vocabulary Fluency (Grade 2-level text)

ReadingComprehension

(Grade 2-level text)

ListeningComprehension


OCTOBER 2015 4

FIGURE 2: GRADE 3 EGRA: PROPORTION OF ZERO-SCORE STUDENTS BY TASK

As Figure 3 shows, students in Grade 5 scored highest on Vocabulary followed by Listening Comprehension. Across all tasks, the average percent correct was 45.1 percent. The lowest scores were observed in Phonemic Awareness, which also had the highest proportion of zero-scores (Figure 4).

FIGURE 3: GRADE 5 EGRA: PERCENT CORRECT SCORES BY TASK

FIGURE 4: GRADE 5 EGRA: PROPORTION OF ZERO-SCORE STUDENTS BY TASK

Fluency

Fluency is a reader’s ability to read accurately with speed and proper pacing; it is a common measure for determining reading skill. EGRA measures accuracy and speed, called correct words per minute (CWPM),

82.2%

54.3%

4.0%

49.3%

70.6%

14.2%

PhonemicAwareness

Phonics Vocabulary Fluency (Grade 2-level text)

ReadingComprehension



8.7%

35.8%

68.5%

57.6%

35.5%

64.5%58.5%

38.3%Average: 45.1%

PhonemicAwareness

Phonics Vocabulary Fluency (Grade2-level text)

ReadingComprehension(Grade 2-level

text)


Fluency (Grade4-level text)


text)

81.0%

37.6%

2.5%

27.3%

43.7%

6.1%

26.9%

39.8%

PhonemicAwareness

Phonics Vocabulary Fluency (Grade2-level text)


text)


Fluency (Grade4-level text)


text)


OCTOBER 2015 5

or the number of words a reader reads correctly in one minute on a timed reading task. At baseline in Sukkur, Grade 3 students’ averaged 20.4 CWPM and Grade 5 students averaged 49.6 CWPM on the Grade 2-level text; Grade 5 students averaged 52.3 CWPM on the Grade 4-level text. As a reference, the Pakistan Reading Project (PRP) baseline proposed fluency levels (in Urdu) to be considered as provisional standards until national reading standards are officially established. According to these standards, Grade 3 students in Sukkur would be classified in the middle of the “early readers” range and Grade 5 students would be in the middle of the “intermediate” range. Importantly, 49.3 percent of Grade 3 students and 27.3 percent of Grade 5 students were unable to read a single word of Grade 2-level text. With Grade 4-level text, 26.9 percent of Grade 5 students could not read a single word correctly.

Comparison of Scores by Gender: In Sukkur, Grade 3 girls had significantly higher scores, overall and in Fluency, Reading Comprehension and Listening Comprehension tasks. In Grade 5, girls outperformed boys in Vocabulary and in Listening Comprehension tasks only.

EGMA Results

The EGMA assessment consisted of six tasks for Grade 3 students and seven tasks for Grade 5 students. The tasks for both the Grade 3 and Grade 5 EGMAs were the same, assessed at the Grade 4 level; the Grade 5 EGMA included an additional task in multiplication, assessed at the Grade 4 level. 9 As Figure 5 and Figure 6 show, Grade 3 students scored highest on Number Identification, Word Problems and Number Discrimination tasks. Students scored higher on the Addition task compared to the Subtraction; the proportion of zero-scores was highest on the Subtraction task. Grade 3 students scored lowest on the Subtraction task.

FIGURE 5: GRADE 3 EGMA: PERCENT CORRECT SCORES BY TASK

9 Number Identification: On this timed task, students were asked to identify 20 numbers ranging up to 999; Number Discrimination: On this untimed task, students were asked to discriminate between a pair of numbers by identifying the larger number of the two; Missing Number: On this untimed task, students were asked to identify the missing number in a pattern (ranging up to 999); Computation (Addition and Subtraction): On these two timed tasks, students were presented with 20 addition and 20 subtraction problems in one minute for each task; Word Problems: On this untimed task, students were presented with a word problem and were expected to respond with the correct answer based on one or several computations needed to answer the question; Multiplication (Grade 5 only): On this timed task, students in Grade 5 were given a printed list of 20 multiplication problems and were asked to multiply two single-digit numbers in one minute.

62.8%

54.9%

27.1%32.7%

26.2%

52.1%

Average: 42.6%

NumberIdentification

NumberDiscrimination

Missing Number Addition Subtraction Word Problems


OCTOBER 2015 6

FIGURE 6: GRADE 3 EGMA: PROPORTION OF ZERO-SCORE STUDENTS BY TASK

As Figure 7 and Figure 8 show, Grade 5 students scored highest in Number Identification, Number Discrimination and Word Problem tasks, while they scored lowest on the Missing Number and Multiplication tasks. Forty-five percent of Grade 5 students were unable to answer a single question correctly on the Multiplication task, and more than 20 percent were unable to answer a single subtraction question.

FIGURE 7: GRADE 5 EGMA: PERCENT CORRECT SCORES BY TASK

FIGURE 8: GRADE 5 EGMA: PROPORTION OF ZERO-SCORE STUDENTS BY TASK

2.0%

10.4%

26.0%

39.6%

45.1%

6.2%



Missing Number Addition Subtraction Word Problems

81.5%76.1%

37.9%

58.4%

49.8%

69.3%

20.1%

Average: 62.2%



MissingNumber

Addition Subtraction Word Problems Multiplication(Grade 5 only)

0.4%2.6%

14.1%17.1%

21.2%

2.1%

45.7%



Missing Number Addition Subtraction Word Problems Multiplication(Grade 5 only)


OCTOBER 2015 7

Comparison of Scores by Gender: In Grade 3, boys had a higher overall score on the EGMA than girls by 7.8 percentage points. In Grade 5, boys outperformed girls, overall, by 12.8 percentage points. At the task level, boys significantly outperformed girls in all tasks in both grades, except for the Missing Number task.

Questionnaire Results

In addition to the EGRA and EGMA assessments, questionnaires were administered with students, teachers, and head teachers in order to identify factors most frequently associated with strong reading and mathematics outcomes. The following are key findings from the questionnaires and their relationship (if significant) with EGRA and EGMA scores:

Student questionnaire:

Reading materials at home: About thirty percent of students said they had no reading materials at home; Grade 3 students who reported having reading materials at home had higher EGRA scores and Grade 5 students who had reading materials at home generally had higher EGMA scores.

Practice reading at home: A majority of students practiced reading at home either reading aloud to another person, silently to self, or reading the Quran. These reading activities were generally associated with higher EGRA scores for Grade 3 students.

Work with peers: More than two-thirds of students reported completing classwork with peers; Grade 5 students’ EGMA scores were significantly higher than those of students who did not complete classwork with peers.

Absenteeism: More than one-third of students missed days from school the previous week; Grade 3 students’ EGRA scores were significantly lower than those who were not absent.

Teacher and head teacher questionnaires:

Teacher qualifications: Teacher qualifications did not have a significant impact on assessment scores.

Head teacher support: Seventy-six percent of head teachers reported that they monitored students results on tests administered by teachers; this practice was associated with higher EGRA scores in Grade 5.

Timetable implementation: Two thirds of head teachers indicated that their schools followed a timetable; of those who did follow a timetable, most set aside time for teaching reading and math. No significant differences in EGRA or EGMA corresponded to timetable adherence.

School Infrastructure: Facility improvements at schools reported by head teachers included: clean water (51%); electricity (69%); and toilet facilities (53%). These facility improvements were not associated with higher assessment scores. No head teacher reported having a library at their school.

Recommendations

Reading

Grade 3 and Grade 5 students had their highest scores in vocabulary and listening comprehension. These results suggest that students’ oral foundation and background knowledge are areas of relative strength. Students’ fluency rates were in the “early reader” range in Grade 3 and at the lower half of the “intermediate range” in Grade 5. These fluency rates, and the low performance of students in phonemic awareness, suggest that generally, students in Sukkur need continued support to build fluency and struggle most with sound identification of words in isolation.

To this end, following recommendations are proposed in order to focus SRP interventions on areas of need for students, teachers, and head teachers:


OCTOBER 2015 8

1. Recommendations to strengthen reading instruction:

Capitalize on students’ relative strengths in the areas of Vocabulary and Listening Comprehension throughout reading instruction. For example, book selection should incorporate topics where students have prior background, where possible.

Ongoing formative assessment should guide instructional decision-making for the teacher to ensure that students’ instructional skills are identified and targeted for instruction.

Small group instruction is provided to students via reading groups that have students together of a similar reading level. Leveled books at the student’s “instructional level” are used in small group reading instruction. “Instruction level” is with text where the student can read between 90-95% of the words with automaticity.

Students should be provided with adequate time each day to practice independent reading skills, regardless of their reading level. Independent reading practice should be with text where the student can read at least 95% of the words with automaticity, have prior knowledge of the topic, and is of high interest to the student. Students of similar reading levels can pair up and read to each other aloud, to practice reading text at their “independent level.” Students can also read independently to themselves to practice reading. Students’ reading endurance is to be increased as they development, starting with 5-10 minutes for younger grades and 20 or more minutes for older grades.

To strengthen reading Fluency, Reading Comprehension, and Vocabulary development, daily read alouds should be incorporated by the classroom teacher. Book selection for read alouds should be at a reading level higher than what the students can read independently. This allows for exposure to different genres, content vocabulary, and builds prior knowledge in students.

Ensure phonemic awareness is taught regularly and explicitly, particularly in Grade 1, as it will provide a foundation for later phonics instruction.

Daily, explicit, structured phonics instruction should be taught in an engaging, multisensory, authentic manner. Adequate practice is incorporated, such that mastery of each decoding skill is attained prior to moving on to the next skill. Phonics skills include asking students to identify the relationship between letters and sounds. Activities may include writing new words and word parts on the blackboard and asking students to decode them.

Incorporate reading practice to enhance reading fluency with a “repeated reading” strategy, whereas a student re-reads a passage more than one time. This enhances comprehension and allows the student to experience reading with enhanced fluency. This strategy can also be used as “paired reading,” where two students of similar reading levels are paired and engage in “repeated reading” with each other.

Incorporate regular and explicit practice with “sight words” to enhance reading automaticity. Sight words are high frequency words that a student can read by sight” (i.e., not by sounding out). Using a “Drill Sandwich” strategy to learn sight words is an effective method for mastery of sight words, whereas adequate challenge is incorporated into the learning set. Typically, a learning set where 70% of the words are mastered and 30% of the words are new is considered to be adequate challenge. If this proves too difficult, the learning set can be made easier (e.g., 90% mastered and 10% unknown).

Provide teachers with explicit strategies for strengthening listening and reading comprehension skills such as pre-reading (asking questions before reading a story, teaching vocabulary, discussing the subject of the story before reading), read-alouds (the teacher reads a story and asks questions along the way to draw students’ attention to important details, query their comprehension, or ask them to predict) and post-reading (such as group, pair, and independent reading, drawing pictures, continuing or dramatizing the story) to bolster comprehension and reading skills, simultaneously.

2. Recommendations to strengthen school support for reading:

Develop a support system for reading teachers so they can share best practices and promising practices across and between colleagues for reading instruction.


OCTOBER 2015 9

Provide high quality, targeted professional learning for teachers regarding the implementation of best practices in reading instruction. Included in this professional learning is a framework to use formative assessment to guide instruction.

Expand time spent reading in and out of class. For example, designate class time for reading each day, ensure that students have access to reading materials at school and at home, develop highly engaging reading events such as a community “reading marathon,” and use volunteers or “cross-age buddies” to build language skills.

Create a library within each classroom so that students have ready access to high interest reading materials.

Incorporate the premise that the learning of all students is enhanced when there is adequate challenge (“not too easy, not too hard”). Recognize that behavioral concerns will most often be displayed when there is not an instructional match for the student (the work being given to the student is either too easy or too difficult). Providing teachers with professional learning in “instructional assessment” is key to student success. “Instructional assessment” incorporates the notion that instruction and materials are matched to the student’s individual skills, prior knowledge, and interests.

3. Recommendations for further research or actions by SRP staff:

Establish standards for reading fluency in the Sindhi language.

Identify critical areas of intervention to focus on schools and/or sub-groups where large proportions of children are being left behind (i.e., those with zero-scores). Targeted interventions may include pullout of non- and struggling readers for intensive teaching of core competencies/foundation skills, early enrichment activities in formal educational care or via informal mother training to help develop pre-reading foundational skills, implementation of procedures for diagnosing reading difficulties, remediation, and the use of formative assessment to ensure students who lack foundational reading skills are identified early.

Mathematics

1. Recommendations to strengthen mathematics instruction:

Provide teachers with strategies for building number sense and computational approaches, which would enable students to identify numbers and patterns and build computational skills (addition, subtraction, and multiplication).

Provide teachers with strategies for diagnosing students experiencing difficulties in number sense and computation, and formative assessment to ensure students who lack foundational mathematics skills are identified early.

Examine trends by sub-groups and geography to determine root causes for the poor performance among girls in both Grades 3 and 5 relative to that of boys. Use these findings to develop targeted intervention for girls.



OCTOBER 2015 10

CHAPTER 1: INTRODUCTION

1.1 The Sindh Reading Program (SRP)

Under the umbrella of the Sindh Basic Education Project (SBEP), the Sindh Reading Program (SRP) is a five-year collaborative initiative of the Government of Sindh (GoS) and USAID/Pakistan (funded by USAID). SRP is designed to improve student performance in early grade reading and mathematics for 410,400 children across eight districts in the province of Sindh in Pakistan.10 SRP’s main objectives, when the project was initiated, were to:

1. Improve teacher competencies for effective early grade reading and mathematics instruction;

2. Improve early grade reading and mathematics assessment practices;

3. Improve student access to supplementary reading and mathematics materials; and

4. Enhance participation of parents and caregivers in support of early grade literacy and

mathematics for out of school children.

SRP is being implemented across eight districts in the province of Sindh in Pakistan: Dadu, Jacobabad, Kashmore, Khairpur, Larkana, Kambar-Shahdadkot, Sukkur, and five towns in Karachi (which are treated as a single district for the purposes of this report). By 2018 (year 5 of the program), one of SRP’s key indicators is to improve reading for at least 200,887 Grade 2 students.11 Through the end of the project, the Education and Literacy Department of the Government of Sindh (GoS), USAID, and SRP have and will collaborate to implement multiple strategies to improve teaching practices and learning performance in early grade literacy with the ultimate goal of improving reading skills among students in participating schools. SRP’s strategies include:

Design and implement a baseline study of student performance in early grade reading and mathematics;

Analyze current performance levels to target project interventions;

Define “grade level” competencies through the development of performance standards;

Implement follow-up studies to monitor project progress towards established goals; and

Institutionalize early grade reading and mathematics assessment practices at the government and school levels.

This report focuses on the first strategy, the design and implementation of the SRP baseline study for one

of the eight target districts: Sukkur. In its original design, along with a focus on improving reading skills

among students in the early grades, SRP also included interventions to improve students’ mathematics

skills. This design was subsequently revised to retain only the core indicator of improving reading skills in

the early grades; however, this revision came after baseline data had already been collected. As such,

10 The focus on mathematics has since been removed from the evaluation and program design; this shift, however, occurred after the baseline data were collected and is discussed below. 11 The target reflects revisions as of April 2015.


OCTOBER 2015 11

baseline data on both students’ literacy and mathematics were collected and analyzed; at midline and

endline, data on literacy only will be collected.

Baseline data on students’ reading and mathematics skills were collected, and the results analyzed and

reported to 1) provide a starting point for measuring progress on improved reading to determine changes

in the reading skills of students participating in the SRP interventions; and 2) to inform the

implementation of SRP interventions for the remaining years of the project. Furthermore, the baseline

study also served as a reference point for establishing the student assessment tools for midline and

endline data collections. The tool used to assess students’ reading skills is called the Early Grade Reading

Assessment (EGRA) and the tool used to assess students’ mathematics skills is called the Early Grade

Mathematics Assessment (EGMA).

In addition to student assessment data, the evaluation also sought to capture contextual variables that may

be associated with strong reading and mathematics outcomes, and thus included a Questionnaire for

students taking the EGRA and EGMA tests, and one for their teachers and head teachers. Student

questions related to language spoken at home, availability of reading materials at home, study habits,

access to communication technology and vehicles at home. Teacher and head teacher questions related

to qualifications, attendance at trainings, scheduling, teacher support, head teacher knowledge of student

performance, teacher observation procedures, School Management Council (SMC) authority and facility

improvements.

The evaluation baseline was conducted in September 2014, prior to the launch of SRP interventions. In accordance with USAID program evaluation guidelines, samples of students at two grade levels – Grade 3 and Grade 5 – were assessed throughout Sindh province so that independent baselines can be established in each district. It should be noted that the results from this baseline are considered a measure of learning at the end of Grades 2 and 4, even though the baseline was conducted for Grades 3 and 5 students. The decision to conduct the baseline with this population was made because of requirement to conduct the baseline before the startup of project activities in Year 1. This timing presents a complication – the effects of “summer learning loss” – i.e., that students typically forget some of what they have learned in a given school year over the summer. The extent of this loss cannot be estimated and the results presented here may in fact be underestimating the actual performance of students prior to the break. In other words, assessments conducted after the summer break will reflect a short-term decline in student achievement. Note that SRP midline and endline assessments will be conducted at the same point in the school year in order to ensure comparability of results across time points. These three time points (baseline, midline and endline), when compared, provide a measure of the growth made by students in reading in schools participating in SRP interventions. 12 As part of the SRP baseline study, the following activities were carried out for each of the districts. Below and in Chapter 2, each activity is described in greater detail.

1. Evaluation Design. A cross-sectional design was followed in accordance with USAID guidelines.

2. Sampling. A stratified cluster random sampling method was used in order to be able to generalize the results to the district level.

3. Instrument development. EGRA/EGMA tools were developed for the Sindh province and piloted.

12 The evaluation utilizes a non-experimental pre-test/post-test design, in which students participating in the intervention are assessed prior to the start of the interventions (pre-test) and following the conclusion of the interventions (post-test); additionally, a mid-point measure is also collected. This design was appropriate for this study since an adequate comparison group (i.e., students in schools not participating in SRP interventions, but similar otherwise) was not feasible. A comparison group is necessary in order to establish impact of the interventions (in other words, to be able to attribute all the changes from pre-test to post-test among participating students to the SRP interventions). Furthermore, adjustments made to SRP’s focus, overall, following the collection of baseline data, dictates that only reading will be assessed at midline and endline.


OCTOBER 2015 12

4. Training. Workshops were conducted to train Master Trainers (MTs), supervisors, enumerators, and Quality Control Officers (QCOs). Enumerators and supervisors were observed to ensure clear understanding and skills adequate to implement the EGRA and EGMA tools.

5. Implementation. The survey was implemented in all targeted schools, followed by data entry and preparation of a clean data file.

6. Analysis. Two types of analyses were conducted: 1) test quality and 2) analysis of baseline data to report findings by subgroup, proportion of students with zero-scores and average percent correct by task.

7. Reporting. One summary report was produced for each of the two language groups – Sindhi and Urdu; one district-level report was produced for each of the seven districts, including this report for Sukkur; and one report was produced for the five towns of Karachi (treated as a district for the purposes of the evaluation).

1.2 Evaluation Design

USAID required a cross-sectional design, assessing students at the same grade levels (including equal representation by gender) over the course of SRP in both reading and mathematics skills. As shown in Figure 9, the design features two grade levels (3 and 5) and three time points (baseline, midline, and endline). The intent of the evaluation is to determine gains in reading across three time-points (2014, 2016 and 2018) among students in SRP schools using results from students in Grades 3 and 5 as indicative of student performance at the ends of Grade 2 and 4. All students were assessed using the same instruments that measured student’s ability at the Grade 2 level. Grade 5 students were assessed on two additional tasks measured at the Grade 4-level (therefore, Grade 5 students had two reading tasks that were each assessed at both the Grade 2 and Grade 4 level). Because students were selected at random, results of this baseline can be generalized to all Grade 2 and Grade 4 students in schools in which students speak that language (Urdu in Karachi and Sindhi outside of Karachi) in each sampled district.

FIGURE 9: EVALUATION DESIGN

Note: Results from Grade 3 students are indicative of results from Grade 2 students at the end of the year; similarly, results from Grade 5 students are indicative of results from Grade 4 students at the end of the year.

1.3 What are EGRA and EGMA?

The Early Grade Reading and Mathematics Assessments (EGRA and EGMA) were developed with funding from USAID and the World Bank. These assessments measure student performance on the most basic foundational skills required for fluency in reading and mathematics. Policymakers have used the results of EGRA and EGMA to develop reforms to improve reading and mathematics performance of students in the early grades. Moreover, these data have been used by educationists to establish student performance standards so that teachers understand expectations for students at each stage of reading and mathematics development and can tailor instruction appropriately. EGRA and EGMA are individually administered, oral assessments that require approximately 35 minutes per student. SRP developed the EGRA and EGMA tools used for the baseline by adapting models from other countries to the Sindh context. Adaptations are necessary in order to ensure cultural appropriateness and alignment with SRP interventions. In Karachi, the assessments were conducted in Urdu and to students

Grade 3

Grade 5

BASELINE

2014 (Y1)

MIDLINE

2016 (Y3)

ENDLINE

2018 (Y5)

Grade 3

Grade 5

Grade 3

Grade 5


OCTOBER 2015 13

in Urdu-language schools, while students in Sindhi-language schools in the remaining seven districts were assessed in Sindhi. Detailed descriptions of each task are provided in Section 2.3.

Chapter 2 details the methodology for the SRP baseline study. EGRA results are presented in Chapter 3. And EGMA results are presented in Chapter 4. Finally, relationships between contextual variables (student and teacher questionnaire results) and EGRA and EGMA performance are presented in Chapter 5. Additional supporting materials are provided in the annexes at the end of this report.


OCTOBER 2015 14

CHAPTER 2: METHODOLOGY

This chapter describes the SRP sampling process, development of the EGRA and EGMA assessment tools and piloting, data collection, data entry, data analysis and results supporting the quality of the assessment tools.13 The same procedures were used in all districts.

2.1 SRP Sampling Design

A stratified cluster random sampling method was used in order to meet two SRP requirements: (1) that equal numbers of girls and boys be assessed, and (2) that results be generalizable to the population of students in the eight districts. In seven of the SRP intervention districts (Dadu, Jacobabad, Kashmore, Khairpur, Larkana, Kambar-Shahdadkot and Sukkur), 70 schools were randomly selected from the total pool of Sindhi-speaking schools in those districts. Similarly, 70 schools were selected from the total pool of Urdu-speaking schools in the five towns of Karachi City being served by SRP (for the purposes of this report, the five towns of Karachi City were treated as a district). The following is a description of the procedure used for the selection of schools and students in each of the seven districts, including Sukkur. Data from the 2013 Sindh Education Management Information System (SEMIS) were used as a basis for sampling.

1. All schools within the district were sorted by their location: urban or rural.

2. Schools were deselected if they did not meet the minimum requirement of at least 12 male students or 12 female students for Grades 3 and 5.

3. Thirty-five boys’ schools and 35 girls’ schools were randomly sampled, proportionately according to urban/rural ratios.

4. Ten additional schools were selected using the same procedure (stratified by location and gender) in case of inaccessibility or inaccurate SEMIS data. Note that mixed-gender schools were sometimes selected as replacement schools because an insufficient number of single-gender replacement schools were available. When a mixed-gender school was selected, only students of the respective gender were included in those samples.

5. Within each school, 12 students were randomly selected in Grade 3 and 12 students in Grade 5.

6. Teams of QCOs (Quality Control Officers) and SRP project staff visited the districts to verify the preliminary sample of schools, and replacements were made wherever the actual number of students was less than the target number of students. In some instances, schools were retained in the final sample if the number of students at each grade level was close to 12 and if the replacement schools did not have a larger number of students.

2.2 Sukkur School and Student Sample

The EGRA and EGMA assessments were administered to 12,793 students in 560 schools across eight districts of Sindh Province that were participating in SRP. The total number of schools that were sampled in Sukkur is shown inSince only 9 percent of schools were located in urban areas, comparing EGRA and EGMA scores by location (urban versus rural) schools was not advised because of the unequal sampling. Table 1. Of the 1,062 schools where Sindhi was the primary language of instruction, 966 schools were in rural areas and 96 in urban areas. An equal number of boys and girls schools was sampled. Since only 9 percent of schools were located in urban areas, comparing EGRA and EGMA scores by location (urban versus rural) schools was not advised because of the unequal sampling.

13 For more details on the tool development process and implementation of testing, see Chemonics (2014) “Process Report for EGRA/EGMA Baseline Study 2014.”


OCTOBER 2015 15

TABLE 1: BASELINE SCHOOLS IN SUKKUR

Location Number of

Schools (n)

Proportion of Sample

(%)

Baseline Schools

(n)

Boys Girls

Rural 966 91% 32 32

Urban 96 9% 3 3

Total 1062 100 35 35

The baseline sample consisted of 70 schools from each district and within each school, 12 students from Grade 3 and 12 students from Grade 5 were assessed, for a total of 13,440 students across all eight districts. In Sukkur, the assessment tools (EGRA, EGMA and student questionnaire) were administered to students in 70 schools for a total sample of 1,583 students (see Table 2). This final dataset for Sukkur represented 94 percent of the total target sample. Additionally, teacher questionnaires were administered to 65 Grade 3 and 65 Grade 5 teachers (representing 93 percent of the target sample), and to 70 head teachers (100 percent of the target sample). The number of teachers on whom data were collected was slightly lower than the target sample because some teachers taught both third and fifth grade.

TABLE 2: BASELINE PARTICIPANTS IN SUKKUR

Grade 3 Grade 5 Total

Boys 499 473 927

Girls 311 297 608

Missing Gender Code 0 3 3

Total Students 810 773 1583

Male Teachers 24 22 46

Female Teachers 41 43 84

Total Teachers 65 65 130

Male Head Teachers 65 65 43

Female Head Teachers -- -- 24

Missing Gender Code 3

Total Head Teachers -- -- 70

2.3 EGRA/EGMA Tool Development

In Sindhi-speaking districts (including Sukkur), EGRA and EGMA were administered in Sindhi; in Urdu-speaking districts, EGRA and EGMA were administered in Urdu. The development of both these EGRA and EGMA tools was a multi-step process that included:

1. The choice of tasks to be included in the tools (the competencies to be assessed); 2. The development of the content for each of these tasks (the items); 3. The piloting of the tools in local schools; 4. The analysis of data from the pilot and the determination of changes required to improve the

tools; and 5. The finalization of the tools.

Note: Students with missing gender codes are excluded from gender-based comparisons but are included in all other analyses.


OCTOBER 2015 16

Selection of EGRA/EGMA Tasks

The baseline EGRA and EGMA tools were developed by SRP to measure core reading and mathematics skills targeted by SRP in its teacher training, instructional materials, and parental involvement interventions. Six types of reading skills were tested with the EGRA for students in Grade 3: phonemic awareness, phonics, vocabulary, fluency, reading comprehension and listening comprehension. The same tasks were used for students in Grade 3 and Grade 5 (using text at a Grade 2 reading level); two additional Fluency and Reading Comprehension tasks (6a and 6b) were used with students in Grade 5 using text at a Grade 4 reading level. Some of these tasks were timed in order to determine the speed with which students were able to decode words (phonics) and read connected text (fluency). A summary of these tasks and the skills they were designed to assess are presented in Table 3 followed by a brief description of each task.

TABLE 3: EGRA TEST STRUCTURE

Core Reading Skills SRP EGRA Tasks (Task Number and Task Name)

Task Length

1. Phonemic Awareness 1. Phoneme Isolation Untimed

2. Phonics 2. Non-word Reading (Decoding) Timed (1 minute)

3. Vocabulary 3. Expressive Vocabulary Untimed

4. Fluency (Grade 2-level text) 4a. Passage Reading Timed (1 minute)

5. Reading Comprehension (Grade 2-level text)

4b. Reading Comprehension Untimed

6. Listening comprehension 5. Listening Comprehension Untimed

7. Fluency (Grade 4-level text) 6a. Passage Reading (Grade 5 only) Timed (1 minute)


6b. Reading Comprehension (Grade 5 only) Untimed

Phonemic Awareness (Phoneme Isolation task): On this untimed task, students were asked to identify a phoneme – the smallest unit of sound in a word – at the end of 10 words. Phonics (Non-Word Reading or Decoding task): On this timed task, students were presented with 50 non-words (words created for this exercise) and asked to read the word using decoding skills (if readers know how to decode, they can read a word even when it is not a real word) in one minute. Vocabulary (Expressive Vocabulary task): On this untimed task, students were presented with 10 pictures and asked to identify what they saw. Fluency (Passage Reading task): On this timed task, students were given one minute to read a passage consisting of 60 words- Grade 3 students were tested using a passage at a Grade 2 reading level and Grade 5 students were tested using both Grade 2-level text and Grade 5-level texts. Reading Comprehension (Reading Comprehension task): On this untimed task, students were asked comprehension questions based on the same 60-word passage they read for the fluency task. Listening Comprehension (Listening Comprehension task): On this untimed task, students were first read a story, then asked three questions to assess their understanding of the story’s meaning. Six types of mathematics skills were tested with the EGMA for students in Grade 3: number identification, number discrimination, word problems, missing number, addition and subtraction. The same tasks were used for students in Grade 3 and 5; one additional multiplication task (Task 7) was used for students in Grade 5. Again, some of these tasks were timed in order to determine the speed with


OCTOBER 2015 17

which students were able to identify numbers and perform basic mathematical procedures. A summary of these tasks is presented in Table 4 followed by a brief description of each task.

TABLE 4: EGMA TEST STRUCTURE

Core Mathematics Skills SRP EGMA Tasks Task Length

1. Number Identification 1. Number identification up to 999 Timed (1 minute)

2. Number Discrimination 2. Number discrimination up to 999

Untimed

3. Missing Number 3. Missing Numbers up to 999 Untimed

4. Addition 4. Addition within 10 Timed (1 minute)

5. Subtraction 5. Subtraction within 10 Timed

6. Word Problems 6. Word Problems Untimed

7. Simple Multiplication (Grade 5 only)

7. Multiplication up to 2 digits Timed (1 minute)

Number Identification: On this timed task, students were asked to identify 20 numbers ranging up to 999. Number Discrimination: On this untimed task, students were asked to discriminate between a pair of numbers by identifying the larger number of the two. Missing Number: On this untimed task, students were asked to identify the missing number in a pattern (ranging up to 999). Computation (Addition and Subtraction): On these two timed tasks, students were presented with 20 addition and 20 subtraction problems using numbers up to 10; students were given one minute for each task. Word Problems: On this untimed task, students were presented with a word problem and were expected to respond with the correct answer based on one or several computations needed to answer the question. Multiplication (Grade 5 only): On this timed task, students in Grade 5 were given a printed list of 20 multiplication problems and were asked to multiply two single-digit numbers in one minute.

EGRA/EGMA Tool Revision

The SRP team conducted a six-day workshop from April 14-19, 2014 at the Bureau of Curriculum (BoC) in Jamshoro to draft the tasks for the EGRA and EGMA tools in both Sindhi and in Urdu. This workshop served as an introduction to EGRA and EGMA and the first capacity building activity for GoS, BoC, Provincial Institute of Teacher Education (PITE), District Examination Committee and Sindh Textbook Board (STBB) officials. A total of 19 participants, 14 men and 5 women, including 16 subject experts (four language and reading experts in Urdu and five in Sindhi, and seven mathematics education experts), two typists and one illustrator attended the workshop. Participants reviewed EGRA and EGMA models used in other countries, examined the proposed tasks against the national curriculum,14 determined the appropriate content- and difficulty- levels for end of Grade 2 and end of Grade 4 and drafted the content for all the assessment tasks. They also developed questionnaires for students, teachers, and head teachers in Sindhi and Urdu to capture contextual information that may be related to EGRA and EGMA outcomes.

14 Though EGRA and EGMA are not curriculum based, participants used the national curriculum in order to ensure that the content and level of difficulty of the EGRA and EGMA tasks were appropriate for these grade levels.


OCTOBER 2015 18

Piloting of EGRA/EGMA Tools

In Hyderabad, the SRP team piloted the EGRA and EGMA tools to determine their validity and reliability before using them for the operational baseline study.15 The pilot was conducted with students in Grades 3 and 5 in May 2014 in 24 randomly selected schools of which half were Sindhi-language and half were Urdu-language. The EGRA and EGMA were administered to students in a one-to-one format using paper-based tools. A total of 40 students were targeted in each school (20 students in Grade 3 and 20 students in Grade 5). Four teams composed of four enumerators each

and one QCO comprised a team that visited each school; in all, the teams assessed 406 students with the Sindhi language tools and 413 with the Urdu language tools. Subject experts were invited to observe the pilot so that they could gain practical insights about the challenges faced by students with the content of the tasks.

Tool Finalization

Following the pilot, data were entered by one of SRP’s partners (VTT) and analyzed in order to calculate statistics to establish the quality of the assessment tools (results are presented at the end of this chapter). To review student results and quality of the assessment tools and to make necessary adjustments to the tools, SRP convened a workshop from July 7-12, 2014 in Hyderabad. For the EGRA, one change that was reviewed was the level of difficulty of the phonemic awareness task for students in Grades 3 and 5 in both languages; a revision seemed necessary based on the relatively low scores observed in the pilot.16 As a result of reviewing the pilot results, the team decided to rewrite the task to focus on the sounds at the end of words, rather than the middle sounds. For the EGMA, the assessment, overall was found to be too long (in the pilot, this assessment lasted, on average, between 50 and 60 minutes, per student). As a result, the team decided to eliminate three tasks, namely, Addition and Subtraction at Level Two and Division.

2.4 Data Collection

Preparation for the baseline data collection in September commenced in June 2014. A timeline was developed to ensure the timely procurement of services and materials, the training of assessment personnel, and the execution of the data collection. To adequately prepare for the baseline data collection, SRP had to solicit for and select subcontractors to oversee data collection, entry and cleaning in each of the eight districts; enumerator training had to be conducted (and final enumerators selected) and data entry and cleaning procedures developed. Each of these tasks are described below.

Subcontractor Selection

In July, SRP issued a Request for Proposals to solicit services from firms to manage data collection, data entry, and cleaning. Nine firms submitted proposals. A committee composed of four senior SRP technical staff evaluated each proposal using predefined criteria. Three firms were selected to manage the recruitment of enumerators, to provide administrative support, and to supervise data collection in each district. The three firms and their areas of management were:

o VTT (Voice Tel Tech): Karachi o MDC: Dadu, Larkana, Khairpur, and Kambar-Shahdadkot o I-SAPS: Jacobabad, Kashmore, and Sukkur

15 Hyderabad was selected because of the availability of both Urdu-language and Sindhi-language schools. 16 The piloted phonemic awareness task resulted in 91.4 percent of Grade 3 students and 82.2 percent of Grade 5 students receiving zero-scores (the term “zero-scores” indicates that a student was unable to answer a single question correctly). With such a large proportion of zero-scores, there is insufficient data to adequately measure the quality of the items.


OCTOBER 2015 19

Training of Master Trainers, QCOs, and Enumerators

Master Trainers and QCOs A 12-day training was organized from August 11-22, 2014 in Karachi for the training of Master Trainers (MTs) and Quality Control Officers (QCOs). The purpose of the MTs and QCOs is to monitor data collection and to ensure that data collection methods are consistent across all enumerators. To ensure that MTs and QCOs were adequately prepared to supervise data collection in the field, SRP made an effort to first recruit candidates with experience in EGRA, or at the very least with experience in educational assessments, evaluation and/or training. From the candidates that attended the training, trainers selected 8 MTs and 8 MTs/QCOs based on the following criteria: 1) they demonstrated adequate understanding of the tools and could administer the assessments with minimal support required; 2) they had high Inter-Rater Reliability scores, and 3) they were available to monitor data collection activities during baseline. These sixteen individuals served as MTs and/or QCOs in teams of two in each of the districts; they supported field work for the teams of enumerators deputed to each district. When team assignments were made, each team consisted of at least one member MT and QCO who had participated in the pilot in May. SRP staff trained the MTs and QCOs who, in turn, led the training sessions for all enumerators and supervisors. Enumerators In each district, enumerators were recruited and selected using a similar approach as that described above for MTs and QCOs. Table 5 outlines the details of the enumerators training, by district.

TABLE 5: ENUMERATORS TRAINING BY DISTRICT

District Enumerator Training Dates

Criteria for Selection Number of Enumerators

Dadu

Aug 24 to 29 Graduation with 2 years’ experience conducting surveys in education

20

Larkana Aug 31 to Sep 5 Same as above 20

Khairpur Aug 31 to Sep 5 Same as above 20

Kambar-Shahdadkot

Aug 24 to 29 Same as above 20

Jacobabad Aug 24 to 29 Same as above 20

Kashmore Aug 31 to Sep 5 Same as above 20

Sukkur Aug 31 to Sep 5 Same as above 20

Data Collection

Data collection began on September 2, 2014 in Dadu, Jacobabad, Kambar-Shahdadkot and Karachi; and on September 9, 2014 in Kashmore, Khairpur, Larkana and Sukkur. All data collection was completed by September 26, 2014 with approximately 95 percent of the target number of students assessed in Grades 3 and 5. To ensure quality of data collection, three supervisors as well as two QCOs monitored data collection and provided feedback to the enumerators during data collection in each district. At the end of each day, enumerators, MTs and QCOs discussed progress and problems encountered that


OCTOBER 2015 20

day. Assessment forms and questionnaires were checked for possible errors or omissions prior to submission for data entry. Despite several challenges, including low student attendance, security risks and limited accessibility to sampled schools, each team visited one school per day and were able to reach most schools as planned; where necessary, replacement schools were used.

Data Entry and Cleaning

In May, the EGRA team developed a customized data entry application so that the data from the booklets and questionnaires could be entered into computers that would then be saved on a networked server. In September, the team trained the Data Coordinator, supervisors, and data entry operators. Following USAID requirements, all student data was entered using the double data entry approach to minimize errors; any discrepancies between the first and second entries were reconciled. A clean data file was then provided to the data analysis team.

2.5 Data Analysis

In January 2015, EGRA statisticians and a psychometrician developed a research plan for the student and teacher data. This plan included the calculation of statistics required to establish the quality of the student assessment tools (reliability as measured by Cronbach’s alpha, item difficulty, item-total correlations), as well as summary scores, by task, for student groups (overall and by gender). Teacher and head teacher questionnaire data were also summarized and their relationship to student assessment results were analyzed. Descriptive and inferential statistics were also generated by task, grade level and gender to summarize results, measure significance of differences between groups, and determine the strength of the relationship between EGRA and EGMA scores and contextual variables (from student, teacher, and head teacher questionnaires). In some cases, double analyses were conducted to confirm the accuracy of the findings. Raw data were cleaned in Excel and SPSS statistical software. All analyses were conducted in SPSS including the production of means tables and significance tests. Final graphs (those presented in this report) were generated using Word and Excel.

2.6 Quality of Assessment Tools

Overall, the quality of the EGRA and EGMA assessments in measuring reading and mathematics skills among students in Sukkur was found to be very strong, as suggested by three statistics: (1) internal consistency (Cronbach’s alpha), which indicates whether the results on the selected EGRA and EGMA tasks are a reliable estimate of the expected score for a student on reading and mathematics, respectively; (2) item difficulty (p-values) which indicates the proportion of students who were able to answer an item correctly – this statistic is also important to discriminate between the performance of test-takers; and (3) item-total correlations (or item discrimination) which indicates the extent to which an item correlates between the question score and overall assessment and discriminates between stronger and weaker students (students who score high on the test overall should also have answered individual items correctly).

Reliability Cronbach’s alpha is a measure of internal consistency which indicates the extent to which the selected tasks can be generalized to all tasks measuring reading (and not only those selected in this version of the tool). Common guidelines for student assessments (such as the EGRA and EGMA) provide a threshold value of 0.75 to establish the reliability of the assessment. Table 6 provides the reliability estimates for the EGRA and EGMA assessments in Sukkur for Grades 3 and 5. In both grades and content areas, the reliability estimates are greater than the threshold value of 0.75 and therefore render the tools as reliable measures of the student’s reading and mathematics skills. These levels are evidence of the tests’ high levels of internal consistency, and indicate that the results on the selected EGRA and EGMA tasks are a reliable estimate of the expected score for a student on reading and mathematics, respectively.


OCTOBER 2015 21

TABLE 6: RELIABILITY ESTIMATES

Assessment Grade Level Tasks Number Cronbach alpha

EGRA Grade 3 6 810 0.78

Grade 5 8 773 0.89

EGMA Grade 3 6 810 0.86

Grade 5 7 773 0.87

Difficulty Another measure of test quality is the use of p-values, which is the proportion (or percentage) of students that got an item correct. Generally, p-values in the range of 0.1 to 0.9 are considered acceptable (and indicate that the item can be retained). For both the EGRA and EGMA assessments in Sukkur, p-values providing a spread on the lower half of the difficulty spectrum. The difficulty values for Grade 3 EGRA ranged from 0.1 to 0.7 for EGRA and 0.1 and 0.86 for EGMA. All items had high item-total correlations for both grades indicating high quality items. The proportion of students answering an item correctly was slightly higher for Grade 5 in Sukkur ranging from 0.1 to 0.9 for EGRA and 0.1 to 0.9 for EGMA. Discrimination A third measure of test quality is the extent to which each task is able to discriminate between strong and weak students (based on their overall performance on the test). This is determined by measuring the strength of the correlation between each task and the overall test (i.e., this metric is called the item-total correlation, but for the EGRA and EGMA, correlations were calculated between tasks and overall score). For item-total correlations, a generally acceptable threshold on assessments like EGRA and EGMA is 0.20 and above. All EGRA and EGMA tasks for both Grades 3 and 5 had item-total correlations greater than 0.20, indicating very good quality for these tasks. Detailed results of test quality measures are reported in Annex 1.


OCTOBER 2015 22

CHAPTER 3: EGRA RESULTS – SUKKUR DISTRICT

This section reports the results for EGRA for students in Sukkur district. The EGRA assessment consisted of six tasks for Grade 3 students and eight tasks for Grade 5 students. First, overall results (percent correct), by grade and across all tasks, are presented as a distribution of students. Second, results are disaggregated by task and gender with results summarized by percent correct and fluency (for timed tasks). Third, the proportion of students who were unable to answer a single question on the task (or zero-scores) are reported. Fourth, results from other Sindhi-language SRP districts are presented for providing context for interpreting results for students in Sukkur. The section concludes with recommendations for supporting students in Sukkur to improve their reading. Additional details on score calculations, including percent correct, timed task scores (fluency) and task score ranges and calculations are provided in Annex 2. EGRA item-level statistics, by task and grade, are provided in Annex 3.

3.1 Overall EGRA Results

The graphs below show the number of students by the percent of total items correct on the EGRA. For example, the tallest bar in the first graph (Grade 3) shows that approximately 70 students had correctly answered 20 percent of the items correctly. This graph (or histogram) provides a snapshot of the number of students at each level of performance, as well as a profile of the group as a whole. The black trendline summarizes the overall pattern of percent correct for the group based on a calculation of averages over the entire group.

As shown in the first graph in Figure 10: Distribution of EGRA Scores, the trendline for Grade 3 peaks to the left of center, indicating a fairly weak performance by most students; on average, Grade 3 students correctly answered 31.1 percent of the items. The trend line for Grade 5 also peaks slightly to the left of center, indicating a slightly lower than average performance by the Grade 5 students on the EGRA assessment. On average, Grade 5 students correctly answered 45.1 percent of the items (second graph in Figure 10).

FIGURE 10: DISTRIBUTION OF EGRA SCORES

Grade 3 Grade 5

3.2 EGRA Results by Task

While the overall results are useful to determine how students in Grades 3 and 5 performed across all EGRA tasks, the remainder of this section focuses on EGRA results by task. Table 7 shows the average


OCTOBER 2015 23

percent correct by task (hereafter, percent correct scores are simply referred to as ‘scores’).17 The average score for students in Grade 3 in Sukkur ranged from 6.8 percent correct to 65.0 percent correct. In Grade 5, the average score ranged from 8.7 percent correct to 64.5 percent correct.

TABLE 7: EGRA PERCENT CORRECT SCORES BY GRADE AND TASK

Core Reading Skills SRP EGRA Task (Number and Task Name)

Grade 3

(% correct)

Grade 5 (% correct)

1. Phonemic Awareness 1. Phoneme Isolation 6.8% 8.7%

2. Phonics 2. Non-word Reading (Decoding) 17.8% 35.8%

3. Vocabulary 3. Expressive Vocabulary 65.0% 68.5%

4. Fluency (Grade 2-level text) 4a. Passage Reading 29.2% 57.6%

5. Reading Comprehension Grade 2-level text)

4b. Reading Comprehension 14.7% 35.5%

6. Listening comprehension 5. Listening Comprehension 52.8% 64.5%

7. Fluency (Grade 4-level text) 6a. Passage reading -- 58.5%


6b. Reading comprehension -- 38.3%

Summary Score (all tasks) 31.1% 45.1%

Fluency

Fluency is a reader’s ability to read accurately with speed and proper pacing; it is a common measure for determining reading skill. EGRA measures accuracy and speed, called correct words per minute (CWPM), or the number of words a reader reads correctly in one minute on a timed reading task. Fluency rates were calculated for three EGRA tasks: non-word reading, passage reading Grade 2-level text and passage reading Grade 4-level text. The average fluency rates in Correct Words per Minute (CWPM), by grade, are shown in Table 8 with maximum scores shown in parentheses. 18 Fluency rates are also shown in box-plot diagrams in Annex 4.

17 The score for each task was calculated using the total number correct and dividing by the number of items. Student responses were coded either as a 0 (incorrect) or 1 (correct). These item scores were summed for each task and divided by the total number of items to calculate the percent correct. For instance, a student who scored 3 out of 5 on Phonemic Awareness has a percent correct score of 60 percent. 18 The maximum figures are much higher than the percent correct calculations because they are based on the number of words successfully read in a minute. These maximum scores should provide a reference for comparing the mean scores also listed. Please note that maximum scores can contain extremely high scores, called outliers. In addition, adjustments were made to the raw scores for those students who finished the task before the end of one minute. For instance, if a student read 50 words correctly in 30 seconds, their words correct per minute score would be 100 (50 words x 60 seconds/30 seconds). Because these calculations are different from percent correct, the maximum scores are higher (see Annex 4).


OCTOBER 2015 24

TABLE 8: AVERAGE PHONICS AND READING FLUENCY BY GRADE

Phonics Fluency Subtest Grade 3

Average CWPM (max)

Grade 5

Average CWPM (max)

2. Non-word reading 9.1 (71.4) 19.2 (169.4)

Reading-Rate Fluency Subtest

Grade 3 Grade 5

4. Passage Reading (Grade 2-level)

20.4 (128.9) 49.6 (180.0)

6a Passage Reading (Grade 4-level)

-- 52.3 (236.0)

At baseline in Sukkur, Grade 3 student’s averaged 20.4 CWPM and Grade 5 students averaged 49.6 CWPM on the Grade 2-level text; Grade 5 students averaged 52.3 CWPM on the Grade 4-level text. As a reference, the Pakistan Reading Project (PRP) baseline proposed fluency levels (in Urdu) to be considered as provisional standards until national reading standards are officially established. According to these standards, Grade 3 students in Sukkur would be classified as “early readers” and Grade 5 students would be in the lower half of the “intermediate” range. Importantly, 49.3 percent of Grade 3 students and 27.3 percent of Grade 5 students were unable to read a single word of Grade 2-level text. With Grade 4-level text, 39.8 percent of Grade 5 students could not read a single word correctly. Proportions of students, by fluency performance level, are discussed in section 3.5.

3.3 Discussion of EGRA Scores by Task

Task 1: Phonemic Awareness To assess students’ phonemic awareness, an untimed phoneme isolation task was included in the EGRA (Task 1). Phonemic isolation is the ability of a learner to identify a phoneme - the smallest unit of sound in a word. It is an oral activity and does not require an ability to read, though it is a strong predictor of a child’s ability to learn to read. According to the International Reading Association, phonemic awareness “skills in reading words produces better outcomes on reading comprehension as well as word and pseudo word reading.”19 In the EGRA, the task for students was to identify the last sound in a series of words - for example, the last sound in the word cat is /t/. Both Grade 3 and Grade 5 students had the lowest average percent correct scores on phonemic awareness compared to all other EGRA tasks. This was also the task with the highest proportion of students with zero-scores (or students who were unable to correctly identify the end sound in a single word; zero-score results are discussed in greater detail later in this chapter). Grade 3 students’ average score was 6.8 percent, which translates to the average Grade 3 student being able to identify less than one sound correctly out of 20. Grade 5 students’ average score was 8.7 percent which translates to the average Grade 5 student being able to identify the last sound in less than one out of ten sounds. Task 2: Phonics To assess students’ core reading skills in phonics, a timed, non-word reading task was included in the EGRA (Task 2). On this task, students were asked to read non-words (words created for this exercise) in order to test their decoding skills. If readers know how to decode, they can read a word, even if it is not a real one. In this timed task, students were asked to read up to 50 non-words in one minute. On this task, Grade 3 students’ scored an average 17.8 percent correct, or 9 non-words per minute. Grade 5 students scored an average 35.8, or 18 non-words per minute.

19 International Reading Association (2002), “Summary of the (U.S.) National Reading Panel Report Teaching Children to Read”. Page 5.


OCTOBER 2015 25

Task 3: Vocabulary To assess students’ vocabulary skills, an untimed, expressive vocabulary task was included in the EGRA (Task 3). On this task, students were presented with ten pictures and asked to identify what they saw. For example, a picture of a boy drinking would be interpreted correctly as “the boy is drinking.” Like phonemic awareness, expressive vocabulary does not require reading ability but is nevertheless an important foundational skill for learning to read as “vocabulary instruction leads to gains in reading comprehension.”20 Both grades scored highest scores in expressive vocabulary. In both grades, the lowest proportion of students with zero-scores was observed on this task. The results show that Grade 3 and 5 students have a strong oral foundation – an essential component of learning to read. Grade 3 students’ average score was 65.0 on this task and Grade 5 students’ average score was 68.5. For students in both grades, these scores translate to the average student having identified between six and seven pictures out of 10 in one minute.

Tasks 4a (Grade 2-level text) and 6a (Grade 4-level text): Fluency To assess students’ fluency, two timed passage reading tasks were included – one using a Grade 2-level text (Task 4a, administered to students in both Grades 3 and 5) and one using a Grade 4-level text (Task 6a, administered to students in Grade 5 only). Students were asked to read, aloud, as much of the 60-word passage as they could in one minute. Like vocabulary, fluency is an important predictor of comprehension. Grade 3 students read on average approximately 20.4 words per minute correctly (Correct Words Per Minute, or CWPM) on when reading Grade 2-level text while Grade 5 students read approximately 49.6 CWPM on the same task. On Task 6a (reading Grade 4 level text), Grade 5 students read on average 52.3 CWPM. While percent correct gives an indication of how students performed on this task in comparison to other tasks on the EGRA, a useful metric for this task is the fluency rate measured in Correct Words per Minute (CWPM). Fluency rates are discussed in detail in section 3.2 above. Task 4b (Grade 2-level text) and 6b (Grade 4-level text): Reading

Comprehension To assess students’ reading comprehension skills, two timed tasks on reading comprehension were included in the EGRA – these tasks included up to five questions on Grade 2-level text (Task 4b, administered to both Grade 3 and 5 students) and up to five questions on Grade 4-level text (Task 6b, administered to Grade 5 students only). The questions assessed students’ comprehension of passages read in previous fluency tasks (Tasks 4a and 6a). The number of questions a student was asked depended on their reading speed: if a student only read the first sentence correctly, she was asked only one question, relating to the content of that sentence. If however, she finished the entire passage, she was asked all five questions. If she had only reached the middle of the passage, she was only asked two or three questions. The results show that reading comprehension remains a difficult task for students in both grades. In Grade 3, students’ average score was 14.7 percent and in Grade 5, the average score was 35.5 percent for Grade 2-level text and 38.3 percent for Grade-4 level text. 21 On average, Grade 3 students were able to answer less than one question out of five correctly while Grade 5 students were able to answer less than two out of five correctly. Because the number of questions asked was dependent on how far the student read in the text, low EGRA comprehension scores are signs of both low levels of reading comprehension and low fluency rates. While the percent correct scores are useful to compare student performance across tasks, the proportion of reading comprehension questions answered correctly is also a useful metric.

20 Ibid. Page 12. 21 These results are based on the number of questions the student answered correctly out of five to ensure comparability across students. Note, however, that the number of questions a student was asked depended on their reading speed: if a student only read the first sentence correctly, she was asked only one question, relating to the content of that sentence. If however, she finished the entire passage, she was asked all five questions; if she had only reached the middle of the passage, she was only asked two or three questions.


OCTOBER 2015 26

Task 5: Listening Comprehension To assess students’ ability to understand the meaning of a story that is read to them, a task on listening comprehension (Task 5) was included in the EGRA. For this task, students did not need to read the story themselves; instead, they were asked to listen while the story was read, out loud, to them. The enumerator then asked students to answer three comprehension questions based on the passage. This test is important because it provides information on learners’ information processing skills (e.g., understanding language and memory) as well as possible disabilities (e.g., auditory, attention). It is also indicative of students’ prior knowledge. Grade 3 students were able to answer on average one and a half questions correctly while Grade 5 students were able to answer almost two questions correctly.

3.4 EGRA Results by Gender

EGRA results were further disaggregated by gender to determine how girls’ and boys’ performance on each task compared. Figure 11 shows the average score, by gender, for Grade 3 and Figure 12 shows the average score, by gender, for Grade 5.

At the task level, Grade 3 girls had significantly higher scores in Fluency, Reading Comprehension and Listening Comprehension than Grade 3 boys. In Grade 5, girls outperformed boys in Vocabulary and Listening Comprehension.

FIGURE 11: GRADE 3 SCORES BY TASK AND GENDER

*Indicates that the performance of the group with the higher mean was statistically significant, p< 0.05.

6.9%

16.7%

63.9%

26.0%

12.9%

49.5%

6.7%

19.5%

66.8%

34.3%

17.8%

58.2%

Gr.3 Boys Avg. …

Grade.3 Girls Avg. Score* 33.9%

PhonemicAwareness

Phonics Vocabulary Fluency*(Grade 2-level text)

ReadingComprehension*


Listeningcomprehension*

G3 Boys G3 Girls G3 Boys Summary Score G3 Girls Summary Score


OCTOBER 2015 27

FIGURE 12: GRADE 5 SCORES BY TASK AND GENDER


For timed tasks, fluency rates were also disaggregated by gender (At Grade 3, girls posted significantly higher fluency rates; that disparity disappeared, however, at Grade 5. Table 9). At Grade 3, girls posted significantly higher fluency rates; that disparity disappeared, however, at Grade 5.

TABLE 9: AVERAGE PHONICS AND READING FLUENCY BY GRADE AND

GENDER

Phonics Fluency Subtest Grade 3 Grade 5

Boys Girls Boys Girls

2. Non-word reading 8.4 10.2%* 19.8 17.9

Reading Fluency Subtest Grade 3 Grade 5


4a. Passage Reading (Grade 2-level)

17.6 24.9%* 49.5 49.0

6a. Passage Reading (Grade 4-level)

-- -- 53.1 50.4

3.5 EGRA Fluency by Performance Levels

Fluency rates can also be used to classify students into categories, or performance level. Using fixed intervals established for districts where Urdu is the medium of instruction (in Punjab under PRP), baseline results for students in Sukkur are summarized. This information is provided as a point of comparison since Sindhi and Urdu are comparable; however, accurate interpretations of students’ performance levels can only be made once standards (proficiency levels) are developed for use in the Sindh Province. The five levels used for Punjab by the PRP baseline are: non-readers (0 WCPM); early readers (1-40 WCPM); intermediate readers (41-80 WCPM); fluent readers (81-120 WCPM); and advanced readers (121 and above WCPM). The proportion of SRP students at each level will be tracked at midline and endline. Table 10 shows that almost half of Grade 3 students and one-quarter of Grade 5 students can be classified as non-readers when asked to read Grade 2-level text. At the other end of the spectrum, one in twenty Grade 3 students and slightly more than one-quarter of Grade 5 students can be classified as

8.4%

36.8%

67.3%58.5%

35.8%

62.4% 59.4%

39.1%

8.9%

33.7%

70.6%

55.8%

34.9%

67.9%

56.5%

36.4%Gr.5 Boys Avg. Score 44.8%

Gr.5 Girls Avg. Score 45.3%

PhonemicAwareness

Phonics Vocabulary* Fluency(Grade 2-level

text)


text)

Listeningcomprehension*

Fluency(Grade 4-level

text)


text)

G5 Boys G5 Girls G5 Boys Summary Score G5 Girls Summary Score


OCTOBER 2015 28

fluent or advanced readers at baseline. Among Grade 5 students, the proportion of students classified in each performance level using Grade 4-level text was relatively comparable to the proportion of students classified using simpler text (Grade 2-level text).

TABLE 10: FLUENCY RATES BY PRP THRESHOLDS AND GRADE

Performance Level

WCPM % of Students

Grade 3 Grade 2-level



Non-Reader 0 49.3% 27.3% 26.9%

Early Reader 1 to 40 29.4% 19.1% 17.5%

Intermediate Reader 41 to 80 16.7% 27.4% 26.8%

Fluent Reader 81 to 120 4.2% 20.1% 19.8%

Advanced Reader 121 and above 0.4% 6.1% 9.1%

3.6 EGRA Fluency and Reading Comprehension by Performance

Levels

More important than how fluently students can read a text is whether or not they understand what they have read. Not surprisingly, these two measures, fluency and reading comprehension, are related. By examining fluency and reading comprehension together, the level of comprehension can be examined as fluency rates increases. In

Category

WCPM

% of Students by Comprehension Level

(Performance Level) 0% 20% 40% 60% 80% 100% Total N

Non-Reader 0 100% 0% 0% 0% 0% 0% 100% 399

Early Reader 1 to 40 65% 19% 7% 6% 3% 0% 100% 238

Intermediate Reader 41 to 80 11% 18% 18% 26% 18% 9% 100% 132

Fluent Reader 81 to 120 6% 6% 9% 31% 25% 22% 100% 32

Advanced Reader 121 and above 0% 0% 33% 0% 33% 33% 100% 3

Total Total 71% 9% 6% 7% 5% 3% 100% 804

(graphically represented in Figure 13), the proportion of students at each comprehension and fluency level (i.e., in each cell) are shown as a percentage of the total students. Reading comprehension levels were calculated in terms of percent correct scores (e.g., 20 percent is the same as correctly answering one question out of five total questions). For example, in Grade 3, 100 percent of non-readers demonstrated no comprehension since they were not asked the comprehension questions. On the other end hand, 22 percent of fluent readers correctly answered all five comprehension questions, thereby demonstrating high levels of fluency and reading comprehension (note, however, that the number of students represented by this proportion is quite small at 32 students). Using 80 percent and higher as a threshold for high reading comprehension levels, 27 percent (18 percent + 9 percent) of intermediate readers and 47 percent (25 percent + 22 percent) of fluent readers meet or exceed this threshold. The data for advanced readers should be interpreted with caution because there were only 3 Grade 3 students who were able to read at or above 121 words per minute.


OCTOBER 2015 29

TABLE 11: GRADE 3 READING FLUENCY AND READING COMPREHENSION

Category

WCPM



Non-Reader 0 100% 0% 0% 0% 0% 0% 100% 399

Early Reader 1 to 40 65% 19% 7% 6% 3% 0% 100% 238


Fluent Reader 81 to 120 6% 6% 9% 31% 25% 22% 100% 32


Total Total 71% 9% 6% 7% 5% 3% 100% 804

Figure 13 is a graphical representation of the data presented in

Category

WCPM



Non-Reader 0 100% 0% 0% 0% 0% 0% 100% 399

Early Reader 1 to 40 65% 19% 7% 6% 3% 0% 100% 238


Fluent Reader 81 to 120 6% 6% 9% 31% 25% 22% 100% 32


Total Total 71% 9% 6% 7% 5% 3% 100% 804

. The blue areas represent students at zero to 60 percent comprehension levels and the orange areas symbolize higher degrees of comprehension. 22 This figure shows that, as one would expect, non-readers and early readers have very low levels of comprehension while more fluent readers have a higher level of comprehension of what they read.

FIGURE 13: GRADE 3 READING PERFORMANCE LEVELS AND READING

COMPREHENSION INTERVALS

22 Eighty percent correct comprehension is a USAID goal for student comprehension.


OCTOBER 2015 30

Fluency and reading comprehension levels for Grade 5 students reading Grade 2-level text are shown in Table 12 and Figure 14. Using 80 percent and higher as a threshold for high reading comprehension levels, 33 percent (21 percent + 12 percent) of intermediate readers, 63 percent (36 percent + 27 percent) of fluent readers, and 66 percent (30 percent + 36 percent) of advanced readers meet or exceed this threshold.


(GRADE 2-LEVEL TEXT)

WCPM

% of Students by Comprehension Level Task 4

Performance Level 0% 20% 40% 60% 80% 100% Total N

Non-Reader 0 100% 0% 0% 0% 0% 0% 100% 211

Early Reader 1 to 40 64% 22% 9% 5% 1% 0% 100% 148


Fluent Reader 81 to 120 5% 1% 15% 17% 36% 27% 100% 155


Total 44% 9% 10% 12% 15% 11% 100% 771

6%

11%

65%

100%

6%

18%

19%

33%

9%

18%

7%

31%

26%

6%

33%

25%

18%

3%

33%

22%

9%

0% 20% 40% 60% 80% 100%

121 WCPM andabove

81 to 120 WCPM

41 to 80 WCPM

1 to 40 WCPM

0 WCPM

0% Comprehension 20% Comprehension 40% Comprehension



OCTOBER 2015 31

FIGURE 14: GRADE 5 READING PERFORMANCE LEVELS AND COMPREHENSION

INTERVALS (GRADE 2-LEVEL TEXT)

Fluency and comprehension levels for Grade 5 students reading Grade 4-level text are shown in Table 13 and in Error! Reference source not found.. Using the 80 percent threshold for high comprehension, 74 percent of advanced readers, 61 percent of fluent readers, and 30 percent of intermediate readers met or exceeded this threshold. These proportions are comparable to those observed when Grade 5 students read Grade 2-level text suggesting that the difficulty of the text did not influence reading fluency and comprehension classifications for Grade 5 students.


(GRADE 4-LEVEL TEXT)

Performance Level WCPM % of Students by Comprehension Level

0% 20% 40% 60% 80% 100% Total N

Non-Reader 0 100% 0% 0% 0% 0% 0% 100% 208

Early Reader 1 to 40 59% 23% 13% 4% 2% 0% 100% 135


Fluent Reader 81 to 120 3% 4% 15% 18% 25% 36% 100% 153


Total Total 40% 10% 10% 13% 12% 15% 100% 769

100%

64%

12%

5%

22%

15%

9%

20%

15%

6%

5%

21%

17%

26%

21%

36%

30%

12%

27%

36%

0 WCPM

1 to 40 WCPM

41 to 80 WCPM

81 to 120 WCPM

121 and above

0% Comprehension 20% Comprehension 40% Comprehension60% Comprehension 80% Comprehension 100% Comprehension


OCTOBER 2015 32

FIGURE 15: GRADE 5 READING PERFORMANCE LEVELS AND COMPREHENSION

INTERVALS (GRADE 4-LEVEL TEXT)

3.7 EGRA Proportion of Zero-Scores (non-readers)

The results presented thus far examine the percent of items answered correctly or fluency rates – i.e., what students were actually able to do, on average. The proportion of zero-scores, however, provides information on the number of students who were unable to correctly answer even a single item on a task. Table 14 shows the proportion of zero-score students by grade and EGRA task. Zero-scores provide critical warning bells to a system, highlighting students who will continue to be left behind if not provided significant remedial help. The largest proportions of zero-scores for both Grade 3 and Grade 5 students were observed in the Phoneme Isolation task (which measures a student’s phonemic awareness) with 82.2 percent of Grade 3 and 81.0 percent of Grade 5 students unable to answer a single question correctly. The second highest proportion of zero-scores were observed in the Reading Comprehension task more than two-thirds of Grade 3 students (70.6 percent) and almost half of Grade 5 students (43.7 percent) unable to correctly answer a single comprehension question on Grade 2-level text. The next highest proportion of zero-scores was observed in the Phonics task with more than half of Grade 3 students (54.3 percent) and more than one-third of Grade 5 students (37.6 percent) unable to answer a single item correctly.

TABLE 14: EGRA PROPORTION OF STUDENTS WITH ZERO-SCORES BY GRADE

AND TASK

Core Reading Skills SRP EGRA Tasks (Task Number and Task Name)

Grade 3

(% students)

Grade 5 (% students)

1. Phonemic Awareness 1. Phoneme Isolation 82.2% 81.0%

2. Phonics 2. Non-word Reading (Decoding) 54.3% 37.6%

3. Vocabulary 3. Expressive Vocabulary 4.0% 2.5%

4. Fluency (Grade 2-level text) 4a. Passage Reading 49.3% 27.3%


4b. Reading Comprehension 70.6% 43.7%

6. Listening comprehension 6. Listening Comprehension 14.2% 6.1%

7. Fluency (Grade 4-level text) 6a. Passage reading -- 26.9%


6b. Reading comprehension -- 39.8%

1

3

8

59

100

4

20

23

10

15

15

13

14

18

28

4

30

25

17

2

44

36

13

0

0% 20% 40% 60% 80% 100%

121 CWPM andabove

81 to 120 CWPM

41 to 80 CWPM

1 to 40 CWPM

0 CWPM



OCTOBER 2015 33

Table 15 disaggregates the proportion of zero-score students by gender. Mirroring the overall EGRA results by gender, this analysis shows that the proportion of zero-scores was significantly higher for Grade 3 boys for Reading Comprehension. In Grade 5, the proportion of zero-scores were proportional for girls and boys.

TABLE 15: EGRA PROPORTION OF STUDENTS WITH ZERO-SCORES BY GRADE,

GENDER AND TASK

Core Reading Skills

SRP EGRA Tasks (Task Number and Task Name)

Grade 3 Grade 5


1. Phonemic Awareness 1. Phoneme Isolation 81.8% 83.0% 80.3% 82.5%

2. Phonics 2. Non-word Reading (Decoding) 57.3% 49.5% 35.3% 41.8%

3. Vocabulary 3. Expressive Vocabulary 4.4% 3.2% 2.7% 2.0%

4. Fluency (Grade 2-level text) 4a. Passage Reading 51.1% 46.3% 25.6% 30.3%


4b. Reading Comprehension 73.5%* 65.9% 43.3% 44.8%

6. Listening comprehension 6. Listening Comprehension 15.6% 11.9% 6.8% 5.1%

7. Fluency (Grade 4-level text) 6a. Passage reading -- -- 25.8% 29.0%


6b. Reading comprehension -- -- 38.7% 42.1%

*Indicates that the performance of the group with the higher score was statistically significant, p< 0.05


OCTOBER 2015 34

3.8 EGRA Summary Scores across Districts

Figure 16 below shows the EGRA summary scores (the average of the scores for the seven EGRA tasks23), for each of the seven Sindhi language districts, and the mean score across all districts (note that the EGRA instrument for Karachi was in Urdu language and so the results cannot be directly compared or presented here). This comparison of district-level scores is provided merely for contextual purposes; these aggregate scores do not provide the level of detail required for improving teaching and learning.

FIGURE 15: EGRA SUMMARY SCORES BY DISTRICT

In the broader context, other assessments of literacy and mathematics have been conducted in Pakistan in recent years, including a nationwide EGRA for the PRP baseline. Each of these assessments used a different tool since the instruments used must respond to that project’s specific research questions. As a result, the methodology and purpose of the nationwide EGRA and PRP projects differed from SRP, including research questions, the ways in which results can and have been used, the tasks, sampling, and data collection procedures. These differences in approach mean that results are not directly comparable across assessments. However, each of these measures do provide a broader context for understanding reading and mathematics performance among students in Pakistan, and provide different vantage points. The extent of comparability, and the circumstances under which these results are comparable, however, is outside the scope of this report.

3.9 EGRA Summary and Recommendations

The SRP EGRA baseline results provide useful information to help guide interventions to improve reading for Sukkur district students. A summary of findings, by grade, is presented below followed by recommendations.

Summary

Grade 3: At baseline, students in Grade 3 had the highest scores in Vocabulary, followed by Listening Comprehension. The lowest scores were observed in Phonemic Awareness, which also had the highest proportion of zero-scores. Across all tasks, the average percent correct was 31.1 percent. Promisingly, the highest scores – in the areas of Vocabulary and Listening Comprehension – are indicators of prior knowledge and precursors to a student’s future success as a reader.

23 Grade 5 students were assessed on Grade 4-level reading and mathematics as well as Grade 2-level. Only the Grade 2-level results are included in the Summary Scores. The Summary Scores thus show the average percent correct for each grade for the same tasks.

25.028.8 29.1 31.1

33.5 33.737.7

43.942.0

40.2

45.1 45.7 46.953.2

Kambar Jacobabad Larkana Sukkur Dadu Khairpur Mirs Kashmore

Grade 3 Grade 5 Grade 3 Mean Grade 5 Mean


OCTOBER 2015 35

Grade 5: Students in Grade 5 scored highest on Vocabulary followed and Listening Comprehension, although more than one-quarter of students could not read a single word (i.e., zero-scores). Across all tasks, the average percent correct was 45.1 percent. The lowest scores were observed in Phonemic Awareness, which also had the highest proportion of zero-scores.

Recommendations

Grade 3 and Grade 5 students had the highest scores in vocabulary and listening comprehension. These findings suggest that students’ oral foundation and background knowledge are areas of relative strength. While Grade 5 students show a relative strength in fluently reading text, their fluency rate should be interpreted within the context of the provisional fluency standards taken from the PRP baseline. These standards suggest that Grade 3 students are in the “early readers” range and Grade 5 students are in the lower half of the “intermediate readers” range. These fluency rates, and the low performance of students in Phonemic awareness, suggest that generally, students in Sukkur need continued support to build both fluency and comprehension and also struggle with sound identification of words in isolation.

To this end, following recommendations are proposed in order to focus SRP interventions on areas of need for students, teachers, and head teachers:

1. Recommendations to strengthen reading instruction:

Capitalize on students’ relative strengths in the areas of Vocabulary and Listening Comprehension throughout reading instruction. For example, book selection should incorporate topics where students have prior background, where possible.

Ongoing formative assessment should guide instructional decision-making for the teacher to ensure that students’ instructional skills are identified and targeted for instruction.

Small group instruction is provided to students via reading groups that have students together of a similar reading level. Leveled books at the student’s “instructional level” are used in small group reading instruction. “Instruction level” is with text where the student can read between 90-95% of the words with automaticity.

Students should be provided with adequate time each day to practice independent reading skills, regardless of their reading level. Independent reading practice should be with text where the student can read at least 95% of the words with automaticity, have prior knowledge of the topic, and is of high interest to the student. Students of similar reading levels can pair up and read to each other aloud, to practice reading text at their “independent level.” Students can also read independently to themselves to practice reading. Students’ reading endurance is to be increased as they development, starting with 5-10 minutes for younger grades and 20 or more minutes for older grades.

To strengthen reading Fluency, Reading Comprehension, and Vocabulary development, daily read alouds should be incorporated by the classroom teacher. Book selection for read alouds should be at a reading level higher than what the students can read independently. This allows for exposure to different genres, content vocabulary, and builds prior knowledge in students.

Ensure phonemic awareness is taught regularly and explicitly, particularly in Grade 1, as it will provide a foundation for later phonics instruction.

Daily, explicit, structured phonics instruction should be taught in an engaging, multisensory, authentic manner. Adequate practice is incorporated, such that mastery of each decoding skill is attained prior to moving on to the next skill. Phonics skills include asking students to identify the relationship between letters and sounds. Activities may include writing new words and word parts on the blackboard and asking students to decode them.

Incorporate reading practice to enhance reading fluency with a “repeated reading” strategy, whereas a student re-reads a passage more than one time. This enhances comprehension and allows the student to experience reading with enhanced fluency. This strategy can also be used as “paired reading,” where two students of similar reading levels are paired and engage in “repeated reading” with each other.


OCTOBER 2015 36

Incorporate regular and explicit practice with “sight words” to enhance reading automaticity. Sight words are high frequency words that a student can read by sight” (i.e., not by sounding out). Using a “Drill Sandwich” strategy to learn sight words is an effective method for mastery of sight words, whereas adequate challenge is incorporated into the learning set. Typically, a learning set where 70% of the words are mastered and 30% of the words are new is considered to be adequate challenge. If this proves too difficult, the learning set can be made easier (e.g., 90% mastered and 10% unknown).

Provide teachers with explicit strategies for strengthening listening and reading comprehension skills such as pre-reading (asking questions before reading a story, teaching vocabulary, discussing the subject of the story before reading), read-alouds (the teacher reads a story and asks questions along the way to draw students’ attention to important details, query their comprehension, or ask them to predict) and post-reading (such as group, pair, and independent reading, drawing pictures, continuing or dramatizing the story) to bolster comprehension and reading skills, simultaneously.

2. Recommendations to strengthen school support for reading:

Develop a support system for reading teachers so they can share best practices and promising practices across and between colleagues for reading instruction.

Provide high quality, targeted professional learning for teachers regarding the implementation of best practices in reading instruction. Included in this professional learning is a framework to use formative assessment to guide instruction.

Expand time spent reading in and out of class. For example, designate class time for reading each day, ensure that students have access to reading materials at school and at home, develop highly engaging reading events such as a community “reading marathon,” and use volunteers or “cross-age buddies” to build language skills.

Create a library within each classroom so that students have ready access to high interest reading materials.

Incorporate the premise that the learning of all students is enhanced when there is adequate challenge (“not too easy, not too hard”). Recognize that behavioral concerns will most often be displayed when there is not an instructional match for the student (the work being given to the student is either too easy or too difficult. Providing teachers with professional learning in “instructional assessment” is key to student success. “Instructional assessment” incorporates the notion that instruction and materials are matched to the student’s individual skills, prior knowledge, and interests.

3. Recommendations for further research or actions by SRP staff:

Establish standards for reading fluency in the Sindhi language.

Identify critical areas of intervention to focus on schools and/or sub-groups where large proportions of children are being left behind (i.e., those with zero-scores). Targeted interventions may include pullout of non- and struggling readers for intensive teaching of core competencies/foundation skills, early enrichment activities in formal educational care or via informal mother training to help develop pre-reading foundational skills, implementation of procedures for diagnosing reading difficulties, remediation, and the use of formative assessment to ensure students who lack foundational reading skills are identified early.

Examine factors that may contribute to boys lagging behind girls in the development of reading skills in Grade 3 and Grade 5. Provide targeted interventions for where needed.


OCTOBER 2015 37

CHAPTER 4: EGMA RESULTS – SUKKUR DISTRICT

This section provides an overview of EGMA results for students in Sukkur district. First, overall results across all tasks are presented as a distribution of students by the percent correct score in Grade 3 and Grade 5. Second, results are disaggregated by task and gender with results summarized by percent correct and fluency (for timed tasks). Third, the proportion of students who were unable to answer a single question on the task (or zero-scores) are reported. The section concludes with recommendations for improved mathematics outcomes as suggested by the EGMA results. Additional details on score calculations, including percent correct, timed task scores (fluency) and task score ranges and calculations are provided in Annex 2. EGMA item-level statistics, by task and grade, are provided in Annex 3.

4.1 Overall EGMA Results

The graphs below show the number of students by the percent of total items correct on the EGMA. For example, the tallest bar in the first graph (Grade 3) shows that approximately 42 students had correctly answered 10 percent of the items correctly. This graph (or histogram) provides a snapshot of the number of students at each level of performance, as well as a profile of the group as a whole. The black trendline summarizes the overall pattern of percent correct for the group based on a calculation of averages over the entire group.

The histograms in Figure 16, show that math skills in the Sukkur district are average – the trend lines peak fairly close to center. The Grade 3 summary mean score was 42.6 percent. The trend line for Grade 5 peaks further to the right, indicating even stronger performance by the Grade 5 students, with a mean score of 62.2 percent correct.

FIGURE 16: DISTRIBUTION OF EGMA SCORES

Grade 3 Grade 5

4.2 EGMA Results by Task

While the overall results are useful to determine how students in Grades 3 and 5 performed across all EGMA tasks, the remainder of this section focuses on EGMA results by task. Table 16 shows the average percent correct score, by task.24 The average score for students in Grade 3 in Sukkur ranged from 26.2 percent to 62.8 percent correct; the average score for students in Grade 5 ranged from 20.1 percent to 81.5 percent correct.

24 The score for each task was calculated using the total number correct and dividing by the number of items.


OCTOBER 2015 38

TABLE 16: EGMA PERCENT CORRECT SCORES BY GRADE AND TASK

Core Mathematics Skills

SRP EGMA Task (Number and Task Name)

Grade 3

(% correct)

Grade 5 (% correct)

1. Number Identification 1. Number identification up to 999 62.8% 81.5%

2. Number Discrimination

2. Number discrimination up to 999 54.9% 76.1%

3. Missing Number 3. Missing Numbers up to 999 27.1% 37.9%

4. Addition, level 1 4. Addition level 1 within 10 32.7% 58.4%

5. Subtraction, level 1 5. Subtraction level 1 within 10 26.2% 49.8%

6. Word Problems 6. Word Problems 52.1% 69.3%

7. Simple Multiplication (Grade 5 only)

7. Multiplication up to 2 digits -- 20.1%

Summary Score (all tasks) 42.6% 62.2%

Fluency

Basic fact fluency in mathematics, also called computation fluency, is defined as “the efficient,

appropriate, and flexible application of single-digit calculation skills and…an essential aspect of

mathematical proficiency.”25 Moreover, “computation fluency in the elementary grades is a strong

predictor of later success in higher-level mathematics coursework.”26 Critical components of fluency

measures include the ability: 1) to quickly and accurately add and subtract or multiply and divide; and 2)

know from memory all sums and products of problems using two one-digit numbers. Fluency rates were

calculated for three EGMA tasks: addition, subtraction and multiplication. 27 The average fluency rates,

by grade, are shown in

Table 17 with maximum scores shown in parentheses.28 Fluency rates are also shown in box-plot

diagrams in Annex 4.

TABLE 17: MATHEMATICS FLUENCY RATES BY GRADE

Mathematics Fluency Tasks (Task Number and Name)

Grade 3

Average Correct Responses per Minute (Max)

Grade 5

Average Correct Responses per Minute (Max)

4. Addition 6.9 (37.7) 13.5 (52.2)

5. Subtraction 5.5 (48.0) 10.6 (46.2)

7. Multiplication up to 2 digits (Grade 5 only)

-- 4.3 (36.4)

25 NCTM. Assessing Basic Fact Fluency. Teaching children mathematics. April 2014. Vol. 20, No. 8 26 Gersten, R., Jordan, N. C., & Flojo, J. R. (2005). Early identification and interventions for students with mathematics difficulties. Journal of Learning Disabilities, 38, 293-304. 27 Though Task 1: Number identification was timed, it is not considered a mathematics fluency task because no computation is required. 28 The maximum figures are much higher than the percent correct calculations because they are based on the number of words successfully read in a minute. These maximum scores should provide a reference for comparing the mean scores also listed. Please note that maximum scores can contain extremely high scores, called outliers. In addition, adjustments were made to the raw scores for those students who finished the task before the end of one minute. (See Annex 4 for detailed calculations).


OCTOBER 2015 39

4.3 Discussion of EGMA Results by Task Task 1: Number Identification To assess students’ ability to identify numbers, a timed number identification task was included in the EGMA (Task 1) that presented students with up to 20 numbers up to 999. Grade 3 students’ average score was 62.8 percent which translates to the average Grade 3 student being able to identify (read) 13 numbers out of 20 in one minute. Grade 5 students’ average score was 81.5 percent which translates to the average Grade 5 student being able to correctly read 16 numbers in one minute. Task 2: Number Discrimination To assess students’ ability to discriminate between numbers up to 999, an untimed task was included in the EGMA (Task 2). On this task, students were presented with a pair of numbers and asked to identify (read) the larger number. Grade 3 students’ average score was 54.9 percent; in other words, the average Grade 3 student could identify the larger number in approximately 11 of the 20 pairs presented to them. Grade 5 students’ average score was 76.1 percent; in other words, the average Grade 5 student correctly identified the larger number in 15 of the 20 pairs. Task 3: Missing Number To assess students’ ability to identify a missing number when given a pattern, an untimed task was included on the EGMA (Task 3). The average score for Grade 3 students was 27.1 percent which translates to the average Grade 3 student being able to correctly identify missing numbers in almost three out of 10 patterns. Grade 5 students’ average score was 37.9 which translates to the average Grade 5 student identifying the missing number correctly for almost four out of 10 number patterns. Tasks 4 and 5: Addition and Subtraction29 To assess students’ computation skills, two timed tasks on addition and subtraction were included on the EGMA. On these tasks, students were provided with a printed list of 20 addition problems consisting of two single-digit numbers with sums no greater than 10, and a printed list of 20 subtraction problems consisting of two single-digit numbers under 10 (e.g. 9 - 2 = …. ). For students in Grade 3, the average score in addition (Task 4) and subtraction (Task 5) was 32.7 percent and 26.2 percent, respectively. Subtraction was the lowest for all tasks in Grade 3. In other words, the average Grade 3 student correctly solved seven addition problems and five subtraction problems out of 20 problems for each task in one minute. For students in Grade 5, the average score in addition (Task 4) and subtraction (Task 5) was 54.8 percent and 49.8 percent, respectively. In other words, the average Grade 5 student correctly solved approximately 10 addition and 10 subtraction problems out of 20 problems for each task in one minute. Task 6: Word Problems To assess students’ ability to solve mathematical computations in the context of a word problem, an untimed task was included on the EGMA (Task 6). Students were given six word problems for which one or more computations were needed in order to arrive at the correct answer. Grade 3 students’ average score was 52.1 percent correct; in other words, the average Grade 3 student was able to solve just over three out of six problems. Grade 5 students’ average score was 69.3 percent correct; in other words, the average Grade 5 student was able to four of six problems correctly. Task 7: Simple multiplication (Grade 5 only) To assess Grade 5 students’ multiplication skills, a timed task was included on the EGMA (Task 7). For this task, students were provided with a printed list of 20 multiplication problems in which they had to mentally multiply two single-digit numbers (e.g. 4 x 3 =…) to arrive at the correct answer. This task proved to be the most difficult for Grade 5 students – their average score was 20.1 percent correct. In

29 Level 1.


OCTOBER 2015 40

other words, Grade 5 students, on average, were able to compute approximately four out of 20 multiplication problems correctly in one minute.

4.4 EGMA Results by Gender

EGMA results were further disaggregated by gender to determine how girls’ and boys’ performance on each task compared. Error! Reference source not found. shows the average score, by gender, for Grade 3 and Figure 18 shows the average score, by gender, for Grade 5. In Grade 3, boys had a higher overall score on the EGMA than girls by 7.8 percentage points and outperformed girls in each task except for Missing Number. In Grade 5, boys outperformed girls by 12.8 percentage points and outperformed girls in every task except for Missing Number.

FIGURE 17: GRADE 3 EGMA SCORES BY GENDER


FIGURE 18: GRADE 5 EGMA SCORES BY GENDER


For timed tasks, fluency rates were also disaggregated by gender (

67.1%

57.7%

26.7% 37.5% 29.8%

54.9%56.0%

50.3%

27.9% 25.0%20.4%

47.5%Gr.3 Boys Avg. Score* 45.6%

Gr. 3 Girls Avg. Score 37.8%

NumberIdentification*

NumberDiscrimination*

Missing Number Addition* Subtraction* Word Problems*

G3 Boys G3 Girls G3 Boys Average G3 Girls Average

87.1%82.0%

38.4%

64.3%

56.5%

73.2%

24.9%

72.5%66.5%

37.2%

48.8%

36.6%

63.2%

12.2%

Gr.5 Boys Avg. Score* 66.9%

Gr. 5 Girls Avg. Score 54.5%

NumberIdentification*

NumberDiscrimination*

Missing Number Addition* Subtraction* Word Problems* Multiplication*

G5 Boys G5 Girls G5 Boys Average G5 Girls Average


OCTOBER 2015 41

Table 18). In Grades 3 and 5, boys significantly outperformed girls in all computation tasks.


OCTOBER 2015 42

TABLE 18: MATHEMATICS COMPUTATION FLUENCY RATE MEANS BY GRADE

AND GENDER

Mathematics Fact Fluency Subtest

(Task Number and Name)

Grade 3 Grade 5


4. Addition 7.8* 5.4 15.1* 10.8

5. Subtraction 6.1* 4.6 12.1* 8.2

7. Multiplication -- -- 5.3* 2.6

*Indicates that the performance of the group with the higher mean was statistically significant, p< 0.05

4.5 EGMA Proportion of Zero-Scores

The results presented thus far examine the percent of items answered correctly, or fluency rates – i.e., what students were actually able to do, on average. The proportion of zero-scores, however, provides information on the number of students who were unable to correctly answer even a single item on a task.

Table 19 shows the proportion of zero-score students by grade and EGMA task. As noted with the EGRA results, zero-scores provide critical warning bells to a system, highlighting students who will continue to be left behind if not provided significant remedial help.

In both grades, some of the highest proportion of zero-scores were observed in addition and subtraction compared to all other tasks. More than one-third of the Grade 3 students had zero-scores in addition (39.6 percent) and almost half in subtraction (45.1 percent). In Grade 5, almost half of students had zero-scores in multiplication (45.7 percent), and approximately one-fifth had zero-scores in addition (17.1 percent) and in subtraction (21.2 percent). The fewest number of students in both grades had zero-scores in the Number Identification and Word Problems tasks.

TABLE 19: EGMA PROPORTION OF STUDENTS WITH ZERO-SCORES BY GRADE

AND TASK

Task (Subtest) Grade 3

(% students) Grade 5

(% students)

1. Number identification 2.0% 0.4%

2. Number discrimination 10.4% 2.6%

3. Missing Number 26.0% 14.1%

4. Addition 39.6% 17.1%

5. Subtraction 45.1% 21.2%

6. Word Problems 6.2% 2.1%

7. Multiplication -- 45.7%

Disaggregating the proportion of zero-score students (Table 20) by gender shows that in both Grade 3 and Grade 5, the proportion of zero-scores for girls was significantly higher for girls than for boys for the computation tasks (addition and subtraction in Grades 3 and 5 and multiplication in Grade 5).


OCTOBER 2015 43

TABLE 20: EGMA PROPORTION OF ZERO-SCORES BY GRADE, GENDER AND

TASK

Task (Subtest) Grade 3 Grade 5


1. Number identification 2.4% 1.3% 0.0% 1.0%

2. Number discrimination 10.4% 10.3% 1.7% 4.0%

3. Missing Number 26.1% 26.0% 14.6% 13.5%

4. Addition 32.9%* 50.5% 12.3%* 24.9%

5. Subtraction 39.1%* 54.7% 14.4%* 32.3%

6. Word Problems 5.0% 8.0% 0.8% 4.0%

7. Multiplication -- -- 40.8%* 53.5%

*Indicates that the performance of the group with the higher score was statistically significant, p< 0.05.

4.6 EGMA Summary and Recommendations

The SRP EGMA baseline results provide useful information to help guide interventions to improve mathematics for Sukkur district students. A summary of findings, by grade, is presented below followed by recommendations.

Summary

Grade 3: Grade 3 students scored highest on Number Identification, Number Discrimination, and Word Problems; the same tasks in which the proportion of zero-scores was lowest. Notably, results for word problems were stronger than the results for Tasks 3-5 (Missing Number, Addition and Subtraction) which suggests that, for many students, computations in the context of word problems were easier than isolated computation problems. Grade 3 students scored lowest on the Missing Number and computation tasks (Addition and Subtraction); computation tasks had the highest proportion of students with zero-scores. Grade 5: Grade 5 students scored highest in Number Identification, Number Discrimination, and Word Problems, while they scored lowest on Multiplication and Missing Number. The relatively higher performance in word problems compared to computation is notable among Grade 5 students as with Grade 3 students. Furthermore, almost half of Grade 5 students were unable to answer a single question correctly on the Multiplication task, and approximately one-fifth of students could not answer a single addition or subtraction problem correctly.

Recommendations

The SRP EGMA baseline results provide useful information to help guide interventions to improve mathematics skills among students in Sukkur district. The following recommendations are based on these results to strengthen mathematics instruction:

Provide teachers with strategies for building number sense and computational approaches, which would enable students to identify numbers and patterns and build computational skills (addition, subtraction, and multiplication).

Provide teachers with strategies for diagnosing students experiencing difficulties in number sense and computation, and formative assessment to ensure students who lack foundational mathematics skills are identified early.

Examine trends by sub-groups and geography to determine root causes for the poor performance among girls in both Grades 3 and 5 relative to that of boys. Use these findings to develop targeted intervention for girls.


OCTOBER 2015 44



OCTOBER 2015 45

CHAPTER 5: CONTEXTUAL VARIABLES AND EGRA

AND EGMA PERFORMANCE

The SRP baseline study also included questionnaires in which a series of questions were posed to the students taking the EGRA and EGMA tests and to their teachers and head teachers. The questions were designed to capture contextual variables, including student, teacher, and Head Teacher characteristics to examine the relationships between these contextual variables and students’ reading and mathematics performance. To this end, responses to the questionnaire were summarized and correlated with the EGRA and EGMA summary scores. The goal of this analysis was to answer the following type of question: “does having reading materials at home correspond to higher EGRA/EGMA scores?” Analyses of these variables did not reveal many significant relationships, but this may have been due to the small sample size of questionnaire results, particularly with the teacher and head teacher data.30 While these relationships are useful to examine as contextual factors that may positively or negatively influence students reading and mathematics performance, the generalizability of these data is limited and, therefore, the results must be interpreted with caution.

Key Findings

Student questionnaire:

Reading materials at home: About thirty percent of students said they had no reading materials at home; Grade 3 students who reported having reading materials at home had higher EGRA scores and Grade 5 students who had reading materials at home generally had higher EGMA scores.

Practice reading at home: A majority of students practiced reading at home either reading aloud to another person, silently to self, or reading the Quran. These reading activities were generally associated with higher EGRA scores for Grade 3 students.

Work with peers: More than two-thirds of students reported completing classwork with peers; Grade 5 students’ EGMA scores were significantly higher than those of students who did not complete classwork with peers.

Absenteeism: More than one-third of students missed days from school the previous week; Grade 3 students’ EGRA scores were significantly lower than those who were not absent.

Teacher and head teacher questionnaires:

Teacher qualifications: Teacher qualifications did not have a significant impact on assessment scores.

Head teacher support: Seventy-six percent of head teachers reported that they monitored students results on tests administered by teachers; this practice was associated with higher EGRA scores in Grade 5.

Timetable implementation: Two thirds of head teachers indicated that their schools followed a timetable; of those who did follow a timetable, most set aside time for teaching reading and math. No significant differences in EGRA or EGMA corresponded to timetable adherence.

School Infrastructure: Facility improvements at schools reported by head teachers included: clean water (51%); electricity (69%); and toilet facilities (53%). These facility improvements were not associated with higher assessment scores. No head teacher reported having a library at their school.

30 Reasons that responses from the Questionnaire must be treated with caution include: the questionnaire was not piloted; small samples can lead to a high margin of error (the number of teachers and head teachers surveyed in each district is relatively small). Note also that there were some students, teachers, and head teachers who did not respond to certain questionnaire items; they were labeled as missing.


OCTOBER 2015 46

Detailed data from student, teacher and head teacher questionnaires are provided in Annex 5 and summarized in the tables below. Table 21 and


OCTOBER 2015 47

Table 22 summarize the student and the teacher and head Teacher responses that corresponded with higher scores on the EGRA or EGMA – these results are analyzed and presented by language of the test and not by district (note that results for Sukkur are presented in the following sections). By language of the test, results are presented only if the difference was statistically significant; if the result was only statistically significant for one language (Urdu or Sindhi), then the language where significant findings were observed is noted.

TABLE 21: CONTEXTUAL VARIABLES FROM STUDENT QUESTIONNAIRE

ASSOCIATED WITH EGRA OR EGMA PERFORMANCE

Student Responses that Corresponded with Higher

Scores on the EGRA or EGMA

Grade 3 Grade 5

EGRA EGMA EGRA EGMA

Language at home is same as test language * (Urdu) * (Sindhi)

Type of reading material at home: Newspapers

* *

Type of reading material at home: Magazines

* *

Type of reading material at home: Books *

Stories Read Aloud to Someone at Home *

Practice Reading Stories Aloud to Someone at Home * *

Practice Silent Reading at Home * * * *

Read Quran at Home * * *

Study or Do Class-work with Classmates * * * *

Did Not Miss Any Days of School in the Previous Week

* * * *

Do Not Work Before or After School * * *

Watch Television at Home *

Have a Computer at Home * *

Type of Vehicle at Home: Motorcycle *

Type of Vehicle at Home: Car * * *Summary scores were significantly higher among students who demonstrated the behavior more often (p < .05)


OCTOBER 2015 48

TABLE 22: CONTEXTUAL VARIABLES FROM TEACHER AND HEAD TEACHER

QUESTIONNAIRE ASSOCIATED WITH EGRA OR EGMA PERFORMANCE

Teacher and Head Teacher Responses that Corresponded with Higher Student Scores on

the EGRA or EGMA

Grade 3 Grade 5

EGRA EGMA EGRA EGMA

Teachers Do not teach multi-grade classroom * *

Use flashcards * (Sindhi)

Use charts * (Sindhi) * (Sindhi) * (Sindhi)

Use novels/books * (Sindhi) * (Sindhi)

Use magazines * (Sindhi) * (Sindhi)

Have individual meetings with parents * (Sindhi) * (Sindhi)

Updating skills through peer support * (Urdu)

Updating skills by attending seminars or educational conferences

* (Sindhi)

Updating skills through in-service training * (Sindhi) * (Sindhi) * (Sindhi)

Have language or reading teacher guide * (Urdu)

Have mathematics teaching guide * (Urdu)

Head Teachers Reviewing student evaluations by supervisors for Union Council’s

* (Urdu)

*Summary scores were significantly higher among teachers/head teachers who demonstrated the behavior more often (p < .05)

The sections below provide results from the student, teacher and head teacher questionnaires for Sukkur district.

5.1 Student Questionnaire

The 1,583 students participating in the EGRA and EGMA assessments in the Sukkur district were also surveyed with the student questionnaire. Student questions related to language spoken at home; availability of reading materials at home; study habits; access to communication technology; and vehicles at home.

Language Spoken at Home

Students were asked what primary language was spoken in the home. Both grades showed similar patterns in the primary language spoken at home. More than 80% of families spoke Sindhi, followed by Sireaki (13%) and Balochi (4%); less than one percent spoke Punjab, Urdu or another language at home. Although the assessments were in Sindhi, about 20 percent of Sukkur students spoke a different primary language at home. Comparing with average EGRA/EGMA summary scores showed that speaking the same language at home as the test had no clear advantage on the EGRA or the EGMA.

Types of Reading Material at Home

Students were asked what types of reading materials were found at their homes (newspapers, magazines, books, or nothing). About thirty percent of students said they had no reading materials at home; Grade 3 students who reported having reading materials at home had higher EGRA scores and Grade 5 students who had reading materials at home generally had higher EGMA scores.


OCTOBER 2015 49

Reading Habits at Home

Students were also asked about their reading habits at home (does someone read aloud to them; do they practice reading aloud to anyone; do they practice silent reading; do they read the Quran at home). About half of students in both Grades reported that someone at home read stories aloud to them. This practice was not associated with higher EGRA or EGMA scores. A majority of students practiced reading at home either reading aloud to another person, silently to self, or reading the Quran. These reading activities were generally associated with higher EGRA scores for Grade 3 students.

Study Habits, School Attendance, Parents’ Knowledge of School, Work Duties

Students were asked questions about study habits, school attendance, parents’ knowledge of school performance and work-related duties. More than two-thirds of students reported completing classwork with peers; Grade 5 students’ EGMA scores were significantly higher than those of students who did not complete classwork with peers. More than one-third of students missed days from school the previous week; Grade 3 students’ EGRA scores were significantly lower than those who were not absent. Approximately half of Grade 3 and Grade 5 students reported that parents had no knowledge of their school performance. This behavior was not associated with higher EGRA or EGMA scores.

Communication Technology and Vehicles at Home

When asked about access to communication technology at home. Almost three-quarters of students reported watching television at home and more than one-third reported listening to the radio. Less than one-fifth of students reported having computers at home. Few students reported having internet at home, and most students did not answer this question. The final section of the student questionnaire asked what types of vehicles were at home (bicycle, motorcycle, or car). About one-fifth of students reported having none of these.

5.2 Teacher and Head Teacher Questionnaires

In Sukkur district, 130 teachers and 70 head teachers were surveyed on questions pertaining to qualifications, attendance at training sessions, and school-related factors such as scheduling, teacher support, head teacher’s knowledge of student performance, teacher observation procedures, School Management Committee (SMC) authority, and facility improvements. Of the 130 teachers surveyed, 65 were Grade 3 teachers (41 female and 24 male) and 64 were Grade 5 teachers (43 female and 22 male). Of the 70 head teachers surveyed, 24 were female and 43 were male teachers.

Teacher Qualifications

Roughly one-third of teachers reported having a M.A./M.Sc/M.Phil. and 45 percent of teachers reported a B.A./B.Sc. as their highest academic qualification. Most teachers had masters or bachelors academic and professional qualifications. Almost two-thirds of teachers taught multi-grade classrooms. These practices were not associated with higher EGRA or EGMA scores.

Teaching Materials and Practices

About 29 percent of teachers reported using books other than text books in the classroom but this was not associated with student assessment scores. Almost all of the teachers conducted individual meetings with parents, had head teachers who encouraged them to use a variety of activities, and gave extra time to slow learners after class. These practices were not associated with higher EGRA or EGMA scores.


OCTOBER 2015 50

Updating Teaching Methods

Teachers were asked to select the method in which they keep current on teaching methods. Almost all respondents noted reading books as their method of updating knowledge. Almost two-thirds reported attending in-service training. Approximately one-fifth sought the internet or attended seminars or educational conferences. These practices were not associated with higher EGRA or EGMA scores.

Head Teacher Qualifications

Most head teachers had either a M.A./M.Sc./M.Phil. or B.A./B.Sc. degree and a B.Ed. as certification professional qualifications.

Head Teacher Classroom Experience and Training

Most head teachers (79 percent) reported teaching classes. Those schools in which the Head Teacher also taught classes had higher EGMA scores in Grade 3. Approximately one-third of the head teachers reported having received any school management training in the previous year; more than half had received training for implementing reading programs and 40 percent for implementing math programs.

Timetable Implementation

The next series of questions pertained to the use of timetables and assigning specific periods to reading and mathematics. Two-thirds of the head teachers indicated that their schools followed a timetable; of those who did follow a timetable, 57 percent set aside time for teaching reading and 63 percent set aside time for teaching math.

Head Teacher Support and Knowledge of Student Progress

Almost all head teachers reported supporting teachers in reading and math instruction.

Head Teacher Observation of Teachers

Seventy-six percent of head teachers reported that they monitored students results on tests administered by teachers; this practice was associated with higher EGRA scores in Grade 5. Less than half (44 percent) indicated that the head teacher had responsibility for observing teachers (others said no one or the Deputy Teacher).

SMC Decision Making Authority and Facility Improvement

Sixty-seven head teachers indicated that their school had an SMC. SMC involvement or responsibility was not associated with consistently higher assessment scores. Many schools had facility improvements such as clean water, electricity, and toilet facilities. Forty-six percent had gender separate toilet facilities at mixed schools and only 3 schools reported having a library.


OCTOBER 2015 51

ANNEXES

Annex 1. Quality of Assessment Tools – Additional Data

i. Cronbach’s alpha

TABLE 23: SUKKUR RELIABILITY ESTIMATES

Assessment Grade Level Tasks Number Alpha

Reading Grade 3 6 810 0.78

Grade 5 8 773 0.89

Mathematics Grade 3 6 810 0.86

Grade 5 7 773 0.87

ii. P-values and Item-total correlations

TABLE 24: EGRA TASKS STATISTICS FOR SUKKUR

Task (Subtest)

Grade 3 Grade 5

p-value Item-Total

p-value Item-Total

1. Phoneme Isolation .07 .39 .09 .22

2. Non-word Reading (Decoding) .18 .76 .36 .80

3. Expressive Vocabulary .65 .28 .69 .31

4a. Passage Reading (Grade 2-level) .29 .74 .58 .89

4b. Reading Comprehension .15 .76 .36 .84

5. Listening Comprehension .53 .35 .64 .37

6a. Passage reading (Grade 4-level) -- -- .58 .86

6b. Reading comprehension (Grade 4-level) -- -- .38 .85

Reading Summary Score -- --


OCTOBER 2015 52

TABLE 25: EGMA TASKS STATISTICS FOR SUKKUR

Task (Subtest) Grade 3 Grade 5

p-value Item-Total p-value Item-Total

1. Number identification up to 999 .63 .77 .81 .68

2. Number discrimination up to 999 .55 .75 .76 .70

3. Missing Number up to 999 .27 .57 .38 .56

4. Addition level 1 within 10 .33 .60 .58 .67

5. Subtraction level 1 within 10 .26 .65 .50 .74

6. Word Problems .52 .59 69 .61

7. Multiplication up to 2 digits -- -- .20 .55

EGMA Summary Score .43 -- .62 --


OCTOBER 2015 53

Annex 2: Score Calculation Definitions

Percent Correct

The results of the EGRA testing were calculated using task and summary scores. Table 26 lists the tasks, stimuli, raw score ranges, and the method for calculating the task and summary scores on the test. For each of the tasks, the stimuli (items) (i.e., questions, letters, sounds, words, and non-words) were worth one score point. The score points were added and since the range of raw scores varies across the tasks, the percent of correct scores was used to report all results. No weighting was used with the tasks to calculate the summary scores. Each task summary score was calculated using the total of the percent correct scores and dividing it by the number of items (i.e., the average percent correct). The overall Reading Summary Score was calculated by adding all of the task summary scores and dividing by nine (total number of tasks) to arrive at the average.

Timed Tasks Scores

The scores on the timed tasks were calculated (adjusted) by taking the number of correct responses (i.e., the raw score) times 60 seconds then dividing that number by the number of seconds used to read the stimulus. For instance, if a student read 25 letters correctly in 30 seconds, their letters-correct-per-minute score would be 50 (25 words x 60 seconds/30 seconds). Given another example, if a student completes 12 additions correctly in 30 seconds, his or her timed task score would be 24 additions per minute (12 additions x 60 seconds/30 seconds).

EGRA Score Ranges and Calculation

TABLE 26: EGRA GRADE 3 SCORE RANGES AND CALCULATIONS

Task (Subtest) Stimuli Score Range Calculation

1. Phoneme Isolation 10 questions (untimed) 0-100 Percent correct of answers

2. Non-word Reading (Decoding) 50 non-words (timed) >100 Words correct per minute

3. Expressive Vocabulary 10 pictures (untimed) 0-100 Percent correct of words

4a. Passage Reading (Grade 2-level) 60 words (timed) >100 Words per minute

4b. Reading Comprehension 5 questions (untimed) 0-100 Percent correct of answers

5. Listening Comprehension 3 questions (untimed) 0-100 Percent correct of answers

Reading Summary Score 6 tasks 0-100 Average of percent correct


OCTOBER 2015 54

TABLE 27: EGRA GRADE 5 SCORE RANGES AND CALCULATIONS

Task (Subtest) Stimuli Score Range

Calculation

1. Phoneme Isolation 10 questions (untimed) 0-10 Percent correct of answers

2. Non-word Reading (Decoding) 50 words (timed) >100 Words correct per minute

3. Expressive Vocabulary 10 pictures (untimed) 0-100 Percent correct of words

4a. Passage Reading (Grade 2-level) 60 words (timed) >100 Words correct per minute

4b. Reading Comprehension 5 questions (untimed) 0-100 Percent correct of answers

5. Listening Comprehension 3 questions (untimed) 0-100 Percent correct of answers

6a. Passage reading (Grade 4-level) 60 words (timed) >100 Words correct per minute

6b. Reading comprehension

(Grade 4-level) 5 questions 0-100 Percent correct of answers

Reading Summary Score 6 tasks 0-100 Average of percent correct

An example of percent correct scores for each of the tasks and as a summary score is provided below. The raw score is divided by the maximum score (the highest score possible in the score range) to produce the percent correct score for each task. Then, the task scores are averaged to produce the summary score. Note that each of the task percent correct scores is weighted equally to provide the summary score.

TABLE 28: EXAMPLE OF EGRA PERCENT CORRECT AND SUMMARY SCORES

Task (Subtest) Maximum Score Raw Score % Correct Score

1. Phoneme Isolation 10 6 60%

3. Expressive Vocabulary 10 5 50%

4b. Reading Comprehension (Grade 2-level) 5 3 60%

5. Listening Comprehension 3 3 100%

6b.Reading Comprehension (Grade 4-level) 5 4 80%

7b. Passage comprehension 5 2 40%

Reading Summary Score -- -- 55.0%

An example of timed task scores (adjusted) is provided below for the five fluency tasks. The formula explained above is used (timed task score = raw score x 60 seconds/seconds used).

TABLE 29: EXAMPLE OF EGRA TIMED TASK SCORES

Task (Subtest) Raw Score Seconds Used Timed Task Score

2. Non-word reading 25 40 37.5

4a or 6a. Passage reading 50 40 75.0


OCTOBER 2015 55

EGMA Score Ranges and Calculation

TABLE 30: EGMA GRADE 3 SCORE RANGES AND CALCULATIONS


Calculation

1. Number Identification up to 999 20 numbers (timed) >100 Percent correct numbers per minute

2.Number Discrimination up to 999

10 questions (untimed)

0-100 Percent correct of numbers

3. Missing Number up to 999 10 numbers (untimed)


4. Addition level 1 within 10 20 problems (timed) >100 Number correct per minute

5. Subtraction level 1 within 10 20 problems (timed) >100 Number correct per minute

6. Word Problems 6 Problems (untimed)

0-100 Percent correct of answers

Mathematics Summary Score 6 tasks 0-100 Average of percent correct

TABLE 31: EGMA GRADE 5 SCORE RANGES AND CALCULATIONS


Calculation

1. Number Identification up to 999

20 numbers (timed) >100 Percent correct numbers per minute

2.Number Discrimination up to 999

10 numbers (untimed)


3. Missing Number up to 999 10 numbers (untimed)


4. Addition level 1 within 10 20 problems (timed) >100 Number correct per minute

5. Subtraction level 1 within 10 20 problems (timed) >100 Number correct per minute

6. Word Problems 6 Problems (untimed)

0-100 Percent correct of answers

7. Multiplication up to 2 digits 20 problems (timed) >100 Number correct per minute

Mathematics Summary Score 7 tasks 0-100 Average of percent correct

An example of percent correct scores for each of the tasks and as a summary score is provided below. The raw score is divided by the maximum score (the highest score possible in the score range) to produce the percent correct score for each task. Then, the task scores are averaged to produce the summary score. Note that each of the task percent correct scores is weighted equally to provide the summary score.


OCTOBER 2015 56

TABLE 32: EXAMPLE OF EGMA PERCENT CORRECT AND SUMMARY SCORES

Task (Subtest) Maximum Score Raw Score % Correct Score

1. Number Identification up to 999 20 6 30.0%

2.Number Discrimination up to 999 10 8 80.0%

3. Missing Number up to 999 10 5 50.0%

4. Addition level 1 within 10 20 42 42.0%

5. Subtraction level 1 within 10 20 14 70.0%

6. Word Problems 6 3 50.0%

7. Multiplication up to 2 digits 20 18 90.0%

Mathematics Summary Score -- -- 55.0%

An example of timed task scores (adjusted) is provided below for the five timed accuracy tasks. The formula explained above is used (timed task score = raw score x 60 seconds/seconds used).

TABLE 33: EXAMPLE OF EGMA TIMED TASK SCORES

Task (Subtest) Raw Score Seconds Used Timed Task Score

1. Number Identification 25 40 37.5

4. Addition Level 1 within 10 50 40 75.0


OCTOBER 2015 57

Annex 3: EGRA Item Statistics by Grade

Table 34 presents statistics for each EGRA item on the untimed tasks. The untimed tasks each have multiple items. For instance, task 1 (phoneme isolation) has ten items, and each has item statistics. Note that the timed tasks are lists of letters, sounds, and words, i.e., not items, so it is not necessary to calculate item statistics for them. Task statistics are presented in the body of the report. These item statistics are calculated in the same way. They show the difficulty and quality of the items. Recall that when constructing a test, we strive for tasks and items that have difficulty values (p-values) that are spread across the range from about 0.1 to 0.90. For Sukkur, the difficulty values ranged from 0.04 to 0.77, indicating a strong range of item difficulties. At Grade 3, total of 17 out of the 18 items had item-total correlations of at least 0.20. At Grade 5, all 23 items surpassed that mark indicating high quality items.

TABLE 34: COMPLETE EGRA ITEM STATISTICS BY GRADE FOR SUKKUR

For the math assessment the difficulty values ranged from 0.07 to 0.93, indicating a very strong range of item difficulties. All items had item-total correlations of at least 0.20 for both grades indicating high quality items.

Task (Subtest) Item Grade 3 Grade 5

P-Value Item-Total P-Value Item-Total

1. Phoneme Isolation

Q1 .08 .72 .12 .77

Q2 .10 .63 .09 .62

Q3 .07 .71 .08 .65

Q4 .07 .65 .10 .68

Q5 .12 .62 .12 .74

Q6 .05 .58 .08 .72

Q7 .05 .65 .09 .72

Q8 .05 .62 .06 .53

Q9 .05 .71 .10 .76

Q10 .04 .52 .04 .46

2. Non-word reading (decoding) -- -- -- -- --

3 Expressive Vocabulary -- -- -- -- --

4. Passage Reading

(grade 2 level)

Q1 .17 .71 .42 .75

Q2 .21 .64 .46 .72

Q3 .10 .60 .23 .57

Q4 .18 .71 .44 .76

Q5 .07 .48 .21 .55

5. Reading Comprehension

Q1 .22 .18 .33 .47

Q2 .72 .28 .86 .35

Q3 .65 .29 .74 .44

6. Passage Reading

(grade 4 level)

Q1 -- -- .42 .62

Q2 -- -- .51 .71

Q3 -- -- .45 .74

Q4 -- -- .27 .58

Q5 -- -- .28 .60


OCTOBER 2015 58

TABLE 35: COMPLETE EGMA ITEM STATISTICS BY GRADE FOR SUKKUR

Task (Subtest) Item Grade 3 Grade 5

P-Value Item-Total P-Value Item-Total

1. Number Identification -- -- -- -- --

2. Number Discrimination

Task 2.3

Q1 .86 .46 .95 .42

Q2 .57 .61 .78 .59

Q3 .68 .69 .89 .65

Q4 .72 .64 .90 .60

Q5 .54 .71 .76 .68

Q6 .48 .72 .76 .72

Q7 .36 .57 .53 .47

Q8 .41 .75 .63 .64

Q9 .45 .76 .70 .73

Q10 .46 .79 .73 .72

3. Missing Numbers

Q1 .51 .28 .61 .32

Q2 .56 .49 .75 .49

Q3 .32 .55 .47 .54

Q4 .36 .58 .47 .56

Q5 .18 .52 .21 .48

Q6 .34 .52 .56 .50

Q7 .14 .53 .18 .46

Q8 .13 .57 .19 .50

Q9 .14 .57 .30 .57

Q10 .07 .48 .06 .30

4. Addition, Level 1 -- -- -- -- --

5. Subtraction, Level 1 -- -- -- -- --

6. Word Problems

Q1 .86 .30 .93 .29

Q2 .60 .42 .79 .44

Q3 .32 .32 .50 .33

Q4 .56 .21 .68 .21

Q5 .38 .42 .62 .38

Q6 .41 .44 .64 .45

7. Simple Multiplication -- -- -- -- --


OCTOBER 2015 59

Annex 4: Box Plots for Phonics and Reading-rate Fluency Tasks

Recently, greater emphasis has been placed on the results of the fluency (timed) tasks over other tasks. In addition to the descriptive statistics presented in the body of the report, we show box plots for the different fluency tasks. Widely used since their development in the 1960s, box plots are a convenient way for graphically presenting numerical data. Box plots have two characteristics: 1) central tendency (i.e., the median, or the middle score in the data) and 2) variation (i.e., the range, with scores grouped by quartile). The boxes (which are actually rectangles) represent the two middle quartiles of the scores and the “whiskers” represent the upper and lower quartiles. The small circles on the ends of the whiskers represent outliers. Figure 19 provides a more detailed explanation for interpreting box plots.

FIGURE 19: EXPLANATION OF BOX PLOTS

Median

The median (middle quartile) marks the mid-point of the data and is shown by the line that divides the box into two parts. Half the scores are greater than or equal to this value and half are less.

Upper quartile Seventy-five percent of the scores fall below the upper quartile.

Lower quartile Twenty-five percent of scores fall below the lower quartile.


OCTOBER 2015 60

Box plots are presented below for reading fluency and mathematics-fact fluency for both grades. At Grade 3, non-word reading and Grade level 2 passage reading fluency are presented. At grade 5 an additional task of Grade 4-level reading was generated. The EGMA mathematics fluency scores follow. At Grade 3 number identification, addition and subtraction are plotted. Simple multiplication was also included at Grade 5.

Reading Fluency Box Plots

The fluency rates for non-word reading (decoding, the far left box plot) were much lower for both grades. The decoding variation (i.e., the range of scores, without outliers) for both was less than 30 words per minute. The reading fluency variation was 40 words at Grade 3. At Grade 5 there was a much larger reading rate variation, over 80 words per minute, indicating a wider range of reading fluency abilities.

FIGURE 20: PHONICS AND READING-

RATE FLUENCY BOX PLOTS FOR

GRADE 3

FIGURE 21: PHONICS AND READING-

RATE FLUENCY BOX PLOTS FOR

GRADE 5


OCTOBER 2015 61

Mathematics Fluency Box Plots

Math-fact fluency scores are plotted below. The box plots infer that most Sukkur students solve fewer than 20 math problems per minute. A few students (third quartile) can solve over 20 at Grade 3 and over 75 problems at Grade 5. At Grade 3 addition and subtraction problem solving rates are much lower and indicate that many students solve basic math problems slowly. At Grade 5, box plots are relatively higher inferring faster problem solving rates. Students were quicker in identifying numbers. Box plot scores for addition and subtraction show over 25 percent of students zero or very low rates because the lower quartile line is not visible.

FIGURE 22: MATHEMATICS FACT-RATE

FLUENCY BOX PLOTS FOR GRADE 3

FIGURE 23: MATHEMATICS FACT-RATE

FLUENCY BOX PLOTS FOR GRADE 5


OCTOBER 2015 62

Annex 5: Questionnaire Results

Student Questionnaire Tables

TABLE 36: DISTRIBUTION OF STUDENTS BY PRIMARY LANGUAGE SPOKEN AT

HOME

Language Grade 3 Grade 5

Number Percent Number Percent

Sindhi 720 89% 675 87%

Siraeki 27 3% 37 5%

Balochi 16 2% 23 3%

Punjabi 3 0% 5 1%

Urdu 7 1% 10 1%

Other 32 4% 0 0%

Missing 3 0% 23 3%

Total 810 100% 773 100%

TABLE 37: AVERAGE EGRA/EGMA SCORES BY STUDENTS SPEAKING THE SAME

LANGUAGE AT HOME AS THE TEST

Home Language Same as

Test

Grade 3 Grade 5

Number

Avg

EGRA Sum

Score

Avg

EGMA Sum

Score

Number

Avg

EGRA Sum

Score

Avg

EGMA Sum

Score

Yes 720 (89%) 31.1% 43.1% 675 (87%) 44.9% 61.4%

No 90 30.7% 39.1% 98 46.4% 67.4%*

Total 810 31.15% 42.6% 773 45.1% 62.2%


OCTOBER 2015 63

TABLE 38: AVERAGE EGRA/EGMA SCORES BY READING MATERIAL AT HOME

Reading Material

Type

Grade 3 Grade 5

Number

Avg EGRA Sum

Score

Avg EGMA

Sum

Score

Number

Avg EGRA Sum

Score

Avg

EGMA Sum

Score

Newspapers Yes 332 (41%) 32.8%* 46.0%* 319 (41%) 46.1% 61.9%

No 478 30.0% 40.4% 454 44.4% 63.4%

Magazines Yes 33 (4%) 31.0 40.1% 51 (7%) 43.2% 59.6%

No 777 31.1% 42.7% 722 45.2% 62.3%

Books Yes 270 (33%) 32.7% 43.4% 191 (25%) 43.9% 61.9%

No 540 30.2% 42.2% 582 45.5% 62.2%

Nothing Yes 247 (30%) 28.2% 38.2% 295 (38%) 44.0% 63.1%

No 563 32.3%* 44.5%* 478 45.8% 61.6%

Total 810 31.15% 42.6% 773 45.1% 62.2%

*Category summary scores were significantly higher (p < .05)

TABLE 39: PERCENTAGE OF STUDENTS BY STORIES READ ALOUD TO

STUDENT AT HOME

Stories Read Aloud to Student at Home

Grade 3 Grade 5


Yes 442 55% 366 47%

No 365 45% 403 52%

Missing 3 0% 4 1%

Total 810 100% 773 100%


OCTOBER 2015 64

TABLE 40: DISTRIBUTION OF SCORES BY PRACTICE READING STORIES ALOUD

TO SOMEONE AT HOME

Practice Reading Stories Aloud to

Someone at Home

Grade 3 Grade 5

Number % Number %

Yes 444 55% 351 45%

No 362 45% 420 54%

Missing 4 0% 2 0%

Total 810 100% 773 100%

TABLE 41: AVERAGE EGRA/EGMA SCORES BY SILENT READING AT HOME

Practice Silent Reading at Home

Grade 3 Grade 5

Number

Avg EGRA Sum

Score

Avg

EGMA Sum

Score

Number

Avg EGRA Sum

Score

Avg EGMA

Sum

Score

Yes 655 (81%) 31.9%* 42.8% 597 (77%) 45.9% 63.0%

No 153 27.5% 41.7% 174 42.3% 59.6%

Missing 2 -- -- 2 -- --

Total 810 31.15% 42.6% 773 45.1% 62.2%

*Summary scores were significantly higher (p < .05)

TABLE 42: AVERAGE EGRA/EGMA SCORES BY READ QURAN AT HOME

Read Quran at Home

Grade 3 Grade 5

Number

Avg EGRA Sum

Score

Avg EGMA

Sum

Score

Number

Avg EGRA Sum

Score

Avg EGMA

Sum

Score

Yes 714 (88%) 31.6%* 42.2% 698 (90%) 45.9%* 62.0%

No 74 26.8% 45.7% 64 35.7% 62.3%

Missing 22 -- -- 11 -- --

Total 810 31.15% 42.6% 773 45.1% 62.2%



OCTOBER 2015 65

TABLE 43: AVERAGE EGRA/EGMA SCORES BY STUDY OR COMPLETE

CLASSWORK WITH CLASSMATES

Study or Do Class-work

with Classmates

Grade 3 Grade 5

Number

Avg EGRA Sum

Score

Avg EGMA

Sum

Score

Number

Avg EGRA Sum

Score

Avg EGMA

Sum

Score

Yes 618 (76%) 31.2% 43.0% 464 (60%) 46.9%* 64.3%*

No 192 30.6% 41.4% 304 42.5% 59.0%

Missing 0 -- -- 5 -- --

Total 810 31.15% 42.6% 773 45.1% 62.2%


TABLE 44: AVERAGE EGRA/EGMA SCORES BY STUDENTS MISSING SCHOOL

DAYS DURING THE PREVIOUS WEEK

Missed Any Days of School in the Previous

Week

Grade 3 Grade 5

Number

Avg EGRA Sum

Score

Avg EGMA

Sum

Score

Number

Avg EGRA Sum

Score

Avg EGMA

Sum

Score

Yes 349 (43%) 30.0% 42.1% 331 (43%) 43.1% 60.3

No 461 31.9% 43.0% 441 46.5%* 63.6%*

Missing 0 -- -- 1 -- --

Total 810 31.15% 42.6% 773 45.1% 62.2%


TABLE 45: AVERAGE EGRA/EGMA SCORES BY PARENT’S KNOWLEDGE OF

SCHOOL PERFORMANCE

Parents Knowledge of School

Performance

Grade 3 Grade 5

Number

Avg EGRA Sum

Score

Avg EGMA

Sum

Score

Number

Avg EGRA Sum

Score

Avg

EGMA Sum

Score

Yes 385 (48%) 28.8% 40.2% 412 (53%) 44.6% 62.1%

No 419 33.1%* 44.8%* 343 45.9% 62.4%

Missing 6 -- -- 18 -- --

Total 810 31.15% 42.6% 773 45.1% 62.2%



OCTOBER 2015 66

TABLE 46: AVERAGE EGRA/EGMA SCORES BY STUDENTS REPORTING

WORKING BEFORE OR AFTER SCHOOL

Work Before or After School

Grade 3 Grade 5

Number

Avg

EGRA Sum

Score

Avg

EGMA Sum

Score

Number

Avg EGRA Sum

Score

Avg EGMA

Sum

Score

Yes 564 (70%) 32.2%* 43.3% 647 (80%) 45.2% 62.0%

No 239 28.0% 40.5% 125 45.0% 62.9%

Missing 7 -- -- 1 -- --

Total 810 31.15% 42.6% 773 45.1% 62.2%

TABLE 47: DISTRIBUTION OF SCHOOLS BY STUDENTS WHO WATCH

TELEVISION AT HOME

Watch Television Grade 3 Grade 5

Number % Number %

Yes 560 69% 564 73%

No 250 31% 209 27%

Missing 0 0% 0 0%

Total 810 100% 773 100%

TABLE 48: DISTRIBUTION OF SCHOOLS BY STUDENTS LISTENING TO THE

RADIO AT HOME

Listen to the Radio at Home

Grade 3 Grade 5

Number % Number %

Yes 313 39% 275 39%

No 496 61% 498 61%

Missing 1 0% 0 0%

Total 810 100% 773 100%


OCTOBER 2015 67

TABLE 49: DISTRIBUTION OF STUDENTS BY COMPUTER ACCESS AT HOME

Computer Access at Home

Grade 3 Grade 5


Yes 132 16% 117 15%

No 676 83% 654 85%

Missing 2 0% 2 0%

Total 810 100% 773 100%

TABLE 50: DISTRIBUTION OF STUDENTS BY TYPE OF VEHICLES AT HOME

Vehicle Grade 3 Grade 5


Bicycle 177 22% 205 27%

Motorcycle 511 63% 507 66%

Car 54 7% 63 8%

None 162 20% 144 19%

Total 810 100% 773 100%

Teacher Questionnaire Tables

TABLE 51: DISTRIBUTION OF TEACHERS’ ACADEMIC QUALIFICATIONS

Degree Grade 3 Grade 5


M.A./M.Sc/M.Phil. 21 32% 31 48%

B.A./B.Sc. 29 45% 22 34%

F.A/F.Sc. 6 9% 7 11%

Matric 7 11% 4 6%

Other 2 3% 1 2%

Total 65 100% 65 100%


OCTOBER 2015 68

TABLE 52: DISTRIBUTION OF TEACHERS’ PROFESSIONAL QUALIFICATIONS

Professional Qualification

Grade 3 Grade 5


M.Ed/M.A. Education 16 25% 22 34%

B.Ed. 23 35% 27 42%

ADE 0 0% 0 0%

C.T. 2 3% 0 0%

P.T.C. 20 31% 16 25%

Other 4 6% 0 0%

Total 65 100% 65 100%

TABLE 53: AVERAGE EGRA/EGMA SCORES BY TEACHING MULTI-GRADE

CLASSES

Teaching Multi-Grade Classroom

Grade 3 Grade 5

Number

EGRA Sum

Score

EGMA Sum

Score

Number

EGRA Sum

Score

EGMA Sum

Score

Yes 42 (65%) 31.8% 42.2% 46 (71%) 43.2% 60.7%

No 23 30.1% 43.6% 19 46.7% 66.9%*

Total 65 31.1% 42.6% 65 45.1% 62.2%


TABLE 54: DISTRIBUTION OF TEACHERS USING BOOKS OTHER THAN

TEXTBOOKS IN THE CLASSROOM

Using Books Other Than Textbooks in

Classroom

Grade 3 Grade 5

Number % Number %

Yes 18 28% 19 29%

No 46 71% 43 66%

Missing 1 2% 3 5%

Total 65 100% 65 100%


OCTOBER 2015 69

TABLE 55: DISTRIBUTION OF CLASSROOM TOOLS USED BY TEACHERS

Classroom Tools Grade 3 Grade 5

Number % Number %

Blackboard 63 97% 3 5%

Chalk 63 97% 62 95%

Whiteboard 4 6% 3 5%

Marker 9 14% 15 23%

Writing Paper 26 40% 32 49%

Textbooks 60 92% 61 94%

Books/Storybooks 15 23% 17 26%

Posters 6 9% 13 20%

Flashcards 10 15% 13 20%

Charts 37 57% 38 58%

Math Instruments 9 14% 14 22%

Counters 1 2% 5 8%

Enough Textbooks 59 91% 60 92%

Do you have language or reading teacher guides

34 52% 32 49%

Do you have mathematics teacher guides

29 45% 31 48%

Total 65 100% 65 100%

TABLE 56: DISTRIBUTION OF TEACHING PRACTICES

Teaching Practices Grade 3 Grade 5


Head Teacher encourages use of a variety of class activities

56 86% 62 95%

Give extra time to slow learners after class 55 85% 49 75%

Conduct individual meetings with parents 62 95% 58 89%

Total 65 100% 65 100%


OCTOBER 2015 70

TABLE 57: METHODS BY WHICH TEACHERS’ UDATE THEIR KNOWLEDGE OF

TEACHING

Updating Knowledge Grade 3 Grade 5


Reading books 62 95% 62 95%

Going on the Internet 14 22% 16 25%

Through peer support 24 37% 26 40%

Attending seminars or educational conferences 15 23% 20 31%

In-service training 39 60% 41 63%

Total 65 100% 65 100%

Head Teacher Questionnaire Tables

TABLE 58: DISTRIBUTION OF HEAD TEACHERS’ ACADEMIC QUALIFICATIONS

Degree Head Teacher

Number Percent

M.A./M.Sc/M.Phil. 29 41%

B.A./B.Sc. 28 40%

F.A/F.Sc. 6 9%

Matric 7 10%

Other 0 0%

Total 70 100%

TABLE 59: DISTRIBUTION OF HEAD TEACHERS’ PROFESSIONAL

QUALIFICATIONS

Certification Head Teacher

Number %

M.Ed/M.A. Education 19 27%

B.Ed. 28 40%

C.T. 1 1%

P.T.C. 22 31%

Total 70 100%


OCTOBER 2015 71

TABLE 60: DISTRIBUTION OF HEAD TEACHERS TEACHING CLASSES

Teaching Class Head Teacher

Number Percent

Yes 55 79%

No 14 20%

Total 70 100%

TABLE 61: DISTRIBUTION OF HEAD TEACHERS WHO RECEIVED TRAINING FOR

IMPLEMENTING READING PROGRAMS

Training in Implementing Reading Programs

Head Teacher

Number %

Yes 41 59%

No 27 39%

Missing 2 3%

Total 70 100%

TABLE 62: DISTRIBUTION OF HEAD TEACHERS WHO RECEIVED TRAINING FOR

IMPLEMENTING MATHEMATICS PROGRAMS

Training in Implementing Math Programs

Head Teacher

Number %

Yes 28 40%

No 37 53%

Missing 5 7%

Total 70 100%

TABLE 63: DISTRIBUTION OF SCHOOLS HAVING A TIMETABLE

School Timetable Head Teacher

Number %

Yes 46 66%

No 24 34%

Total 70 100%


OCTOBER 2015 72

TABLE 64: AVERAGE EGRA/EGMA SCORES BY SCHOOL TIMETABLE

Timetable

Grade 3 Grade 5

Number

Avg EGRA Sum

Score

Avg EGMA Sum

Score

Avg EGRA Sum

Score

Avg EGMA Sum

Score

Yes 49 (70%) 29.8% 36.5% 44.4%* 58.7%*

No 21 25.1% 31.8% 35.6% 42.4%

Total 70 28.8% 35.6% 42.0% 54.6%

TABLE 65: AVERAGE EGRA/EGMA SCORES BY INCLUDING READING IN THE

TIMETABLE

Timetable

Grade 3 Grade 5

Number

Avg EGRA Sum

Score

Avg EGMA Sum

Score

Avg EGRA Sum

Score

Avg EGMA Sum

Score

Yes 35 (50%) 40.4% 50.0% 57.8%* 72.0%*

No 14 35.9% 41.9% 46.1% 58.0%

Omit (no timetable) 21 -- -- -- --

Total 70 37.7% 46.8% 52.3% 65.4%

TABLE 66: DISTRIBUTION OF SCHOOLS BY INCLUDING MATH IN THE

TIMETABLE

Math Period Head Teacher

Number %

Yes 44 63%

No 1 1%

Omit (no timetable) 30 43%

Total 70 100%


OCTOBER 2015 73

TABLE 67: DISTRIBUTION OF HEAD TEACHERS SUPPORTING READING

INSTRUCTION

Support Reading Instruction

Head Teacher

Number %

Yes 60 86%

No 10 14%

Total 70 100.0%

TABLE 68: DISTRIBUTION OF HEAD TEACHERS SUPPORTING MATHEMATICS

INSTRUCTION

Support Mathematics Instruction

Head Teacher

Number %

Yes 57 81%

No 13 19%

Total 70 100.0%

TABLE 69: DISTRIBUTION OF HEAD TEACHER’S KNOWLEDGE ABOUT

STUDENT PROGRESS

Progress Method Head Teacher

Number %

Classroom observation 42 60%

Monitor student test results 29 41%

Evaluate students orally myself 45 64%

Review students’ assignments 27 39%

Teacher-provided progress reports 15 21%

Review End-of-Year Test Results 43 61%

Union Council supervisors evaluate students

20 29%

Total 70 100%


OCTOBER 2015 74

TABLE 70: AVERAGE EGRA/EGMA SCORES BY HEAD TEACHERS’ OBSERVING

TEACHERS’ USE OF LESSON PLANS

Observe Teachers’ Use of Lesson

Plans

Grade 3 Grade 5

Number

Avg

EGRA Sum

Score

Avg

EGMA Sum

Score

Avg

EGRA Sum

Score

Avg

EGMA Sum

Score

Yes 53 (76%) 31.3% 42.8% 46.2%* 62.7%

No 16 29.3% 41.8% 39.9% 59.7%

Total 70 31.15% 42.6% 45.1% 62.2%


TABLE 71: DISTRIBUTION OF THE PERSON RESPONSIBLE FOR OBSERVING

TEACHERS IN THE CLASSROOM

Head Teacher

Number %

Head Teacher 31 44%

Deputy Head Teacher 8 11%

No one 16 23%

Total 70 100%

TABLE 72: DISTRIBUTION OF SMC DECISION MAKING

AUTHORITY/RESPONSIBILITY

SMC Responsibility Head Teacher

Number %

Management Problems 50 71%

Students’ problems/solutions 46 66%

School improvement planning 56 80%

Review school improvement progress 50 71%

Review Finances 40 57%

Participate in SMC-fund spending 48 69%

Manage infrastructure/equipment 45 64%

Discuss school curriculum 24 34%

Raise funds 21 30%

Manage procurement/distribution of textbooks

34 49%


OCTOBER 2015 75

TABLE 73: DISTRIBUTION OF FACILITY IMPROVEMENTS

Type of Student Observation Head Teacher

Number Percent

Clean water 55 79%

Electricity 50 71%

Toilet facilities 53 76%

Gender separate toilet facilities at mixed-gender schools

32 46%

Library 3 4%

USAID SINDH READING PROGRAM EARLY GRADE READING …The tasks for both the Grade 3 and Grade 5 EGMAs...

Documents

Transcript of USAID SINDH READING PROGRAM EARLY GRADE READING …The tasks for both the Grade 3 and Grade 5 EGMAs...