Educational Assessment

Sandra MacKinnon, Director

Assessment and Evaluation Branch (Anglophone Division)

Department of Education and Early Childhood Development

Atlantic Education International

July 22, 2014

0

0 .2

0 .4

0 .6

0 .8

1 .0

-3 -2 -1 0 1 2 3

b

Ab i l i ty

Pro

bab

ilit

y

Item Characteristic Curve: ITEM 0001

a = 0 .5 3 9 b = 0 .2 0 2

1. Educational Assessment in General

~Balanced Assessment

Large-Scale Assessment

2. The New Brunswick Provincial Assessment Program

~Mission

~Provincial Assessments (mandate, operational standards,

processes)

•English Second Language Assessment / Reassessment Reading Writing Oral Proficiency Listening

TOPICS

© 2005, Dynamic Measurement Group 3

GPS On Where are we?

You are Here

Port

Where do we want to be ?

How do we get there?

How are we doing?

ActualCourse

DesiredCourse

Assessment is the GPS for Educators

10

20

30

40

Dec.Scores

Feb.Scores

Jan.Scores

M archScores

AprilScores

MayScores

JuneScores

60

50

Rea

ding

Aimline

Balanced Assessment

• No single assessment approach is 'better' than another as each serves a slightly different role.

• All types of assessment must be planned with its purpose in mind.

Standardized testing, particularly, is justified as one resource to aid learning and should not be mistaken for a comprehensive verdict on learning or school quality. ~ Peter Grimmett & Paul Shaker

 

New Brunswick Assessment Program (NBAP)

Mission StatementThe mission for the Provincial Assessment and Evaluation Branch is to inform all stakeholders of student achievement by constructing valid, educationally beneficial, evaluations at critical points in the system. The purpose of the New Brunswick Assessment Program (NBAP) is to: determine if students demonstrate the skills and concepts outlined in New Brunswick curriculum documents; report provincially on how well students have achieved provincial standards at given points in their schooling; assist schools, districts, and the province in monitoring student learning and; coordinate the administration of national and international assessments.

New Brunswick Assessment Program

Operational Standards Processes How items are chosen

Standards for Educational and Psychological Testing

The Standards for Educational and Psychological Testing, 2008 edition are a collaborative publication from the American Educational Research Association, the American Psychological Association and the National Council on Measurement in Education. The intent of the Standards is to promote the sound and ethical use of tests and to provide a basis for evaluating the quality of testing practices.

The New Brunswick Assessment and Evaluation Branch adheres to the Standards when developing, administering, scoring and analysing large scale assessments within the scope of our mandate. 

Operational Standards

Standards for Educational and Psychological Testing

The Assessment Cycle

There are four (4) stages within the overall Assessment Cycle for each of the 13 assessments; the (1) Development Stage, (2) Administration Stage, (3) Data Stage and the (4) Communication Stage.

Within each of the stages, there are several phases which require the expertise from a variety of staff.

Each of these stages require strict adherence to the Standards for Educational and Psychological Testing and the Principles for Fair Student Assessment in Canada.. These standards and principles will be referenced where appropriate.

Processes

0

0 .2

0 .4

0 .6

0 .8

1 .0

-3 -2 -1 0 1 2 3

b

Ab i l i ty

Pro

bab

ilit

y

Item Characteristic Curve: ITEM0001

a = 0 .5 3 9 b = 0 .2 0 2

1 - 7

8 - 14

15 - 21

22 - 28

29 - 35

36 - 42

43 - 48

Matrix Plot of Item Characteristic Curves for Book A

1 - 7

8 - 14

15 - 21

22 - 28

29 - 35

36 - 42

43 - 48

Matrix Plot of Item Characteristic Curves for Book A

English Second Language Assessment / Reassessment

ESLA/R

AGENDA - General

• ESLA Overview– 4 components– Summary of the data– What constitutes “Successful”– What is the standard for passing ESLA?– What are some important points that should be kept

in mind when writing the test?

Focus Areas

• Writing– How do we mark the writing?– Exemplars and Rationales

• Oral Proficiency Interviews– How long does the interview take?– Common topics and criteria for scoring– OPI Exemplars and Rationales

ESLA AT A GLANCE; GENERAL OVERVIEW

Test Specifications - OverviewEnglish Second Language Assessment (Grade 11)English Second Language Reassessment (Grades 12)

The English Second Language Assessment/Reassessment (ESLA/R) is an assessment of students’ English literacy abilities in all four language skills.

The ESLA/R is a requirement for students attending the Concord Colleges of Sino-Canada who wish to obtain a New Brunswick High School Diploma.

The ESLA/R is a criteria - standards based assessment where students are required to meet targets appropriate to two years of non-immersion English language instruction. Students in the Concord Colleges apply for acceptance to post-secondary English institutions.

English Second Language Assessment / Reassessment• 4 Components• Resembles the format of TOEFL except at a

high school level

ORAL

READING Weighted Equally

WRITING

LISTENING

OVERALL CALCULATION OF ESLA/ROral = H2*10

(Numeric oral value multiplied by 10)

Listening = L2*2.50

(Listening score multiplied by 2.50)

Overall score = SUM(I2,K2,M2,P2)/4 .

(overall score adding the four weighted scores, then dividing by 4.

Oral Ratings: ('U'=0) ('N'=3) ('B'=4) ('B+'=5) ('I'=6) ('I+'=7) ('A'=8) ('A+'=9) ('S'=10)

Writing Ratings: (DNW=*), ( BAA=1), (AA=2), and (SA=3)

cut point 43

0 - 43 Unsuccessful

ESLA/R READING

Reading Performance Levels ESLA

Reading Performance Levels

Reading Test Specifications

Book A --- Reading --- 90 minutes

7-9 Reading Passages

45-50 Number of items

Responding to Reading

Reading; Test difficulty and equating

• Unlike Writing and Oral Interviews, reading and listening are scored according to ‘cut points’.

• As the reading passages and items change, we need to ensure that the overall difficulty of the reading assessment is the same year over year – so that one cohort is being assessed on the same difficulty level as all other cohorts from previous administrations.

• So…how do we do this?

Test equating and cut point setting

ESLA/R WRITING

Performance Levels in Writing

Writing Test Specifications

Time allotment: Exercise I 60 minutesExercise II 60 minutes

The Writing component should be administered as either the first or last component. For Writing, students will answer directly in Booklet C. The Writing component consists of two writing activities: Exercise I and Exercise II.Writing prompts are provided on a separate sheet.

Marking the Writing Piece…

Writing is assessed using the Writing Achievement Standards – ESLA.

In order for a piece of writing to be deemed Appropriate Achievement or Strong Achievement it must possess all of the writing traits listed for that category.

ESLA/R LISTENING

Listening Stats

Listening Stats

Listening Test Specs

Time allotment: 30 minutes

 

Each booklet contains an information label with the student’s Chinese and English names as well as an identification number. Test administrators must ensure that each student receives the appropriate booklet.

 

Test administrators will read the items to the students using Instructions for Administrators of the Listening Assessment.

Students will respond to all Listening items directly in their booklet by circling either letter A, B, C, or D – whichever corresponds with the best response.

Book B - Listening consists of two components:

Part I - Answering Questions (Items 1-20)

Part II - Understanding Statements (Items 21-40)

Helpful hints for listening…

• Practice the “who, what, when, where, why” line of questions with the appropriate response.

• Warm-up activity to begin each class.

ESLA/R ORAL

Oral Interview Stats

OPI’s

Time allotment: 15 - 20 minutes

 

Interviews will be scheduled at twenty-minute intervals: the interview itself will be 10 - 15 minutes in duration. The interviewer will use the remaining time to determine and record the student's oral proficiency level and to prepare for the next interview.

FOCUS AREA; ORAL INTERVIEWS AND WRITING

OR

AL

INT

ER

VIE

W

ST

AN

DA

RD

S

LEVEL CRITERIA UNRATEABLE Can provide no intelligible answers including name

*Talks in own native language

NOVICE * Repeat/rephrase questions often Speaks in isolated words and phrases Not able to participate in conversations Some memorized material and ability to use it * Very limited vocabulary

BASIC

Survival Interviewer works hard to keep a conversation going

Can survive in a second language Can say a lot of simple things; buy basic food items, provide personal information, standard social exchange, school directions * Can get into, through and out of a situation Reactive, answering but not asking many questions – will ask if prompted Fluency slow, strained, except for routine expressions Pronunciation errors frequent / intelligible to someone who is used to ESL

BASIC PLUS

Some give and take in conversations

Can participate in short conversations, descriptions, narrations Shows more independence, spontaneity, asks questions but may require prompts to do so

INTERMEDIATE

More reciprocity - The interview begins to have a more conversational feel.

Can provide information Provide autobiographical information in some detail Give directions, accurate instructions, in area of personal experience or interest Describe / tell using past, present and future tenses Speak confidently, hesitates, uses paraphrases and fillers Pronunciation intelligible, often faulty Vocabulary sufficient to speak confidently

WR

ITIN

G S

TA

ND

AR

DS

ESLA Writing Exemplars

Package 1

Exemplars with rationales attached

QUESTIONS?