A Common Testing Framework for Computerized Spoken ......You will hear two brief stories. After each...

37
Ordinate Corporation ALTE, Berlin, Germany May 2005 1 A Common Testing Framework for Computerized Spoken Language Tests in Multiple Languages 19 May 2005 Jennifer Balogh Ordinate Corporation, Menlo Park, California

Transcript of A Common Testing Framework for Computerized Spoken ......You will hear two brief stories. After each...

Page 1: A Common Testing Framework for Computerized Spoken ......You will hear two brief stories. After each story, you will have 30 seconds to retell it in Spanish as best you can. Try to

Ordinate Corporation ALTE, Berlin, Germany May 2005 1

A Common Testing Framework for Computerized Spoken Language

Tests in Multiple Languages

19 May 2005

Jennifer BaloghOrdinate Corporation, Menlo Park, California

Page 2: A Common Testing Framework for Computerized Spoken ......You will hear two brief stories. After each story, you will have 30 seconds to retell it in Spanish as best you can. Try to

Ordinate Corporation ALTE, Berlin, Germany May 2005 2

Oral Proficiency Interviews

Interviewer 2 Candidate

Here is a question

Interviewer 1

Page 3: A Common Testing Framework for Computerized Spoken ......You will hear two brief stories. After each story, you will have 30 seconds to retell it in Spanish as best you can. Try to

Ordinate Corporation ALTE, Berlin, Germany May 2005 3

Oral Proficiency Interviews

Interviewer 2 Candidate

Here is an answer

Interviewer 1

Page 4: A Common Testing Framework for Computerized Spoken ......You will hear two brief stories. After each story, you will have 30 seconds to retell it in Spanish as best you can. Try to

Ordinate Corporation ALTE, Berlin, Germany May 2005 4

Oral Proficiency Interviews

Interviewer 2 Candidate

Here is an answer

Interviewer 1

Page 5: A Common Testing Framework for Computerized Spoken ......You will hear two brief stories. After each story, you will have 30 seconds to retell it in Spanish as best you can. Try to

Ordinate Corporation ALTE, Berlin, Germany May 2005 5

Oral Proficiency Interviews

Interviewer 2 Candidate

Here is an answer

Interviewer 1

Page 6: A Common Testing Framework for Computerized Spoken ......You will hear two brief stories. After each story, you will have 30 seconds to retell it in Spanish as best you can. Try to

Ordinate Corporation ALTE, Berlin, Germany May 2005 6

Oral Proficiency Interviews

Interviewer 2 Candidate

Here is an answer

Interviewer 1

Page 7: A Common Testing Framework for Computerized Spoken ......You will hear two brief stories. After each story, you will have 30 seconds to retell it in Spanish as best you can. Try to

Ordinate Corporation ALTE, Berlin, Germany May 2005 7

Goals• Create reliable & valid spoken language tests

• Automatically administered• Automatically scored

• Use a common framework for all tests• Multiple languages• Quick development

Page 8: A Common Testing Framework for Computerized Spoken ......You will hear two brief stories. After each story, you will have 30 seconds to retell it in Spanish as best you can. Try to

Ordinate Corporation ALTE, Berlin, Germany May 2005 8

Automatic Administration

SST Call: 1-800-444-7277 or 1-650-328-0336

Test Identification Number

1234 5678 Introduction: Thank you for calling the Ordinate testing system. Please enter your Test Identification Number on the telephone keypad. Now, please say the city and country you are calling from. Now, please follow the instructions for Parts A through G.

Part A: Reading. Please read the sentences as you are instructed.

1. Julio recibió de regalo una nueva bicicleta último modelo. 2. Iba a la universidad y al trabajo en su nueva bicicleta todos los días. 3. Un día a la salida del trabajo se dio cuenta de que se la habían robado. 4. Se quedó muy triste pensando que no le había durado mucho tiempo. 5. Habían quedado en encontrarse en el parque, al mediodía. 6. Mientras paseaban Jorge le contó sus planes y le preguntó si quería compartirlos. 7. Ella se quedó pensando un rato, luego le dijo que sí, que le acompañaría. 8. Felices, se abrazaron y decidieron ir a comer juntos al restaurante de la esquina.

Part B: Repeat. Please repeat each sentence that you hear. Example: a voice says, "Le gustaba cantar canciones románticas." and you say, "Le gustaba cantar canciones románticas."

Part C: Opposites. Now, when you hear a word, just say the opposite. Example: a voice says, "alto" and you say "bajo". a voice says, "abierto" and you say "cerrado".

Part D: Questions. Now, please just give a simple answer to the questions. Example: a voice says, "Si estuviera enfermo, ¿a quién iría a ver: a un médico o a un vendedor?" and you say, "un médico" or "a un médico".

Part E: Sentence Builds. Now, please rearrange the word groups into a sentence. Example: a voice says, "y lo vimos" ... "nos asomamos" ... "marcharse" and you say, "Nos asomamos y lo vimos marcharse."

Part F: Open Questions. You will have 30 seconds to answer each of two questions. The questions will be about family life or personal choices. Each question will be spoken twice, followed by a beep. When you hear the beep, you will have 30 seconds to answer the question. At the end of the 30 seconds, another beep will signal the end of the time you have to answer.

Part G: Story Retelling. You will hear two brief stories. After each story, you will have 30 seconds to retell it in Spanish as best you can. Try to retell as much of the story as you can in Spanish, including the situation, characters, actions and ending.

Telephone Number to the System

Test Identification Number

Page 9: A Common Testing Framework for Computerized Spoken ......You will hear two brief stories. After each story, you will have 30 seconds to retell it in Spanish as best you can. Try to

Ordinate Corporation ALTE, Berlin, Germany May 2005 9

Process flow1. Proctor downloads test from Ordinate web site and prints out

2. Candidate reviews test sheet & instructions (5min)

3. Candidate calls Ordinate – toll free number on test sheet

4. Candidate keys in unique (8-digit) Test Identification Number

5. Candidate takes test (~10min)

6. Candidate/Proctor retrieves score from Ordinate web site (within minutes after taking the test)

Page 10: A Common Testing Framework for Computerized Spoken ......You will hear two brief stories. After each story, you will have 30 seconds to retell it in Spanish as best you can. Try to

Ordinate Corporation ALTE, Berlin, Germany May 2005 10

Page 11: A Common Testing Framework for Computerized Spoken ......You will hear two brief stories. After each story, you will have 30 seconds to retell it in Spanish as best you can. Try to

Ordinate Corporation ALTE, Berlin, Germany May 2005 11

Common Testing Framework

Test Architecture Testing System Development andValidation Process

Task Type

Task A

Task B

Task C

Task D

Task E

Task F

Task G

Phone Network

Phone Network

Page 12: A Common Testing Framework for Computerized Spoken ......You will hear two brief stories. After each story, you will have 30 seconds to retell it in Spanish as best you can. Try to

Ordinate Corporation ALTE, Berlin, Germany May 2005 12

Common Testing Framework

Test Architecture Testing System Development andValidation Process

Task Type

Task A

Task B

Task C

Task D

Task E

Task F

Task G

Phone Network

Phone Network

Page 13: A Common Testing Framework for Computerized Spoken ......You will hear two brief stories. After each story, you will have 30 seconds to retell it in Spanish as best you can. Try to

Ordinate Corporation ALTE, Berlin, Germany May 2005 13

Test ConstructMeasures facility in spoken language

hear utteranceextract wordsget phrase structuredecode propositionscontextualizeinfer demand (if any)

articulate responsebuild clause structureselect lexical items construct phrasesselect registerdecide on response

Adapted from Levelt, 1989

Listen

Speak

hear utteranceextract wordsget phrase structuredecode propositionscontextualizeinfer demand (if any)

articulate responsebuild clause structureselect lexical items construct phrasesselect registerdecide on response

Adapted from Levelt, 1989

Listen

Speak

Page 14: A Common Testing Framework for Computerized Spoken ......You will hear two brief stories. After each story, you will have 30 seconds to retell it in Spanish as best you can. Try to

Ordinate Corporation ALTE, Berlin, Germany May 2005 14

Theory of Language Proficiency:Automaticity

resources

Limited understanding and ability to

respond

Page 15: A Common Testing Framework for Computerized Spoken ......You will hear two brief stories. After each story, you will have 30 seconds to retell it in Spanish as best you can. Try to

Ordinate Corporation ALTE, Berlin, Germany May 2005 15

Theory of Language Proficiency:Automaticity

resources

Better understanding and ability to

respondLanguage

model

Page 16: A Common Testing Framework for Computerized Spoken ......You will hear two brief stories. After each story, you will have 30 seconds to retell it in Spanish as best you can. Try to

Ordinate Corporation ALTE, Berlin, Germany May 2005 16

Theory of Language Proficiency:Automaticity

resources

Counsel, persuade,

advise Fluent listening and

speakingLanguage

model

Page 17: A Common Testing Framework for Computerized Spoken ......You will hear two brief stories. After each story, you will have 30 seconds to retell it in Spanish as best you can. Try to

Ordinate Corporation ALTE, Berlin, Germany May 2005 17

Test DesignTest Part Task Type ExamplePart A Read Aloud Julio había recibido de regalo una hermosa bicicleta último

modelo. Julio was given the latest model of a beautiful bicycle as a gift.

Part B Repeat Sentences “El joven camina por la calle.”The man walks along the street.

Part C Say the Opposite “alto”high

Part D Answer Short Questions

“¿Cuántas patas tiene un perro?”How many legs does a dog have?

Part E Build Sentences “te / María / ama”you / Maria / loves

Part F Answer Open Questions

“¿Prefiere usted vivir en la ciudad o en el campo? Por favor explique su elección.” Do you prefer to live in the city or thecountryside? Please explain your choice.

Part G Retell Stories “Tres niñas caminaban a la orilla de un arroyo cuando vieron a un pajarito con las patitas enterradas en el barro...”

Page 18: A Common Testing Framework for Computerized Spoken ......You will hear two brief stories. After each story, you will have 30 seconds to retell it in Spanish as best you can. Try to

Ordinate Corporation ALTE, Berlin, Germany May 2005 18

Page 19: A Common Testing Framework for Computerized Spoken ......You will hear two brief stories. After each story, you will have 30 seconds to retell it in Spanish as best you can. Try to

Ordinate Corporation ALTE, Berlin, Germany May 2005 19

Scoring

Sentence Mastery FluencyVocabulary

MannerContent

Pronunciation

Page 20: A Common Testing Framework for Computerized Spoken ......You will hear two brief stories. After each story, you will have 30 seconds to retell it in Spanish as best you can. Try to

Ordinate Corporation ALTE, Berlin, Germany May 2005 20

Scoring LogicSentence MasteryFluency

Read Ans. Short QuestionRepeat Sentence Build S OQ St ROpposite

Pronunciation Vocabulary

HumanScoring

SST = (30% Sent.M, 20% Vocab, 30% Fluency, 20% Pron)

Page 21: A Common Testing Framework for Computerized Spoken ......You will hear two brief stories. After each story, you will have 30 seconds to retell it in Spanish as best you can. Try to

Ordinate Corporation ALTE, Berlin, Germany May 2005 21

Common Testing Framework

Test Architecture Testing System Development andValidation Process

Task Type

Task A

Task B

Task C

Task D

Task E

Task F

Task G

Phone Network

Phone Network

Page 22: A Common Testing Framework for Computerized Spoken ......You will hear two brief stories. After each story, you will have 30 seconds to retell it in Spanish as best you can. Try to

Ordinate Corporation ALTE, Berlin, Germany May 2005 22

TDS

Japan

KoreaTDS

Hong Kong

TDS

Shanghai

Taiwan

TDS

India

USA

Netherlands

Test Delivery Server (TDS)

Public Switched Telephone

Network (PSTN)

Public Switched Telephone

Network (PSTN)

serversservers DatabaseDatabase

Internet

Virtual Private Network (VPN)

TDS

TDS

TDS

Page 23: A Common Testing Framework for Computerized Spoken ......You will hear two brief stories. After each story, you will have 30 seconds to retell it in Spanish as best you can. Try to

Ordinate Corporation ALTE, Berlin, Germany May 2005 23

Common Testing Framework

Test Architecture Testing System Development andValidation Process

Task Type

Task A

Task B

Task C

Task D

Task E

Task F

Task G

Phone Network

Phone Network

Page 24: A Common Testing Framework for Computerized Spoken ......You will hear two brief stories. After each story, you will have 30 seconds to retell it in Spanish as best you can. Try to

Ordinate Corporation ALTE, Berlin, Germany May 2005 24

DevelopmentPLAN

Define Test Construct

Design Test

Create Item Specification

Page 25: A Common Testing Framework for Computerized Spoken ......You will hear two brief stories. After each story, you will have 30 seconds to retell it in Spanish as best you can. Try to

Ordinate Corporation ALTE, Berlin, Germany May 2005 25

DevelopmentPLAN DEVELOP

Define Test Construct

Write Items

Review Items

Design Test

Create Item Specification

Record Items

Create Data Collection System

Page 26: A Common Testing Framework for Computerized Spoken ......You will hear two brief stories. After each story, you will have 30 seconds to retell it in Spanish as best you can. Try to

Ordinate Corporation ALTE, Berlin, Germany May 2005 26

DevelopmentSCOREPLAN DEVELOP

Define Test Construct

Write Items Collect Data

Design Test

Create Item Specification

Review Items

Train Speech

RecognizerRecord Items Create

ModelsCreate Data Collection System

Generate Scores

Page 27: A Common Testing Framework for Computerized Spoken ......You will hear two brief stories. After each story, you will have 30 seconds to retell it in Spanish as best you can. Try to

Ordinate Corporation ALTE, Berlin, Germany May 2005 27

Validation• Reliability

• Validity• Separation of Natives and Non-Natives• Correlation with Human Ratings

Page 28: A Common Testing Framework for Computerized Spoken ......You will hear two brief stories. After each story, you will have 30 seconds to retell it in Spanish as best you can. Try to

Ordinate Corporation ALTE, Berlin, Germany May 2005 28

ReliabilityScore Types SET-10 SST

Overall 0.97 0.96

0.96

0.77

0.94

0.95

Sentence Mastery 0.93

Vocabulary 0.88

Fluency 0.95

Pronunciation 0.97

Page 29: A Common Testing Framework for Computerized Spoken ......You will hear two brief stories. After each story, you will have 30 seconds to retell it in Spanish as best you can. Try to

Ordinate Corporation ALTE, Berlin, Germany May 2005 29

Separation of Natives and Non-Natives

Page 30: A Common Testing Framework for Computerized Spoken ......You will hear two brief stories. After each story, you will have 30 seconds to retell it in Spanish as best you can. Try to

Ordinate Corporation ALTE, Berlin, Germany May 2005 30

Machine and Human Ratings

Read Short QuestionRepeat Sentence Build S OQ St ROpposite

Machine Scores

Read Short QuestionRepeat Sentence Build S OQ St ROpposite

Estimates(2 human raters per)

Human Interview Scores

Candidate

Here is a question

Interviewer 1Interviewer 2

Page 31: A Common Testing Framework for Computerized Spoken ......You will hear two brief stories. After each story, you will have 30 seconds to retell it in Spanish as best you can. Try to

Ordinate Corporation ALTE, Berlin, Germany May 2005 31

SET-10 Concurrent ValidityInstrument r n

TSE 0.88 58

New TOEFL Speaking 0.84 321

Common European Framework, 1st experiment 0.84 121

Common European Framework, 2nd experiment 0.94 150

Common European Framework, 3rd experiment 0.88 303

Page 32: A Common Testing Framework for Computerized Spoken ......You will hear two brief stories. After each story, you will have 30 seconds to retell it in Spanish as best you can. Try to

Ordinate Corporation ALTE, Berlin, Germany May 2005 32

SST ~ ACTFL OPITwo Raters ~ Machine

Different Materialr = 0.86

Page 33: A Common Testing Framework for Computerized Spoken ......You will hear two brief stories. After each story, you will have 30 seconds to retell it in Spanish as best you can. Try to

Ordinate Corporation ALTE, Berlin, Germany May 2005 33

SST ~ SPT OPI

Two Raters ~ Machine Different Material

r = 0.92

Page 34: A Common Testing Framework for Computerized Spoken ......You will hear two brief stories. After each story, you will have 30 seconds to retell it in Spanish as best you can. Try to

Ordinate Corporation ALTE, Berlin, Germany May 2005 34

SET-10 ~ CEF Estimates

20

30

40

50

60

70

80

90

-8

Council of Europe Language Levels

PhonePassSET-10OverallScore

Non-Nativespeakers(n=268)

Nativespeakers(n=35)

<A1 A1 A2 B1 B2 C1 C2

n=301r=0.88

Page 35: A Common Testing Framework for Computerized Spoken ......You will hear two brief stories. After each story, you will have 30 seconds to retell it in Spanish as best you can. Try to

Ordinate Corporation ALTE, Berlin, Germany May 2005 35

SST ~ ILR Estimate-SPTTwo Raters ~ Machine

Different Materialr = 0.89

Page 36: A Common Testing Framework for Computerized Spoken ......You will hear two brief stories. After each story, you will have 30 seconds to retell it in Spanish as best you can. Try to

Ordinate Corporation ALTE, Berlin, Germany May 2005 36

Common Testing Framework

Test Architecture Testing System Development andValidation Process

Task Type

Task A

Task B

Task C

Task D

Task E

Task F

Task G

Phone Network

Phone Network

Page 37: A Common Testing Framework for Computerized Spoken ......You will hear two brief stories. After each story, you will have 30 seconds to retell it in Spanish as best you can. Try to

Ordinate Corporation ALTE, Berlin, Germany May 2005 37

Conclusions• Common Testing Framework

Can be used for any language

Allows for quick development

• Spoken Language TestsAutomatically administered and scored

Reliable

Valid


Story Retell Narratives in Five School-Aged Children with ...

Story Retell Narratives in Five School-Aged Children with ...

W.2 - K...4 ENGLISH LANGUAGE ARTS - Week 2 Lesson 2: Retell a Story Learning Objective: Students will be able to retell a nursery rhyme by sequencing events in the correct order. English

W.2 - K...4 ENGLISH LANGUAGE ARTS - Week 2 Lesson 2: Retell a Story Learning Objective: Students will be able to retell a nursery rhyme by sequencing events in the correct order. English

Retelling Corduroy - Reach Assoc Corduroy.pdf · Retelling Corduroy ... In this module, students will learn how to retell key events of a story in ... *Elementary Teaching Task Rubric

Retelling Corduroy - Reach Assoc Corduroy.pdf · Retelling Corduroy ... In this module, students will learn how to retell key events of a story in ... *Elementary Teaching Task Rubric

Mermaid Theatre of Nova Scotia Brown Bear, Brown Bear€¦ · 3 Objectives: The student will retell the story using sequential words. The student will examine sequence in numbers

Mermaid Theatre of Nova Scotia Brown Bear, Brown Bear€¦ · 3 Objectives: The student will retell the story using sequential words. The student will examine sequence in numbers

Bilingual Spanish/English Story Retell Databases · PDF fileBilingual Spanish/English Story Retell ... Language samples for the Bilingual Spanish/English Story Retell reference databases

Bilingual Spanish/English Story Retell Databases · PDF fileBilingual Spanish/English Story Retell ... Language samples for the Bilingual Spanish/English Story Retell reference databases

LT: I can retell what an abolitionist did to try to stop ... · pretending you are this abolitionist. LT: I can retell what an abolitionist did to try to stop slavery William Lloyd

LT: I can retell what an abolitionist did to try to stop ... · pretending you are this abolitionist. LT: I can retell what an abolitionist did to try to stop slavery William Lloyd

Gathering · Web viewCreate your own puppets and use them to retell the story of the resurrection. You might want to make some props too.If you like – instead of creating puppets,

Gathering · Web viewCreate your own puppets and use them to retell the story of the resurrection. You might want to make some props too.If you like – instead of creating puppets,

retell hand 2

retell hand 2

When you know these, pharmaceuticals you will abandon, wellness you will join, first vita plus you will choose first

When you know these, pharmaceuticals you will abandon, wellness you will join, first vita plus you will choose first

Integrating Language Development in the Content · PDF fileConcept Mapping •Now. select 3 – 5 transition words . to. use . as you . retell. your . partner. what you . just shared.

Integrating Language Development in the Content · PDF fileConcept Mapping •Now. select 3 – 5 transition words . to. use . as you . retell. your . partner. what you . just shared.

First Person & Third Person need to retell it as a THIRD person narrative. How are you going to do that? What do you need to do with the pronouns? 5.

First Person & Third Person need to retell it as a THIRD person narrative. How are you going to do that? What do you need to do with the pronouns? 5.

Name: Teacher: Benchmark Assessment€¦ · 15/10/2018  · Grade 1/Benchmark 3.3 Retell: Going to Market Now tell me as much as you can about the story you just read. Ready, begin.

Name: Teacher: Benchmark Assessment€¦ · 15/10/2018  · Grade 1/Benchmark 3.3 Retell: Going to Market Now tell me as much as you can about the story you just read. Ready, begin.

Children who stutter: Bilingual Issues in Assessment ... · Tagalog ! Portuguese ! Polish ! You will see these children! ... conversation, monologue, story retell, reading, picture

Children who stutter: Bilingual Issues in Assessment ... · Tagalog ! Portuguese ! Polish ! You will see these children! ... conversation, monologue, story retell, reading, picture

€¦ · Web viewPupils also learn about carnival and Easter traditions in Spain and Latin America. Finally pupils will learn vocabulary to describe family members and retell the

€¦ · Web viewPupils also learn about carnival and Easter traditions in Spain and Latin America. Finally pupils will learn vocabulary to describe family members and retell the

NONFICTION 1ST GRADE RETELL - Common Core Kingdom

NONFICTION 1ST GRADE RETELL - Common Core Kingdom

).d…  · Web viewWeek 4 Monday 11th May to Friday 15th May. Year: 3/4 . Subject:ENGLISH. Lesson for:ALL WEEK _____ We are learning about: You will be using this stimulus to retell

).d…  · Web viewWeek 4 Monday 11th May to Friday 15th May. Year: 3/4 . Subject:ENGLISH. Lesson for:ALL WEEK _____ We are learning about: You will be using this stimulus to retell

will retell story using picture books and pictures · *Story retell using prompts or pictures in sequential order *Tell me the story using temporal concepts *Tell story - present

will retell story using picture books and pictures · *Story retell using prompts or pictures in sequential order *Tell me the story using temporal concepts *Tell story - present

RETELL FAQs

RETELL FAQs

that you see what you think about that And what it …media.collegeboard.com/digitalServices/pdf/diversity/2013/... · headline*for a presentation ... •Retell a folktale by writing

that you see what you think about that And what it …media.collegeboard.com/digitalServices/pdf/diversity/2013/... · headline*for a presentation ... •Retell a folktale by writing

Story Style and Structure. After reading a story you ask children to retell the story. Noah tells you the ending. Fatima offers a beginning, middle, and.

Story Style and Structure. After reading a story you ask children to retell the story. Noah tells you the ending. Fatima offers a beginning, middle, and.