Assessing speaking skills using human raters in an ......TOEFL iBT® Test Speaking Section • Test...
Transcript of Assessing speaking skills using human raters in an ......TOEFL iBT® Test Speaking Section • Test...
Copyright © 2017 by Educational Testing Service. All rights reserved. ETS, the ETS logo, MEASURING the POWER of LEARNING., E-RATER, TOEFL, TOEFL IBT and PROPELL
are registered trademarks of Educational Testing Service (ETS) in the United States and other countries. 767906
Assessing speaking skills
using human raters
in an Internet-based environment
Joanna Wrzesińska ETS Global
Copyright © 2017 by Educational Testing Service. All rights reserved. ETS, the ETS logo, MEASURING the POWER of LEARNING., E-RATER, TOEFL, TOEFL IBT and PROPELL are
registered trademarks of Educational Testing Service (ETS) in the United States and other countries. 767906
What we will be doing today
1. Introduction: Review the iBT Speaking section content
2. Discuss rater training
3. Explain and use the scoring guides and
benchmarks for Speaking to rate real test-taker
responses
Copyright © 2017 by Educational Testing Service. All rights reserved. ETS, the ETS logo, MEASURING the POWER of LEARNING., E-RATER, TOEFL, TOEFL IBT and PROPELL
are registered trademarks of Educational Testing Service (ETS) in the United States and other countries. 767906
Speaking
Copyright © 2017 by Educational Testing Service. All rights reserved. ETS, the ETS logo, MEASURING the POWER of LEARNING., E-RATER, TOEFL, TOEFL IBT and PROPELL are
registered trademarks of Educational Testing Service (ETS) in the United States and other countries. 767906
TOEFL iBT® Test
Speaking Section • Test takers respond to 6 tasks
• 2 independent tasks about familiar topics
– Responses based on personal experience
• 4 integrated tasks
– 2 Reading/Listening/Speaking tasks (1 academic, 1 campus-based)
– 2 Listening/Speaking tasks (1 academic, 1 campus-based)
– Responses based on what was read and heard
• Time: 20 minutes
Copyright © 2017 by Educational Testing Service. All rights reserved. ETS, the ETS logo, MEASURING the POWER of LEARNING., E-RATER, TOEFL, TOEFL IBT and PROPELL are
registered trademarks of Educational Testing Service (ETS) in the United States and other countries. 767906
TOEFL® Rater Training & Scoring
Apply Intensive ETS
Training Daily Calibration
Test
Monitoring by Scoring
Leader
Analyze Results
1 3 2 5 4
Copyright © 2017 by Educational Testing Service. All rights reserved. ETS, the ETS logo, MEASURING the POWER of LEARNING., E-RATER, TOEFL, TOEFL IBT and PROPELL are
registered trademarks of Educational Testing Service (ETS) in the United States and other countries. 767906
Speaking Scores
• Holistic scoring
• Test takers are rated on:
– Delivery
– Language use
– Topic development
• Ratings are 0 - 4
– Converted to a scaled score of 0 - 30
Copyright © 2017 by Educational Testing Service. All rights reserved. ETS, the ETS logo, MEASURING the POWER of LEARNING., E-RATER, TOEFL, TOEFL IBT and PROPELL are
registered trademarks of Educational Testing Service (ETS) in the United States and other countries. 767906
Speaking Benchmarks
• What is a Benchmark?
• Listen to the 4 Benchmarks for Speaking Task
#1 - Independent Speaking
Copyright © 2017 by Educational Testing Service. All rights reserved. ETS, the ETS logo, MEASURING the POWER of LEARNING., E-RATER, TOEFL, TOEFL IBT and PROPELL are
registered trademarks of Educational Testing Service (ETS) in the United States and other countries. 767906
Copyright © 2017 by Educational Testing Service. All rights reserved. ETS, the ETS logo, MEASURING the POWER of LEARNING., E-RATER, TOEFL, TOEFL IBT and PROPELL are
registered trademarks of Educational Testing Service (ETS) in the United States and other countries. 767906
Most test takers should be able to respond to this topic easily, although it is likely that
very few will limit their discussion to just “one way” their life will be different. Even test
takers with responses at the highest level may introduce more than one aspect of their
lives that they think will be different. These responses are still eligible for a 4 as long as
some detail and development is included in their discussion.
Lower level respondents may merely list a number of ways that their lives may be
changed without much development of any of them (“I will be married, have children, a
different job,” etc.); these types of responses will most likely not be eligible for a score of
4 for topic development.
Some candidates may misinterpret the prompt and talk about the world in general
(technology, etc.) without connecting it to their own experience, or misinterpret the prompt
in some other way (e.g., talking about ten years ago). Responses that misinterpret the
prompt generally will not be eligible for high marks. Consult your Scoring Leader
whenever you are unsure of the appropriate score when considering topic development.
Topic Notes
Copyright © 2017 by Educational Testing Service. All rights reserved. ETS, the ETS logo, MEASURING the POWER of LEARNING., E-RATER, TOEFL, TOEFL IBT and PROPELL are
registered trademarks of Educational Testing Service (ETS) in the United States and other countries. 767906
TOEFL iBT® Speaking Section
Benchmark speaking responses • Benchmarks – independent speaking
• Annotations provide narrative description of the level
• Each score represents a range of ability
– Level 4
– Level 3
– Level 2
– Level 1
Copyright © 2017 by Educational Testing Service. All rights reserved. ETS, the ETS logo, MEASURING the POWER of LEARNING., E-RATER, TOEFL, TOEFL IBT and PROPELL are
registered trademarks of Educational Testing Service (ETS) in the United States and other countries. 767906
Now Try Rating the Independent Task
– Listen to each response
– Use the Independent Speaking Rubric
– Give a rating from 1 to 4; take notes on rationale (how it fits the descriptor)
– Discuss your rationale in groups or pairs
Test taker number 1
Test taker number 2
Copyright © 2017 by Educational Testing Service. All rights reserved. ETS, the ETS logo, MEASURING the POWER of LEARNING., E-RATER, TOEFL, TOEFL IBT and PROPELL are
registered trademarks of Educational Testing Service (ETS) in the United States and other countries. 767906
Integrated Speaking Task • Task #4 - General/Specific
• A Reading/Listening/Speaking Academic Topic
• Consists of two sources: a short reading and a lecture, followed by a single question
– Test taker reads a paragraph (~100 words) about an academic topic
– Test taker hears an excerpt from a lecture illustrating the topic (~90 seconds)
– The test taker is asked to briefly combine and convey important information from the reading and lecture
• Note taking allowed
• Prep time: 30 seconds
• Response time: 60 seconds
Copyright © 2017 by Educational Testing Service. All rights reserved. ETS, the ETS logo, MEASURING the POWER of LEARNING., E-RATER, TOEFL, TOEFL IBT and PROPELL are
registered trademarks of Educational Testing Service (ETS) in the United States and other countries. 767906
Copyright © 2017 by Educational Testing Service. All rights reserved. ETS, the ETS logo, MEASURING the POWER of LEARNING., E-RATER, TOEFL, TOEFL IBT and PROPELL are
registered trademarks of Educational Testing Service (ETS) in the United States and other countries. 767906
Copyright © 2017 by Educational Testing Service. All rights reserved. ETS, the ETS logo, MEASURING the POWER of LEARNING., E-RATER, TOEFL, TOEFL IBT and PROPELL are
registered trademarks of Educational Testing Service (ETS) in the United States and other countries. 767906
Copyright © 2017 by Educational Testing Service. All rights reserved. ETS, the ETS logo, MEASURING the POWER of LEARNING., E-RATER, TOEFL, TOEFL IBT and PROPELL are
registered trademarks of Educational Testing Service (ETS) in the United States and other countries. 767906
Key Points
• Relict behavior is a behavior that an animal displays that once served a
purpose (for survival) but no longer serves that purpose.
• In the past, the (North American) pronghorn’s speed was critical for its
survival because its predator, the lion, also ran very fast.
• Lions are now extinct (in North America), but the pronghorn still displays the
relict behavior of running extremely fast even though that is no longer
required for survival (as it is so much faster than its current predators,
bobcats and coyotes).
Copyright © 2017 by Educational Testing Service. All rights reserved. ETS, the ETS logo, MEASURING the POWER of LEARNING., E-RATER, TOEFL, TOEFL IBT and PROPELL are
registered trademarks of Educational Testing Service (ETS) in the United States and other countries. 767906
Use the Integrated Speaking Rubric to
match the characteristics of each response
to a score.
– Test Taker 1
– Test Taker 2
Now Try Rating the Integrated
Task
Copyright © 2017 by Educational Testing Service. All rights reserved. ETS, the ETS logo, MEASURING the POWER of LEARNING., E-RATER, TOEFL, TOEFL IBT and PROPELL
are registered trademarks of Educational Testing Service (ETS) in the United States and other countries. 767906
Thank you!