1

IATEFL Testing, Evaluation and Assessment Special Interest Group (TEASIG) Autumn Seminar and Workshops supported by PEARSON English

TECHNOLOGY FOR TEACHERS IN ASSESSMENT – THE IMMEDIATE FUTURE 1&2 November, 2018 – PEARSON English, 80 Strand, LONDON

PLENARY SPEAKERS

David Booth – Integrated skills testing and automated scoring – where are we going? Larry Davis & Veronica Timpe Laughlin – Assessing spoken interaction: How can technology help? Peter Cormack – Teaching to the test- construct relevant and irrelevant approaches

Aylin Ünaldi – Automatic Text Analysis in Reading Research and its use in Reading Assessment

2

CONFERENCE THEME:

Technology, despite its extremely broad context, is omni-present in todays teaching world and its benefits are evident across many different teaching domains. The use of the internet, tablets, laptops, mobile phones, learning apps, social media... the list of where technology has influenced and infiltrated teachers' and learners' worlds is endless. Learners can benefit from the use of technology like never before. Even the recently accepted expression "e-learning" differentiates today's teaching and learning from an older style of course book and whiteboard.

So much has changed in such a short space of time, but many teachers feel they are running just to stand still. The endless list of new software and apps means daily challenges that are difficult to meet. The very speed of technological change has lead to widespread resistance.

Testing and assessment has not been spared the technological onslaught either. Larger and smaller scale testing has clearly benefitted from advances such as online testing, ease of preparation material, more objective marking, ability to source and prepare authentic material, whilst audio and video software has made test material development so much easier. But, like learning, technology has brought about criticism and resistance in the world of testing and assessment. Amongst major concerns are keyboard inputs in listening tests where older students can struggle, adding construct irrelevant variance. Using online testing to assess spoken interaction has been criticized for not replicating a real-world human communication. Certain test service providers advertise online testing as a huge advantage without any clear evidence to support this. Additionally test security can easily be compromised by the ability to record tests on hidden portable electronic devices.

All is not lost however! TEASIG's Autumn seminar is aimed at opening doors for teachers with practical insights and hand-on workshops which will aim to support and encourage teachers involved or likely to be in assessment and testing. It will offer advice and theoretical analysis to show how technology can really help teachers really benefit from technology in all modes of assessment which in turn can be transmitted to their learners.

Numbers for this event are limited and you can register now at: http://secure.iatefl.org/events/event.php?id=161

3

WELCOME MESSAGE:

Dear colleagues, It is a great pleasure to welcome you all to the IATEFL TEASIG (Testing, Evaluation, and Assessment Special Interest Group) Autumn Seminar and Workshop supported by Pearson English. The TEASIG 2018 autumn event will take place in Central London on November 1 and 2. You will enjoy the privilege of listening to 4 plenary talks and attending 4 long workshop sessions, on “Technology for Teachers in Assessment – The Immediate Future”, presented by invited expert speakers. We are excited and proud that we will achieve our mission of reaching out to you, by offering both seminars and workshops presented by experts in the field. We ultimately aim to help you benefit from technology in various modes of assessment, which hopefully can lead to better learning as well as more reliable and valid assessment of learning. We strongly believe the workshop sessions as well as the presentation of theoretical analysis will boost your confidence in using relevant practices for a more appropriate and comprehensive integration of technology in assessment. In the face of current paradigms, the aim of this event is to get you to discuss various ways of using technology in assessment and encouraging its integration, whenever appropriate. Finally, we would like to express our sincere gratitude to our speakers and Pearson English for their invaluable contributions and to you for your interest and time. We very much hope that you will find this event stimulating and rewarding, and hope to see you again at the future IATEFL TEASIG events. Very best wishes, IATEFL TEASIG Committee

4

CONFERENCE PROGRAMME:

DATE TIME EVENT

1.11

0930 – 0945 Opening

0945 – 1030 Plenary 1 - David Booth – Director of Test Development, PEARSON ENGLISH Integrated skills testing and automated scoring – where are we going? Room number - 10th floor Conference room

1030 – 1100 Coffee Break

1100 – 1230 40 minute-

workshops -repeated after a 10 minute-

break

Workshop A David Booth - PEARSON ENGLISH Using Artificial Intelligence and Automated Scoring across different contexts of learning Room number – Embankment Floor, Emb 2/3

Workshop B Peter Cormack- PEARSON ENGLISH Technology and test preparation Room number –Embankment Floor, Emb 115D/E

1230 – 1400 Lunch

1400 – 1445

Plenary 2 - Larry Davis & Veronica Timpe Laughlin - Center for English Language Learning and Assessment Educational Testing Service "Assessing spoken interaction: How can technology help?" Room number - 10th floor Conference room

1445 – 1515 Coffee Break

1515 – 1645 90 minute-workshops

Workshop C Joanna Wrzesinska - ETS Global Assessing speaking skills using human raters in an Internet-based environment Room number–Embankment Floor, Emb 2/3

Workshop D Alex Thorp - TRINITY How far down the digital road will EL assessment go? Room number –Embankment Floor, Emb 115D/E

1645 – 1700 Day one closing

1700 – 2000 Drinks reception

DATE TIME EVENT

2.11

0900 – 0910 Opening

0910 – 0955 Plenary 3 – Peter Cormack, English Language Assessment Director, PEARSON ENGLISH Teaching to the test- construct relevant and irrelevant approaches Room number - 10th floor Conference room

0955 – 1030 Coffee Break

1030 – 1200 90 minute-workshops

Workshop E Joanna Wrzesinska - ETS Global Assessing speaking skills using human raters in an Internet-based environment Room number–Embankment Floor, Emb 2/3

Workshop F Alex Thorp - TRINITY How far down the digital road will EL assessment go? Room number–Embankment Floor, Emb 115D/E

1200 – 1330 Lunch

1330 – 1415 Plenary 4 - Aylin Ünaldi, Assistant Professor, Bogazici University Automatic Text Analysis in Reading Research and its use in Reading Assessment Room number - 10th floor Conference room

1415 - 1445 Panel –Discussion on all issues raised during the event with the four plenary speakers

1445 – 1500 Closure

5

SPEAKERS BIOS AND ABSTRACTS:

David Booth is the Director of Test Development at Pearson English. He is responsible for ensuring that all test development processes are executed to the highest standard to ensure that test material is of the highest quality and fit for purpose. David has also contributed articles on specific test development projects in published research notes. His other interests include corpus linguistics and the application of corpora to assessment quality, including the quality of test items. He is also interested in assessment for specific purposes and has worked on tests for specific business clients as well as testing for the aviation sector. David has a Masters Degree in Education, from the Institute of Education, London University, and is a Member of the Chartered Institute of Marketing.

Integrated skills testing and automated scoring – where are we going?

Plenary: Integrated skills testing is becoming more prevalent in high stakes

testing. The presentation will look at a test of high stakes English and how

integrated skills items give greater validity and accuracy to the assessment and

test takers’ scores. The presentation will question some of the current

approaches to testing language skills where the construct of the language skill

tested is not clearly defined. The talk will then focus on the challenges faced in

developing and improving integrated skills testing in the context of automated

scoring.

Using Artificial Intelligence and Automated Scoring across different contexts of learning Workshop: The workshop will describe the implementation of artificial intelligence (AI) and automated scoring in 2 different contexts – Academic English and Young Learners. The workshop will look in detail at the validation work required to ensure that test items and test instruments are assessing students appropriately using the AI tools and automated scoring. The talk will describe how those assessments were developed incorporating AI and automated scoring and how these elements were aligned to existing proficiency standards including the CEFR.

6

Larry Davis is managing research scientist in the Center for English Language Learning and Assessment at Educational Testing Service (ETS) in Princeton, New Jersey. He has taught English at the university level in China and the U.S. and taught university-level teacher training courses. At ETS his work focuses on the assessment of speaking, including research and development of technology-based speaking tasks and automated scoring. E-mail: ldavis@ets.org

Assessing spoken interaction: How can technology help?

Authors: Larry Davis (presenting) and Veronika Timpe-Laughlin

Advances in technology have long promoted innovation in language assessment, but how might technological affordances improve the assessment of interactive speaking ability? This presentation will explore the potential for new technologies to make the assessment of dialogic speech more convenient, robust and practical. Examples will be presented that show how different technologies, from current capabilities to innovative tools, might be implemented in tasks that assess learners’ abilities to engage in types of speaking that call for interaction. In addition, we will consider how technology influences the inferences that can be made regarding test taker’s abilities, and will illustrate approaches that teachers might take to implement technology in their own speaking assessments.

Peter Cormack is English Language Assessment Manager at Pearson English. In this role, he manages the item development process for PTE Academic, working together with item writers and reviewers to ensure the validity, reliability and quality of new items being developed, as well as providing support for test preparation material development for Pearson’s computer-based English language tests. Before joining Pearson, he worked as an EFL teacher, teacher trainer and assessment developer in Europe and the Middle East. He is currently a post-graduate student in Language Testing at the University of Lancaster.

Teaching to the test- construct relevant and irrelevant approaches Plenary: When tests are high stakes, it is inevitable that teaching, learning and test preparation can result in the narrowing of the intended curriculum. Allied to this, there is an increasing interest in going one step further and manipulating the surface level characteristics of responses in the belief that particular formats or presentational styles will influence scoring rubrics. This session will explore the construct relevant and irrelevant nature of preparation techniques associated with machine scored responses across the construct of language assessment and present evidence on how testing agencies combat such preparation or gaming techniques.

Technology and test preparation Workshop: The workshop will look at models of delivering test preparation, practice and feedback to learners entirely online, and participants will consider the pros and cons of these new ways of preparing for exams.

7

Aylin Ünaldı is currently working as an assistant professor at the Department of Foreign Language Education; Education Faculty of Boğaziçi University. She has an MA in Applied Linguistics (University of Reading, UK), a PhD in Applied Linguistics-Language Testing (Bogazici University, Turkey) and a PhD in Language Assessment from the University of Bedfordshire (UK). She has extensive experience in foreign language teaching, teacher training, test development and validation. Her research interests include language test validation, academic literacy (reading and writing) in foreign language, reading across texts and reading into writing as an integrated academic skill. She worked as an ELT instructor, test writer, researcher and assessment and curriculum development consultant in national and international institutions.

Automatic Text Analysis in Reading Research and its use in Reading Assessment Reading process is shaped by an intricate interaction of the input text characteristics, task purpose and reader abilities. This talk will discuss recent technological advances which enable us to analyse many textual features automatically so that we can more efficiently judge the appropriacy of texts for inclusion in exams.

Joanna Wrzesinska joined ETS Global in 2011 and currently holds the position of Senior Academic Relations & ELT Coordinator, based in Warsaw, Poland. Joanna is an experienced teacher trainer; previously she worked as an EFL teacher and academic director in the private sector.

Assessing speaking skills using human raters in an Internet-based environment

This interactive workshop will discuss how the speaking skills are tested and scored on the TOEFL iBT® test. Both of the processes are implemented in an internet-delivered environment to ensure fairness and reliability of test scores. The participants in the workshop will explore examples of independent and integrated speaking tasks, learn how the raters are trained, and will have an opportunity to score some real test taker speaking responses using the authentic TOEFL iBT® Scoring Rubrics. Towards the end of the workshop participants will be encouraged to consider how elements of the processes discussed could be applied to the benefit of their own teaching and testing practice.

8

Alex Thorp is the Lead Academic, Language (Europe) at Trinity College London. He previously gained over 15 years' experience heading a teacher training department in the UK. With special interests in English Language Assessment, CLIL, Neuro-linguistics and Evidence-based teaching, he publishes and presents the world over with the aim of supporting best practices in language teaching and testing.

How far down the digital road will EL assessment go? There is an increasing degree of confusion surrounding the future of English language assessment, whether this will be fully digitised, AI driven with automated marking systems, or whether there will remain a place for pen-and-paper or face-to-face interactive assessment. This session will summarise recent developments in language testing approaches and methodology, and consider whether adoption of modern technologies can work in harmony with these approaches to deliver valid and reliable tests.

9

VENUE INFORMATION

80 Strand - Location and Venue Information We are looking forward to you joining our TEASIG meeting taking place on November 1 and 2 2018. Noted below you will see some useful information that will help you during your stay here in London.

Where are we?

London Transport

As a general rule it's more time & cost efficient to use public transport directly from the airport into London. Then, once you're in London to either use the London Underground (tube) or take a taxi/ Uber. Here are some transport information links from the four main airports surrounding London.

When you get here

- Please aim to arrive at the Pearson 80 Strand office 30 minutes before the start of the meeting

- Attendees will have a name badge that they need to sign for which will be located on a table

immediately to the right of the building entry

- We will then show you the way to the event area

80 Strand, London, WC2R 0RL Tel: 020 7010 2000

10

Getting online at Pearson

- Select ‘Guest@Pearson’ from your wifi settings options

- When prompted, you will need to click the ‘request access’ link to register for an account. If you

have previously registered, then please enter your login details to gain access

- The mobile number that you enter upon registration will be the one that we message your

username and password to, so please ensure that this is correct

- Please ask for the passcode from your Pearson contact, to add that to the registration form

- After filling out all sections on the registration, click generate account.

- You should now receive a text message with your username and password which is valid for

five days

- On the login page, after you click connect, you will need to accept the T’s & C’s

- You should now be successfully connected to the Pearson wireless internet

- For use on following days, you do not need to register again, you can just select the login option

for up to five days internet access.

Catering - Tea, coffee, refreshments and lunch will be served at the venue.

- Opening reception will take place at the venue on the 10th floor with an amazing London view.

All delegates are invited to the reception. This will be a great chance for both the speakers and

delegates to get to know each other in a friendly atmosphere with drinks and finger food. They

are all sponsored by Pearson English.

11

IATEFL and TEASIG IATEFL is the International Association of Teachers of English as a Foreign Language which has the aim of linking, developing and supporting English language teaching professionals worldwide. IATEFL is a charity organization, founded in 1967 by the late Dr Bill Lee. IATEFL now has over 4,000 members in 100 countries and associate members from 120+ countries. There are many different types of IATEFL membership to suit all ELT professionals. Find out why you should you join IATEFL and how you can benefit. https://members.iatefl.org/ IATEFL has 16 SIGs (Special Interest Groups), including TEASIG. Find out more. https://www.iatefl.org/special-interest-groups/introduction IATEFL TEASIG (Testing, Evaluation and Assessment Special Interest Group) TEASIG was established in 1986 and has played an active role in fostering assessment literacy in the areas of testing, evaluation and assessment. TEASIG’s mission is to share, spread, support and enhance knowledge of assessment and assessment literacy among ELT professionals internationally through webinars, conferences, sessions, discussions, publications and networks for its members and others. TEASIG

• organises events around the world;

• organises a pre-conference event and runs a TEASIG Day at the IATEFL annual conference;

• organizes regular webinars

• produces newsletters and publications on TEA-related issues

The name, Testing, Evaluation and Assessment, reflects the wide area of interests represented. TEASIG’s goal is to reflect these various interests, whether they concern classroom assessment and tests, external standardized examinations, large scale testing, evaluation of individuals, courses, teaching, programmes or institutions. Join IATEFL TEASIG and

become part of a large network of like-minded colleagues

pay reduced fees at all IATEFL and TEASIG conferences

have access to recordings of TEASIG webinars

enjoy the opportunity to meet face-to-face with experts in the field

pay reduced fees at other conferences around the world through IATEFL associate members

receive all TEASIG publications

regular updates on TEASIG and TEA-related events For more information see the TEASIG website http://tea.iatefl.org/ and join TEASIG Social Media – Facebook https://www.facebook.com/Teasig, Twitter https://twitter.com/iatefl_teasig and LinkedIn https://www.linkedin.com/groups/8369406 to participate in discussions about TEA, meet fellow testers and TEA professionals and share ideas

INST

ITU

TIO

NS

GLO

BAL

Using PTE Academic for admissions

_

Illustration by Lucy Vigrass

Cover photo by Angela Ithyle

Welcome to PTE AcademicPearson Test of English Academic (PTE Academic) is the leading computer-based English language proficiency test used by universities, professional associations and governments.

This guide provides detailed information on the test along with guidance on recognising PTE Academic – which will allow test takers to apply to your institution using their PTE Academic test score.

ABOUT THE TESTTEST STRUCTURE• Single 3 hour test session

• 3 parts: speaking and writing; reading; and listening

• 20 different question types

Drawing from 20 different task types and a constantly- changing bank of questions, test takers will need to genuinely demonstrate their English language skills.

Preparing for PTE Academic ensures that test takers are genuinely improving their English language skills, not just test skills.

Part Content Time allowed

Intro Introduction 1 minute

Part 1 Speaking and writing 77 - 93 minutes

Part 2 Reading 32 - 41 minutes

Optional scheduled break 10 minutes

Part 3 Listening 45 - 57 minutes

_

QUESTION TYPESPTE Academic consists of three main parts and there are 20 different question types. The test is an integrated skills test, each question type assessing one language skill or a combination of language skills. Question types are task-based, representing the range of functions and situations that applicants will encounter in an English speaking environment. The table below shows the test structure, question types and time allowed.

Part of Test Question Type Time Allowed

Part 1:

Speaking & Writing

Personal introduction 1 minute

Read aloud

30 - 35 minutes

Repeat sentence

Describe image

Re-tell lecture

Answer short question

2 x Summarise written text 20 minutes

Summarise written text or Write Essay 10 or 20 minutes

Write Essay 20 minutes

Part 2: Reading Multiple-choice, choose single answer

32–41 minutes

Multiple-choice, choose multiple answers

Re-order paragraphs

Reading: Fill in the blanks

Reading & Writing: Fill in the blanks

Optional scheduled break 10 minutes

Part 3:

Listening

Summarise spoken text 20–30 minutes

Multiple-choice, choose multiple answers

23–28 minutes

Fill in the blanks

Highlight correct summary

Multiple-choice, choose single answer

Select missing word

Highlight incorrect words

Write from dictation

SET AND USE SCORESPTE Academic is scored against the Global Scale of English, a granular score scale from 10–90 that is aligned with the Common European Framework of Reference (CEFR). The scale enables institutions to more accurately assess an applicant’s level of English prior to enrolling in an academic course. Learn more: pearsonpte.com/scores

Note: This score alignment is recommended by PTE Academic.

SCORING PTE ACADEMICUnlike other English tests, all questions in PTE Academic are machine scored, including the speaking section, using an automated scoring system.For question types that require speaking and writing, this technology allows us to assess multiple sub-skills (or enabling skills) from each response.

• Scores for some question types are based on correctness alone, while others are based on correctness, formal aspects and the quality of the response.

• Formal aspects refer to the form of the response. For example, if the response is over or under the word limit for a particular question type.

• The quality of the response is represented in the enabling skills. For example, in the question type Re-tell lecture the response is scored on skills such as oral fluency and pronunciation.

• Scores for question types assessing speaking and writing skills are generated by automated scoring systems. There are two types of scoring; correct/incorrect or partial credit.

• To learn more about scoring, download the Score Guide: pearsonpte.com/scores

HOW DO I SET MY SCORE?Instead of using wide score bands that can be misinterpreted, PTE Academic granular scores enable institutions to pinpoint each student’s ability.

• The concordance table below illustrates how PTE Academic and IELTS scores align.

• To set your score, simply choose the equivalent PTE Academic score to the other test(s) you already accept.

• If you would like guidance on setting your score requirement for PTE Academic, please contact PTE Academic: pearsonpte.com/institutions

PTE AcademicTest Taker Score ReportExample Test TakerTest Taker ID: PTE123456789

Date of Birth: 06 January 1980Country of Citizenship: United KingdomCountry of Residence: United KingdomEmail Address: pltsupport@pearson.com

Registration ID: 123456789

Test Date: 13 September 2016Test Centre Country: United StatesTest Centre ID: 00001First-Time Test Taker: Yes

Report Issue Date: 18 September 2016Scores Valid Until: 13 September 2018

Score Report representation right may differ from current score reports.

Overall Score

Score Scale

Communicative SkillsListening

Reading

Speaking

Writing

Enabling SkillsGrammar

Oral Fluency

Pronunciation

Spelling

Vocabulary

Written Disclosure

Communicative Skills Listening 51

Reading 69

Speaking 47

Writing 74

Enabling Skills Grammar 70

Oral Fluency 37

Pronunciation 47

Spelling 61

Vocabulary 74

Written Disclosure 61

The overall Score for the PTE Academic is based on the test taker’s performance on all items in the test. The scores for Communicative Skills and Enabling Skills are based on the test taker’s performance on only those items that pertain to these skills specifically. As many items contribute to more than one Communicative or Enabling Skill, the Overall Score cannot be computed directly from the Communicative Skill scores or from the Enabling Skill scores. The graph below indicates this test taker’s Communicative Skills and Enabling Skills relative to his or her Overall Score.

When comparing the Overall Score and the scores for Communicative Skills and Enabling Skills, please be aware that there is some imprecision in all measurement, depending on a variety of factors. For more information on interpreting PTE Academic scores please refer to Interpreting the PTE Academic Score Report which is available at www.pearsonpte.com/pteacademic/scores.

NOTE TO INSTITUTIONS: This score report is not valid unless authenticated on the PTE Academic Score Report Website: www.pearsonvue.com/ptescores.

Test Ta

ke

r, Exam

ple

23200678

Overall Score: 56

Skills Profile

10 14 18 22 26 30 34 38 42 46 50 54 58 62 66 70 74 78 82 86 90

HOW DO I RECEIVE SCORES?To ensure you maximise the security of PTE Academic scores, you should never accept paper or PDF copies of the score report.You have the ability to access score report data in two ways:• Score Report Website (SRW).• Automated Script.

Our Score Report Website (SRW) lets you:• Access test taker scores assigned to your institution.• Search by time period, name, date of birth,

or specific score report through Registration ID.• Print, save or export all data as PDF, CSV,

or TXT files.

• Listen to the sample audio files recorded by test takers.

• View and edit your institution’s contact information and user accounts.

• Visit pearsonpte.com/srw for more information.

Automated Script• If your institution needs to receive scores as raw

data that can be imported into another database or system, you can run an automated script.

• This downloading method produces the same exact score report information as a TXT or CSV file that can be imported into your institution’s relevant system.

HOW DO I INTERPRET THE SCORE REPORT? The PTE Academic Score Report includes the test taker’s overall score, communicative skills scores and enabling skills scores.

1. Overall ScoreReflects the student’s overall English language ability and is based on performance throughout the test.

2. Communicative skillsListening, reading, speaking and writing are based on all test questions that assess these skills, either as a single skill or together with other skills.

3. Enabling skillsGrammar, oral fluency, pronunciation, spelling, vocabulary and written discourse are based on all test questions assessing one or more of these skills.

Photo by Angela Ithyle

Our online practice materials are the best practice applicants can get with fast results. We offer a range of free and paid preparation materials including test tutorial, videos, Scored Practice Tests and practice material created by test developers.

FREE PREPARATION MATERIALS• Learn about PTE Academic with a free test taker

handbook available in 7 languages; an online tutorial providing a detailed overview of the test; plus FAQs.

• Prepare for the test with a combination of free skills videos or an unscored practice test.

PREPARATION BOOKS• The Official Guide to PTE Academic containing

over 200 practice questions, analysis of sample responses and test-taking tips.

• Practice Tests Plus: Test takers can see if they are ready for PTE Academic with authentic practice, comprehensive guidance and strategies for dealing with every task in the test.

TEST PREPARATION COURSES• Visit the website for a list of PTE Academic

preparation courses delivered by private language schools.

ADVANCED PREPARATIONTest takers can achieve their goals sooner, and save money, when they purchase online preparation packs allowing them to practice at home. These include a fully scored practice test that tells test takers when they are ready to book a full test.

For all resources visit: pearsonpte.com/prepare

Preparation

Pearson Language Testing, 80 Strand, London, WC2R 0RL, United Kingdom

pearsonpte.com facebook.com/pteacademic youtube.com/pearsonpte

18PT

E31

pearsonpte.com

Learn more about