Post on 24-May-2018
PELA Symposium 15 Feb 2017
Introduction
The Association for Academic Language and Learning
The Association for Academic Language and Learning (AALL) is a national association that was formed in November 2005. It is the organisational body for specialists who work with university students to enhance their learning and academic English. More broadly, it aims to enhance and extend teaching and learning opportunities through policies, practices and research. Academic Language and Learning specialists work with both local and international students at every level from first year through to postgraduate. They recognise that the linguistic challenges that students face are not remedial, but intrinsic to the learning experience of every student. All students need to understand, participate and communicate effectively in the discourses and cultures of enquiry in their disciplines.
Aims and Context of the SymposiumPost-entry language assessments (PELAs) take many forms, but all share a common goal to identify the language development needs of students. As you are aware, their use is increasing rapidly in Australian Higher Education. It is in this context that university staff, and academic language and learning specialists in particular, are called on to design and implement language diagnostics. Language diagnostics have a specific purpose to identify language strengths and weaknesses so that feedback can be provided and action taken.
The Association of Academic Language and Learning, in collaboration with Curtin University and with support from Murdoch, Edith Cowan University and the University of Western Australia have come together to bring two leading experts on language testing (Dr Ute Knoch and Professor John Read) to Perth share their knowledge and expertise with us. In particular, this Symposium aims to advance our theoretical knowledge and practical skills in academic language testing. In particular, the aim is to build our capacity to evaluate the effectiveness of existing PELAs in our institutions, and to improve our capacity to design and validate PELAs to meet the learning needs of all students.
Rationale
Post-entry language assessments (PELAs) take many forms, adopt differing processes of implementation, and serve many purposes (Knoch & Elder, 2013), yet all share a common goal to identify the language development needs of students (Dunworth, 2009, 2013; Murray, 2010). Their use is increasing rapidly in Australian Higher Education (Harris, 2013) as regulatory bodies focus on post-enrolment language assessment and development (Murray, 2010). In this context, there are expectations on ALL staff to design and implement valid and reliable diagnostic instruments. Given the diversity of the disciplines that inform academic language and literacy practice, it is not surprising that ALL staff are from “diverse and various discipline backgrounds” (Carmichael, Hicks, McGowan and Van der Wal, 1999). And, even though many have expertise in Applied Linguistics, particularly in English language teaching/TESOL, few have specific expertise to design and validate language tests.
Diagnostic AssessmentsDiagnostic language testing is a notoriously difficult and under-researched area of language assessment (Elder & Read, 2015), but it is agreed that diagnostic tests have a specific purpose to identify language strengths and weaknesses so that feedback can be provided and action taken (Alderson, 2005 as cited in Elder & Read, 2015).
The lack of research and attention on diagnostic language testing means that insufficient resources are readily available to assist staff to develop, validate and implement PELA. Coupled with insufficient financial resourcing that would otherwise allow them to purchase commercially available products, ALL staff often bear the burden of responsibility for design and implementation of PELA alone.
PELAsThe current situation has led to the proliferation of in-house PELAs that are rarely formally validated (Knoch & Elder, 2013) and while these PELAs may be well-designed and valid instruments, universities may be currently unable to provide evidence of this. It is in this context that regulatory bodies are likely to call upon institutions to defend the veracity of PELA processes and procedures and ALL staff will be on the front line in articulating and defending the work they do. Within this process, the legitimacy of ALL work will be scrutinised.
This symposium aim is to advance the theoretical knowledge and practical skills of staff involved in post-entry language testing. The symposium will develop the skills and knowledge of attendees and build their capacity to: a) evaluate the effectiveness of existing PELAs in their institutions and, b) to design and validate a range of different types of PELA to suit institutional purposes that will withstand external scrutiny. It is anticipated that this initiative will reinforce the expertise, significance and influence of the Association for Academic Language and Learning in quality assurance in Australian Higher Education.
The speakers
Jill DownieProfessor Downie is an established national and international researcher with numerous grants, publications and conference presentations, including keynote addresses.
In her role as DVC, Academic, Jill is responsible for Curtin Learning and Teaching and the quality assurance processes of learning and teaching across the University including the Academic Language Learning Policy, Library and Learning Centre; Student Support Services including Counselling and Health Services, Sport and Recreation, Housing, Child Care; Student Services from Admissions though the continuum to Graduation; Academic Registrar's Office; Ethics, Equity and Social Justice, and the Centre for Aboriginal Studies (CAS) and Indigenous strategy.
Alex BarthelAlex Barthel is a higher education consultant in academic language and learning. He was the Director of the University of Technology, Sydney Academic Language & Learning Centre.
He was the inaugural President of the Association for Academic Language & Learning and a member of the DEEWR/AUQA Steering Committee that developed the ‘English language standards in higher education’. His main interests include the development of English language policy and procedures in the context of English language standards in higher education.
Alex has received recognition for his achievements through Teaching and Learning awards from the Australian Government, University of Technology Sydney, and from the NSW TAFE Commission. Alex is formally registered as Expert in Academic Language and Learning with the national Tertiary Education Quality Standards Agency (TEQSA).
Professor John ReadJohn Read is a Professor of Applied Linguistics. He is the former Head of the Applied Language Studies and Linguistics at the University of Auckland. His main area of specialisation is language testing and assessment, with a particular focus on vocabulary assessment and the testing of English for academic and professional purposes. He has been extensively involved with the International English Language Testing System (IELTS) as a researcher, consultant and test administrator, and has directed three IELTS-funded research projects on various aspects of the test. He has been a consultant to Aviation Services Ltd and Air New Zealand on the design of English language assessments for pilots and air traffic controllers. At the University of Auckland, he is Academic Coordinator of the Diagnostic English Language Needs Assessment (DELNA), which assesses the academic language needs of students entering the university.John was President of the International Language Testing Association (ILTA) from 2011 to 2012, having previously served as Vice-President (2009-10) and a member of the Executive Board (1992-93, 2000-01). He was co-editor of the international research journal Language Testing from 2002 to 2006. More recently, he completed a four-year term as a member of the Committee of Examiners for the Test of English as a Foreign Language (TOEFL) (2011-15).
In his recent publication Post-Admission Language Assessment of University Students (PALA)John emphasises that follow up and follow through are crucial element in the successful use of PELAs. As John reminds us – post-admission language assessment is not an end in itself. In this volume, John also highlights the practical considerations of balancing resources with outcomes with a reminder that language support is a central responsibility that universities have to their students – an effective diagnostic instrument cannot reach its goals without strong institutional support.
Dr Ute KnochDr Ute Knoch is the Director of the Language Testing Research Centre at the University of Melbourne.
She has published widely with nearly 40 peer-reviewed publications which have been published in journals such as Language Testing, Language Assessment Quarterly, TESOL Quarterly, Applied Linguistics, Assessing Writing, Journal of Second Language Writing and Language for Specific Purposes.
Her research interests are in the area of writing assessment, rating processes, assessing languages for academic and professional purposes, and placement testing.
She is currently the Co-president of the Association for Language Testing and Assessment of Australian and New Zealand (ALTAANZ) and on the Executive Board of the International Language Testing Association (ILTA).
In 2014, Ute was awarded the TOEFL Outstanding Young Scholar Award by the Educational Testing Service (Princeton, US), recognizing her contribution to language assessment. In her recent chapter in Post-admission Language Assessment of University Students co-authored with Cathy Elder and Sally O’Hagan Ute argues that PELA must be ‘embedded in a more enlightened university policy which places a premium on the provision of opportunities for English language development’.
Presentations
Title page
Teaching & Learning Aymposium
Post-entry language assessment
Association for Academic Language & Learningand
Curtin University15 February 2017
Alex BARTHELHE consultant in ALL
0
PELA symposium
• 1. Definitions
• 2. Do you PELA?
• 3. Questions
16© 2017, A. Barthel
0
1. Definitions
• Post-Enrolment/Entry Language Assessment (PELA) includes any form of integrated language assessment used to diagnose the level of ELP of all or specified cohorts of students. Types, designs, content, modes, etc. of currently used PELA in Australia are summarised in the table below:
17© 2017, A. Barthel
0
1. Features of PELA in Oz universities (DoP, 2013)
18© 2017, A. Barthel
characteristic range of types
organisationdesign
content
mode
target
feedback
follow-up
0
1. Features of PELA in Oz universities (DoP, 2013)
19© 2017, A. Barthel
characteristic range of types
organisation • institution wide / faculty based / unit based
design
content
mode
target
feedback
follow-up
0
1. Features of PELA in Oz universities (DoP, 2013)
20© 2017, A. Barthel
characteristic range of types
organisation • institution wide / faculty based / unit based
design • in-house / co-designed /purchased ‘off the shelf’• single instrument / 2 part instrument: screening & diagnostic component
content • •
mode
target
feedback
follow-up
0
1. Features of PELA in Oz universities (DoP, 2013)
21© 2017, A. Barthel
characteristic range of types
organisation • institution wide / faculty based / unit based
design • in-house / co-designed /purchased ‘off the shelf’• single instrument / 2 part instrument: screening & diagnostic component
content • skills-based tasks with emphasis on writing / ‘indirect’ tasks e.g. cloze texts / oral presentations• general proficiency / generic academic register / discipline specific register
mode
target
feedback
follow-up
0
1. Features of PELA in Oz universities (DoP, 2013)
22© 2017, A. Barthel
characteristic range of types
organisation • institution wide / faculty based / unit based
design • in-house / co-designed /purchased ‘off the shelf’• single instrument / 2 part instrument: screening & diagnostic component
content • skills-based tasks with emphasis on writing / ‘indirect’ tasks e.g. cloze texts / oral presentations• general proficiency / generic academic register / discipline specific register
mode • paper based / online• timed / untimed• compulsory / optional• supervised / self-administered• secure / freely available• available all year / available at certain times• automated marking / marked by hand
target
feedback
follow-up
0
1. Features of PELA in Oz universities (DoP, 2013)
23© 2017, A. Barthel
characteristic range of types
organisation • institution wide / faculty based / unit based
design • in-house / co-designed /purchased ‘off the shelf’• single instrument / 2 part instrument: screening & diagnostic component
content • skills-based tasks with emphasis on writing / ‘indirect’ tasks e.g. cloze texts / oral presentations• general proficiency / generic academic register / discipline specific register
mode • paper based / online• timed / untimed• compulsory / optional• supervised / self-administered• secure / freely available• available all year / available at certain times• automated marking / marked by hand
target • all students / international students / new students / EAL students entering HE below a certain identified level / students enrolled in a particular course/unit
feedback
follow-up
0
1. Features of PELA in Oz universities (DoP, 2013)
24© 2017, A. Barthel
characteristic range of types
organisation • institution wide / faculty based / unit based
design • in-house / co-designed /purchased ‘off the shelf’• single instrument / 2 part instrument: screening & diagnostic component
content • skills-based tasks with emphasis on writing / ‘indirect’ tasks e.g. cloze texts / oral presentations• general proficiency / generic academic register / discipline specific register
mode • paper based / online• timed / untimed• compulsory / optional• supervised / self-administered• secure / freely available• available all year / available at certain times• automated marking / marked by hand
target • all students / international students / new students / EAL students entering HE below a certain identified level / students enrolled in a particular course/unit
feedback • score / band / descriptor / individual written feedback• confidential / distributed to academic coordinator or ALL unit
follow-up
0
1. Features of PELA in Oz universities (DoP, 2013)
25© 2017, A. Barthel
characteristic range of types
organisation • institution wide / faculty based / unit based
design • in-house / co-designed /purchased ‘off the shelf’• single instrument / 2 part instrument: screening & diagnostic component
content • skills-based tasks with emphasis on writing / ‘indirect’ tasks e.g. cloze texts / oral presentations• general proficiency / generic academic register / discipline specific register
mode • paper based / online• timed / untimed• compulsory / optional• supervised / self-administered• secure / freely available• available all year / available at certain times• automated marking / marked by hand
target • all students / international students / new students / EAL students entering HE below a certain identified level / students enrolled in a particular course/unit
feedback • score / band / descriptor / individual written feedback• confidential / distributed to academic coordinator or ALL unit
follow-up • at initiative of student / mandatory enrolment in an ALL program / contact from an ALL educator to discuss needs
0
2. Do you PELA?
26© 2017, A. Barthel
responses 2011 n=37 98% 2017 n=33 85%
01 do you PELA YES n=23 62 % YES n=23 70 %
0
2. Do you PELA?
27© 2017, A. Barthel
responses 2011 n=37 98% 2017 n=33 85%
01 do you PELA YES n=23 62 % YES n=23 70 %
02 university wide of the 23: 46 % of the 23: 52 %
0
2. Do you PELA?
28© 2017, A. Barthel
responses 2011 n=37 98% 2017 n=33 85%
01 do you PELA YES n=23 62 % YES n=23 70 %
02 university wide of the 23: 46 % of the 23: 52 %
03 faculties 21 % 61 %
0
2. Do you PELA?
29© 2017, A. Barthel
responses 2011 n=37 98% 2017 n=33 85%
01 do you PELA YES n=23 62 % YES n=23 70 %
02 university wide of the 23: 46 % of the 23: 52 %
03 faculties 21 % 61 %
04 compulsory 83 % 52 %
• for all students 78 % 78 %
• selected cohorts 13 % 17 %
0
2. Do you PELA?
30© 2017, A. Barthel
responses 2011 n=37 98% 2017 n=33 85%
01 do you PELA YES n=23 62 % YES n=23 70 %
02 university wide of the 23: 46 % of the 23: 52 %
03 faculties 21 % 61 %
04 compulsory 83 % 52 %
• for all students 78 % 78 %
• selected cohorts 13 % 17 %
05 assessing general English 70 % 78 %
• academic writing 83 % 83 %
• reading 52 % 61 %
• listening 30 % 13 %
• speaking 9 % (n=1) 4 %
0
2. Do you PELA? (continued)
31© 2017, A. Barthel
responses 2011 n=37 98% 2017 n=33 85%
06 locally developed of the 23: 83 % of the 23: 83 %
• commercially purchased 17 % 17 %
0
2. Do you PELA? (continued)
32© 2017, A. Barthel
responses 2011 n=37 98% 2017 n=33 85%
06 locally developed of the 23: 83 % of the 23: 83 %
• commercially purchased 17 % 17 %
07 paper based 71 % 57 %
• online 52 % 57 %
0
2. Do you PELA? (continued)
33© 2017, A. Barthel
responses 2011 n=37 98% 2017 n=33 85%
06 locally developed of the 23: 83 % of the 23: 83 %
• commercially purchased 17 % 17 %
07 paper based 71 % 57 %
• online 52 % 57 %
08 marked by A.L.L. staff 78 % 61 %
• by other uni staff 30 % 22 %
• externally/automatically 6 % 30 %
0
2. Do you PELA? (continued)
34© 2017, A. Barthel
responses 2011 n=37 98% 2017 n=33 85%
06 locally developed of the 23: 83 % of the 23: 83 %
• commercially purchased 17 % 17 %
07 paper based 71 % 57 %
• online 52 % 57 %
08 marked by A.L.L. staff 78 % 61 %
• by other uni staff 30 % 22 %
• externally/automatically 6 % 30 %
09 results sent to students 100 % 100 %
• to faculties 65 % 61 %
• to others 39 % 22 %
0
2. Do you PELA? (continued)
35© 2017, A. Barthel
responses 2011 n=37 98% 2017 n=33 85%
06 locally developed of the 23: 83 % of the 23: 83 %
• commercially purchased 17 % 17 %
07 paper based 71 % 57 %
• online 52 % 57 %
08 marked by A.L.L. staff 78 % 61 %
• by other uni staff 30 % 22 %
• externally/automatically 6 % 30 %
09 results sent to students 100 % 100 %
• to faculties 65 % 61 %
• to others 39 % 22 %
10 URL with info about PELA 40 % 39 %
0
2. Do you PELA? (continued)
36© 2017, A. Barthel
responses 2011 n=37 98% 2017 n=33 85%
06 locally developed of the 23: 83 % of the 23: 83 %
• commercially purchased 17 % 17 %
07 paper based 71 % 57 %
• online 52 % 57 %
08 marked by A.L.L. staff 78 % 61 %
• by other uni staff 30 % 22 %
• externally/automatically 6 % 30 %
09 results sent to students 100 % 100 %
• to faculties 65 % 61 %
• to others 39 % 22 %
10 URL with info about PELA 40 % 39 %
11 additional comments 57 % 26 %
0
3. Questions1. how useful / effective is PELA? assessing R? W? L? S?
2. should PELA be generic or discipline specific?
3. optional or compulsory? enforceable?
4. stand-alone or linked to A.L.L. strategies?
5. post-entry (pre-course) or integrated?
6. validation / reliability?
7. how to link PELA to academic performance?
8. ……?37© 2017, A. Barthel
thank you
Alex BarthelHigher Education consultant in academic language & learning
lexybar@gmail.com
© 2017, A. Barthel 38
ISSUES IN THE DESIGN OFPOST-ENTRY LANGUAGE ASSESSMENTSAALL SYMPOSIUM, CURTIN UNIVERSITY, 15 FEB 2017
JOHN READUNIVERSITY OF AUCKLAND
OUTLINE OF THE ISSUES
The nature of PELAs, as compared to international proficiency tests
The diagnostic dimension The constructs that may underlie a PELA Design implications of the constructs Postgraduate vs. undergraduate PELAs The interaction between PELAs and provisions for language
development
THE DISTINCT FEATURES OF PELASAS COMPARED TO PROFICIENCY TESTS
Proficiency Tests (IELTS, TOEFL, PTE)
PELAs
Used for high-stakes admissions decisions
Funded by high fees charged to test-takers
Generic and inflexible in nature
Highly professional in design, administration and score reporting
Results evaluated by users primarily on a pass/fail basis
Used for low(er)-stakes advising after admission
Funded by the university or the faculty
Can address local needs Variable in the quality of
the assessment, depending on the expertise available
Results in profile form to inform subsequent language development
THE DIAGNOSTIC DIMENSION
THE DIAGNOSTIC DIMENSION
Several PELAs have the term “diagnostic” in their title. The naming of these PELAs tends to predate the current interest
in language diagnosis among language testers. It is useful to consider post hoc the extent to which the PELAs are
diagnostic in nature.
LANGUAGE DIAGNOSISSOME KEY REFERENCES
Alderson, J.C. (2005). Diagnosing foreign language proficiency. London: Continuum. Alderson, J. C., Brunfaut, T., and Harding, L. (2015). Towards a theory of diagnosis in
second and foreign language assessment: Insights from professional practice across diverse fields. Applied Linguistics, 36(2), 236−260.
Alderson, J. C., Haapakangas, E.-L., Huhta, A., Nieminen, L., & Ullakonoja, R. (2014). The diagnosis of reading in a second or foreign language. New York: Routledge.
Lee, Y-W. (Ed.) (2015). Special issue: Future of diagnostic language testing. Language Testing, 32(3).
SOME FEATURES OF DIAGNOSTIC ASSESSMENT(ALDERSON, 2005)
A focus on areas of difficulty and the reasons for poor performance.
An analytic approach to language ability◦ Language elements: vocabulary, grammar, phonology◦ Subskills in reading and listening; speaking and writing
A basis for informed decisions about subsequent teaching and learning.
For adult learners, a process of raising awareness and encouraging learner autonomy.
THE DELNA PROCESS AT AUCKLAND
46
Computer-based Screening
of all 1st Year Undergraduates
No further assessment
Paper-based Diagnosis
OK
Not OK
General advice on academic language enhancement OK
Consultation with Language
Advisor
Not OK
Individual advice on recommendedor required language development
DELNA AS A DIAGNOSTIC ASSESSMENT
The basic purpose of identifying students whose academic language proficiency may not be adequate.
The use of measures of language knowledge (vocabulary, timed reading) in the Screening phase, including the speeded element.
The analytic rating scale for Writing, with separate ratings for Fluency, Content and Form and further sub-categories.
The role of the Language Advisor in interpreting a student’s Diagnosis results and advising on language enhancement options.
THE UNDERLYING CONSTRUCTSAND THEIR DESIGN OPTIONS
THE CONSTRUCTS UNDERLYING A PELAFOUR OPTIONS
GENERIC (Academic) language knowledge/competence Academic language proficiency/academic literacy
DISCIPLINE-SPECIFIC Academic literacies Professional communication skills
THE CONSTRUCTS UNDERLYING A PELASOME USEFUL REFERENCES
Davies, A. (2008). Assessing academic English. Testing English proficiency 1950-89: The IELTS solution. Cambridge: Cambridge University Press.
Murray, N. (2010). Considerations in the post-enrolment assessment of English language proficiency: From the Australian context. Language Assessment Quarterly, 7(4),343-358.
Murray, N. (2016). Standards of English in higher education: Issues, challenges and strategies. Cambridge: Cambridge University Press. Chapter 3.
Read, J. (2015). Assessing English proficiency for university study. Basingstoke, UK: Palgrave Macmillan. Chapters 6 and 7.
DESIGN OPTIONS(ACADEMIC LANGUAGE KNOWLEDGE
DISCRETE-POINT Vocabulary: multiple-choice, matching, gap-filling, (timed) Yes/No Grammar: multiple-choice, gap-filling, sentence rewriting
INTEGRATIVE Cloze procedure / C-Test Cloze-elide Dictation / Elicited imitation
DESIGN OPTIONSACADEMIC LANGUAGE PROFICIENCY
Assessment tasks with a skills focus, designed to simulate to some extent academic study tasks
Exemplified by the tasks in the major international proficiency tests (IELTS, TOEFL, PTE)
May be independent (focused on a single macro-skill) or integrated (requiring a combination of skills, such as Listening + Reading Writing)
THE DELNA TASKS AND THEIR CONSTRUCTS
SCREENING (Computer-based) Vocabulary Timed Reading (cloze-elide)
DIAGNOSIS (Paper-based) Listening to a mini-lecture Reading academic-type texts Writing an interpretation of a graph
Academic Language Knowledge
Academic Language Proficiency
DESIGN OPTIONSACADEMIC LITERACIES
Key feature: tailored to the study skills, language knowledge and discourse norms required in a particular discipline or faculty.
Normally requires consultation, if not collaboration, with subject lecturers.
Best-known model: the integrated writing task used in Sydney’s MASUS (Measuring Academic Skills of University Students)
A SAMPLE MASUS TASKBONANNO & JONES, 2007
Design Architecture (undergraduate) Input: campus tour, library research session, information on referencing
conventions Prompt:
Identify two or three architectural styles present on the main campus of the University of Sydney. Do you think that having so many different architectural ideals reflected in the buildings and spaces on the campus gives an unharmonious effect?
http://sydney.edu.au/stuserv/documents/learning_centre/MASUS.pdf55
BUILDINGS AT SYDNEY UNIVERSITY
56
PROFESSIONAL COMMUNICATION SKILLS(MURRAY, 2010, 2016)
Intercultural competence A cultural relativistic orientation Interpersonal skills Conversancy in the discourses and behaviours associated with
particular domains Non-verbal communication skills Group and leadership skills
DESIGN OPTIONSPROFESSIONAL COMMUNICATION SKILLS
Questionable whether such skills can or should be assessed in a PELA.
Better assessed through appropriate course work or an exit/graduating assessment, especially where students need to seek professional registration after graduating.
May influence the design of a PELA based on the academic literacies construct.
OTHER PELA DESIGN ISSUES
PELAS FOR POSTGRADUATE STUDENTS
Few if any (?) PELAs designed specifically for postgraduates. If their undergraduate study was non-English-medium, the same
kind of PELA (as for undergrads) may be appropriate. Despite experience of co-authored articles and conference
presentations in English, diagnosis of academic language knowledge may still be important.
DELNA for doctoral candidates replaces the independent writing task (for undergrads) with a more integrated reading-writing task.
THE INTERACTION OF PELAS AND SUBSEQUENT LANGUAGE DEVELOPMENT
If provision for language development is largely generic in nature, PELAs based on academic language knowledge and proficiency are most suitable.
Discipline-specific language development opens the opportunity for a PELA based on an academic literacies construct, depending on the level of cooperation between language/literacy tutors and subject specialists.
MODELS OF COLLABORATION IN LANGUAGE DEVELOPMENT (JONES, BONANNO & SCOULLER, 2001)
Weak Adjunct ModelGeneric tutorials by learning advisors on academic writing outside of class hours
Strong Adjunct ModelA stronger focus on writing genres that are relevant to the students’ discipline
Integrative ModelLearning advisors give presentations or workshops on discipline-specific aspects of academic literacy during class hours
Embedded ModelA course curriculum focused on literacy in the discipline, designed collaboratively by learning advisors and the subject lecturers who will actually teach the course.
AN INCLUSIVE MODEL(WINGATE, 2015)
Academic literacy instruction should focus on an understanding of the genres associated with the students’ academic subjects, rather than taking the generic EAP approach.
All students should have access to this instruction, regardless of language background. Language support for non-native speakers should be provided in addition to the academic literacy instruction.
The instruction needs to be integrated with the teaching of content subjects so that ideally academic literacy is assessed as part of the subject curriculum.
Academic literacy instruction requires collaboration between writing experts and subject experts to develop the curriculum jointly.
EMBEDDED LANGUAGE DEVELOPMENTSOME KEY REFERENCES
Arkoudis, S., Baik, C., & Richardson, S. (2012). English language standards in higher education. Camberwell, Vic.: ACER Press.
Arkoudis, S., & Kelly, P. (2016). Shifting the narrative: International students and communication skills in higher education. IERN Research Digest, 8. International Education Association of Australia. Available: www.ieaa.org.au/documents/item/664.
Jones, J., Bonanno, H., & Scouller, K. (2001). Staff and student roles in central and faculty-based learning support: Changing partnerships. Paper presented at the 2001 National Language and Academic Skills Conference. Available at: http://learning.uow.edu.au/LAS2001/selected/jones_1.pdf.
Wingate, U. (2015). Academic literacy and student diversity: The case for inclusive practice. Bristol: Multilingual Matters.
OUTLINE OF THE ISSUES
The nature of PELAs, as compared to international proficiency tests
The diagnostic dimension The constructs that may underlie a PELA Design implications of the constructs Postgraduate vs. undergraduate PELAs The interaction between PELAs and provisions
for language development
OKHave I
covered them all?
Thanks for your attention!
EVALUATING POST-ENTRY LANGUAGE ASSESSMENTS (PELAS)Ute KnochLanguage Testing Research CentreUniversity of Melbourne
PELA EVALUATION
Studies set out to evaluate specific aspects of PELAs and policies (e.g. Read, 2008; Elder & von Randow, 2008; Davies and Elder, 2005; Bright & von Randow, 2004; Knoch, 2012)
DEGREES OF PROFICIENCY PROJECT
Presented Evaluation as one aspect to consider when designing/implementing a PELA
Shown as the last tile
PELA EVALUATION
2013 - Knoch & Elder proposed framework for PELA
evaluation (PLTA) Read (2015) applied this framework to the DELNA
used at the University of Auckland
AIMS OF TODAY’S TALK1. Outline the key aspects of the framework
(Knoch & Elder, 2013) show examples of such an evaluation in light of three different PELAs I have
worked on2. Show impact of policy on evaluation outcome
3. Discuss the impact of practicality
4. Timing of the evaluation
PELA EVALUATION ARGUMENT Based on validity theory from educational evaluation Includes a number of building blocks which are evaluated
together to form an overall argument Important: focusses not just narrowly on the assessment but
also evaluates the decisions made based on the assessment and the consequences for the candidates and other stakeholders
Directly examines the policy that supports the assessment implementation
OVERVIEW OF PELA VALIDITY ARGUMENT
Beneficial consequences
Appropriacy of decisions
Relevance of PELA domain
Reliability of results across administrations
Characteristics of test,
scoring and test
administration
SUMMARY OF PELA VA
LIDITY ARGUMENT
Consequences of P
ELA are beneficial
Decisions made based onapprop
PELA are equitable and riate
Relevance of assess
dom
Reliability of
ment to academic ain
results
Characteristics of test,
scoring and test
administration
Warrants Sources of supporting evidence
1. Raters can implement scoring procedures consistently
Statistical analysis of ratings
2.Test administration conditions are clearly articulated and appropriate
Student questionnaires/interviews; review of test administration protocol; observation of test sessions; interviews with test invigilators
3. Instructions and tasks are clear to all test takers
Student questionnaires/interviews
4.Test is pitched at appropriate difficulty level and test tasks/items discriminate consistently between more and less able candidates
Statistical analysis of test properties (i.e. item difficulty, discrimination, internal consistency)
OVERVIEW OF PELA VALIDITY ARGUMENT
Beneficial consequences
Appropriacy of decisions
Relevance of PELA domain
Reliability of results
Characteristics of test,
scoring and test
administration
THREE SAMPLE PELAS
Diagnostic English
Language Assessment
(DELA) - UniMelb• Assessment of listing,
reading and writing
• Pen-and-paper
Academic English Screening Test (AEST)
– trial UniMelb
• Online Screening test
Diagnostic Pharmacy Writing Assessment
– University Y• Detailed diagnosis of writing
• Pen-and-paper
DELA (DIAGNOSTIC ENGLISH LANGUAGE ASSESSMENT) – UNIVERSITY OF MELBOURNEPen-and-paper assessment of listening, reading, and
writingListening and reading marked using detailed marking
keysWriting assessed by human raters (single marked due to
costs) using three criteria (organisation, content, form)Various equated versions of the test materials availableStudents receive feedback descriptors for reading andlistening and three writing criteria
DELA (DIAGNOSTIC ENGLISH LANGUAGE ASSESSMENT) – UNIVERSITY OF MELBOURNE
Assessment compulsory for specific student groups (IELTS below 7 or equivalent)
students placed into one of three groups: at risk, borderline, proficient. At risk students required to enrol in compulsory
support Language support following DELA depends on faculty Support not well-linked to needs of students Diagnostic potential of test not mined as basis for support No follow-up of students who do not enrol in
‘compulsory support’
CHARACTERISTICS OF TEST, SCORING AND ADMINISTRATION
Warrants DELAInstructions and tasks are clear to all test takers Test is pitched at appropriate difficulty level Test tasks/items discriminate consistently betweenmore or less able candidates
Scoring is consistent /?Test administration conditions are clearly articulatedand appropriate
Evaluation
RELIABILITY OF RESULTS ACROSS ADMINISTRATIONSWarrants DELA
Different test forms are parallel in design Appropriate equating procedures are used to ensure equivalency across forms
Sufficient tasks are included to provide stable estimatesof test taker ability
/?
Test administration conditions are consistent
Evaluation
RELEVANCE OF PELA TO DOMAINWarrants DELA
Characteristics of test tasks are similar to those required ofstudents in the academic domain
Scoring criteria and rubrics capture relevant aspects Test results are good predictors of language performance inacademic domain
Scores derived from the test provide sufficient information about candidates’ academic language proficiency
/?
Performance on PELA relates to performance on other assessments of academic language proficiency
Evaluation
APPROPRIACY OF DECISIONS BASED ON TEST
Warrants DELAStudents are correctly categorised based on theirtest scores
/?
The test results include feedback on test performance and a recommendation
Recommendation is closely linked to on-campus support
Assessment results are distributed in a timelymanner
/
Test results are available to all relevant stakeholders
Test users understand the meaning and intended uses of scores
Evaluation
CONSEQUENCESWarrants DELA
All targeted test takers sit for the test The test does not result in stigma or disadvantage for students Test takers’ perceptions of the test and its usefulness are positive /?The feedback from the test is useful and directly informs students’ future learning /?Students act on the test recommendation Follow-up language development options provided for students are appropriate and ongoing if needed
Learners taking up support options improve their English overthe course of their studies
/?
Students who fail to act on the recommendations are more likely to struggle in their academic studies
Evaluation
SUMMARY OF DELA EVALUATION ARGUMENT
DELACharacteristics of test, scoring and test administration
Reliability of results Relevance of PELA domain Appropriacy of decisions
Beneficial consequences
THREE SAMPLE PELAS
Diagnostic English
Language Assessment
(DELA) - UniMelb• Assessment of listing,
reading and writing
• Pen-and-paper
Academic English Screening Test (AEST)
– UniMelb trial
• Online Screening test
Diagnostic Pharmacy Writing Assessment
– University Y• Detailed diagnosis of writing
• Pen-and-paper
ACADEMIC ENGLISH SCREENING TEST (AEST) – TRIAL UNIMELB
Online screening test: two test sections: text completion & speeded reading – 25 minutes Automated scoring & instant test result
Students take assessment at home in their own timeCombined scores on the screening tasks
are used to predict which students are likely to be linguistically at risk
Students grouped into one of three groups: proficient,borderline, at risk
ACADEMIC ENGLISH SCREENING TEST (AEST)
Administered universally to ALL incoming undergraduate and postgraduate students
Students receive recommendation and list ofsuggested courses (if applicable)At risk student encouraged to take uprecommended support
CHARACTERISTICS OF TEST, SCORING AND ADMINISTRATION
Warrants AESTInstructions and tasks are clear to all test takers Test is pitched at appropriate difficulty level Test tasks/items discriminate consistently between more or less able candidates
Scoring is consistent Test administration conditions are clearly articulated and appropriate /?
Evaluation
RELIABILITY OF RESULTS ACROSS TEST ADMINISTRATIONS
Warrants AESTDifferent test forms are parallel in design Appropriate equating procedures are used to ensure equivalency across forms
Sufficient tasks are included to provide stable estimates of test taker ability Test administration conditions are consistent ?
Evaluation
RELEVANCE OF PELA TO DOMAINWarrants AESTCharacteristics of test tasks are similar to those required of students in the academic domain
Test results are good predictors of languageperformance in academic domain /?Scores derived from the test provide sufficient information about candidates’ academic language proficiency
/?
Performance on PELA relates to performance on other assessments of academic language proficiency
Evaluation /?
APPROPRIACY OF DECISIONS BASED ON TEST
Warrants AESTStudents are correctly categorised based on their test scores /?The test results include feedback on test performanceand a recommendation
/
Recommendation is closely linked to on-campus support Assessment results are distributed in a timely manner Test results are available to all relevant stakeholders Test users understand the meaning and intended usesof scores
?Evaluation
/?
CONSEQUENCESWarrants AEST
All targeted test takers sit for the test The test does not result in stigma or disadvantage for students Test takers’ perceptions of the test and its usefulness are positive
/?The feedback from the test is useful and directly informs their future learning /?Students act on the test recommendation Follow-up language development options provided for students are appropriate and ongoing if needed Learners who take up support options improve their English over the course of their studies
?Students who fail to act on the recommendations are more likely to struggle in their academic studies
?
Evaluation
/?
SUMMARY OF AEST (UNIMELB)AEST
Characteristics of test, scoring and test administration
Reliability of results across administrations
Relevance of PELA domain /?Appropriacy of decisions /?Beneficial consequences /?
THREE SAMPLE PELAS
Diagnostic English
Language Assessment
(DELA) - UniMelb• Assessment of listing,
reading and writing
• Pen-and-paper
Academic English Screening Test
(AEST)–UniMelb• Online
Screening test
Diagnostic Pharmacy Writing Assessment
– University Y• Detailed diagnosis of writing
• Pen-and-paper
PHARMACY – DIAGNOSTIC WRITING ASSESSMENT
Administered to all new incoming undergraduatesPen-and-paper assessment of writingWriting task based on pharmacy/pharmaceutical
sciences-relevant topicsWriting assessed by human raters (single marked due
to costs) using very detailed diagnostic criteria
PHARMACY – DIAGNOSTIC WRITING ASSESSMENT
All students, regardless of background, are asked to sit the assessment during class time
Students receive these detailed feedbackdescriptors three days following administrationStudents are also grouped into one of three groups
(proficient, borderline, at risk)Lowest group receives support – workshops are
designed to align with the key areas of feedback on the sheet
CHARACTERISTICS OF TEST, SCORING AND ADMINISTRATION
Warrants PharmacyInstructions and tasks are clear to all test takers Test is pitched at appropriate difficulty level Test tasks/items discriminate consistently between more or less able candidates
Scoring is consistent /?Test administration conditions are clearly articulated and appropriate Evaluation /?
RELIABILITY OF RESULTS ACROSS ADMINISTRATIONS
Warrants PharmacyDifferent test forms are parallel in design Appropriate equating procedures are used to ensure equivalency across forms n/aSufficient tasks are included to provide stable estimates oftest taker ability
Test administration conditions are consistent Evaluation /?
RELEVANCE OF PELA TO DOMAINWarrants PharmacyCharacteristics of test tasks are similar to those required of students inthe academic domain
Scoring criteria and rubrics capture relevant aspects Test results are good predictors of language performance in academic domain /?Scores derived from the test provide sufficient information about candidates’ academic language proficiency
/?Performance on PELA relates to performance on other assessments of academic language proficiency Evaluation
APPROPRIACY OF DECISIONS BASED ON TEST
Warrants PharmacyStudents are correctly categorised based on their test scores /?The test results include feedback on test performance and a recommendation
Recommendation is closely linked to on-campus support
Assessment results are distributed in a timely manner Test results are available to all relevant stakeholders Test users understand the meaning and intended uses ofscores
Research underway
Evaluation
CONSEQUENCESWarrants Pharmac
yAll targeted test takers sit for the test The test does not result in stigma or disadvantage for students
Test takers’ perceptions of the test and its usefulness are positive The feedback from the test is useful and directly informs their future learning Students act on the test recommendation Follow-up language development options provided for students are appropriate and ongoing if needed Learners who take up support options improve their English over the course of their studies tbcStudents who fail to act on the recommendations are more likely to struggle in their academic studies tbc
Evaluation
SUMMARY OF PHARMACY PELAPharmacy
Characteristics of test, scoring and testadministration
/?
Reliability of results /?Relevance of PELA domain /?Appropriacy of decisions Beneficial consequences
IMPACT OF POLICY ON EVALUATION OUTCOME
Institutional policy plays a key part in determining evaluation outcomesAEST UniMelb trial AEST University XCharacteristics of test, scoring and test administration
Reliability of results Relevance of PELA domain /? /?
Appropriacy of decisions /? /?
Beneficial consequences /?
PRACTICALITYBeneficial consequences
Appropriacy of decisions
Relevance of PELA domain
Reliability of resultsCharacteristics of test, scoring and test administration
Practicality
Low-stakes environment with little funding and large test taker populations
PRACTICALITY – SUMMARY OF THREE PELAS
DELA AEST PharmacyCharacteristics of test, scoring and test administration
/?
Reliability ofresults
/?Relevance of PELA domain
/? /?
Appropriacy ofdecisions /?
Beneficial consequences
TIMING AND DIRECTIONALITY OF PELA VALIDITY ARGUMENT
Beneficial consequences
Appropriacy of decisions
Relevance of PELA domain
Reliability of results
Characteristics of test,
scoring and test
administration
IN SUMMARYPELA Evaluation involves a number of levelsPolicy in which PELA is embedded has
outcome on evaluation
Practicality
Evaluation should be fore fronted at the time of PELA design and policy formation
Thanks• Thanks to AALL for awarding the grant to bring these two speakers to Perth
• Thanks to the AALL membership, without whom this even would not be possible
• Thanks to the WA-ALL Subcommittee – Steve Johnson - Murdoch, Meriel Griffiths – the Uni of WA, and Jo Ashton Edith Cowan Uni for their assistance in organising this event
• Thanks to my faculty counterparts Trish Dooey, Janine Rutledge and Reva Ramiah – who keep my feet on the ground and my head above water – thank you
• I’d also like to thank Craig Zimitat and Sonia Ferns from Curtin Learning and Teaching, Celia Cornwell, the Dean International Curtin Health Sciences, Jana Brock, the DVCE academic Murdoch University, Siri Barret-Lennard from Study Smarter at the University of WA and Michael Stein from Edith Cowan Learning and Teaching for their generous contributions to this event
• Finally, please join me once again in thanking our speakers who have been so generous in coming here to share their knowledge and expertise with us