Assessment literacy: What is it

and how best to achieve it?

Fiona Barker

Cambridge English Language Assessment

25 April 2014, Budapest

© UCLES 2014

Outline

1. Rationale for this workshop

2. Our experiences of assessment

3. Our view of assessment

4. An exam board’s approach to assessment

5. Impacts of assessment

6. Connecting teaching, learning and assessment

7. Find out more: assessment resources

8. Applying assessment knowledge in your context

© UCLES 2014

1. Rationale

• Explore some of the key concepts of assessment that all

practitioners should have a working knowledge of in order

to teach effectively and to make the most of assessment

opportunities.

• Think about how our learners would benefit from a better

understanding of why and how they are assessed and

how to support them to become more autonomous.

• We will draw upon existing approaches and our

experiences of learning about and practising assessment

in a variety of contexts, including language schools and in

an exam board. © UCLES 2014

To start us thinking, ask yourself:

• What do I know about assessment already?

• What would I like to know about assessment?

• What do I need to do to get to this level of

understanding and be able to apply it?

© UCLES 2014

2. Our experiences of assessment

1. Learning

2. Being

3. Doing

© UCLES 2014

2.1 Learning about assessment

1. How did you learn about assessment in your role as a

teacher/trainer/manager etc.?

2. Were you explicitly taught about assessment or

expected to pick it up by yourself, or something else

happened?

© UCLES 2014

2.2 Being assessed

1. What is your earliest memory of being assessed?

2. What was positive/negative about it for you?

3. Do you have another positive or negative memory of

being assessed?

© UCLES 2014

2.3 Doing assessment

• What is your top tip for assessing students?

© UCLES 2014

3. Our view of assessment

• What is it?

• Are there different types?

• How can it be done?

© UCLES 2014

What is assessment?

• What words do you associate with assessment?

© UCLES 2014

© UCLES 2014

www.wordle.net

Are there different types?

• Name as many different types of assessment as you can

• Tests

• Quizzes

• Portfolios

• Recordings

• Presentations

• etc

© UCLES 2014

Aspects of assessment

Who?

• Assesses: Self / Peer /

Teacher / Examiner

• Takes part: singleton /

pair / group

When?

• planned or ad hoc

• regularly – e.g. every

lesson, end of week /

topic / module / term /

etc.

What conditions?

• Exam or freer

• Timed or not

• Support or resources

permitted

• Online or offline

• Face to face or distance

Where?

• Exam hall / classroom /

home / testing centre /

other

Why?

• Many purposes© UCLES 2014

4. An exam board’s approach

• Purpose of assessment

• Brief history

• Essential test qualities

• How to evaluate a test

• Topical issues

© UCLES 2014

Purpose of assessment

• Language assessment aims to measure a hidden

trait (i.e. language ability).

• This measurement allows us to make inferences

about an individual’s language ability.

• We make inferences on the basis of an individual’s

observed behaviour(s) in the assessment situation,

usually on the basis of test scores attached to them –

score interpretation.

• We need to think about the correspondence between

general language use in a non-test context (the

target language use situation) and specific

performance in a language test. © UCLES 2014

Measuring constructs

How?

Link hidden trait to observable behaviour

Attach test scores to observable behaviour

© UCLES 2014

Brief history

• Discrete point tests (1960s)Language broken down into components and tested separately, e.g. reading, grammar, phonology

• Integrative testing (1970s)Tests tap into several competencies, e.g. writing a composition, cloze test, dictation

• Communicative language testing & performance based assessment (1980s onwards)Correspondence between test and non-test situations, e.g. oral/written production, open-ended responses, integrated tasks

© UCLES 2014

Example: Measuring Anxiety

• What behaviours are associated with anxiety?

• How could we measure these behaviours?

• We would then attach a level or score to these

behaviours to interpret what they mean.

© UCLES 2014

Essential test qualities

Impact

Validity

Practicality

Reliability

Q

© UCLES 2014

Validity

CONSTRUCT VALIDITY: central concept

Extent to which test scores can be considered a true

reflection of underlying ability the test is trying to measure

Q: Does the test measure what it is supposed to measure?

FITNESS FOR PURPOSE

Tests/test scores are not ‘valid’, they are VALID FOR A

PURPOSE

© UCLES 2014

Reliability

Extent to which test scores are consistent and accurate,

and therefore free of measurement error

Q: Is there consistency of measurement?

Test reliability and rater reliability

• Objectively marked tests, e.g. a multiple choice test

� statistical measures

• Subjectively marked tests, e.g. a writing/speaking test

� rater reliability

Q: How can test reliability be increased?

© UCLES 2014

Enhancing test reliability

• Take enough samples of behaviour.

• Do not allow candidates too much freedom:

Write a composition on tourism vs. Write a composition on how

the tourist industry in this country might be developed.

• Write unambiguous items.

• Provide clear and explicit instructions.

• Ensure candidate familiarity with format/testing techniques.

• Provide uniform administration conditions.

• Make scoring as objective as possible.

• Train scorers and get multiple, independent scoring.(Hughes, 1989)

© UCLES 2014

Impact

The effect of a test on test takers, education systems and

society more widely:

‘the larger framing and social meaning of assessment’ (McNamara, 2000)

• Micro: effect on classrooms

(washback)

• Macro: effect on society

© UCLES 2014

Ensuring positive washback

and impact

• Test only those abilities whose development you want to

encourage and not what is easiest to test.

• Give sufficient weight to those abilities.

• Sample widely from the non-test domain.

• Use direct testing (i.e. performance skills).

• Make testing criterion-referenced.

• Provide assistance to teachers.

© UCLES 2014

Practicality

Balancing required resources and available resources

© UCLES 2014

Balancing test qualitiesG

Validity Reliability

Impact Practicality

© UCLES 2014

How to evaluate a test

Claims ����

Evidence ����

Weakness/strength of claims

Building a validity argument

© UCLES 2014

Useful framework for evaluating tests

Cyril Weir’s (2005) ‘socio-cognitive framework’ for

validating language tests

� a framework of questions about the validity of

language tests

© UCLES 2014

Weir’s (2005) Socio-cognitive framework

4. How far can we

depend on the

scores of the test?

2. Are the cognitive

processes required to

complete the tasks

appropriate?

1. How are TT

characteristics catered

for by this test?

6. What external

evidence is there

beyond test scores

that the test is doing a

good job?

3. Are the

characteristics

of the test

tasks and

administration

fair to TTs?

5. What

effects does

the test have

on its

stakeholders? © UCLES 2014

Topical issues in assessment

• Language testing’s links to educational / social / political

policy.

• Public accountability and ethical behaviour of test

producers and users.

• Technological advances are reshaping the design and

delivery of language tests.

• Growing understanding of language acquisition,

development and use and advances in linguistics are

affecting how we define and assess language

proficiency.

• We are reconceptualising communication in relation to

pedagogy and assessment.

© UCLES 2014

To summarise:

A test should:• Consistently provide accurate measures of precisely

the abilities in which we are interested (VALIDITY & RELIABILITY)

• Have a beneficial effect on teaching and learning (IMPACT)

• Be economical in terms of time and money (PRACTICALITY)

Be FIT FOR PURPOSEDifferent test purposes:

to measure communicative language abilityto measure lexico-grammatical knowledgeto measure success in achieving course objectivesto assist in placement of students into different groups G

© UCLES 2014

5. Impact of assessment

• Who is affected by assessment in general?

• What are some potential benefits?

• What challenges can you think of?

• What can we do to alleviate these challenges? One

approach links learning, teaching and assessment

© UCLES 2014

6. Teaching, Learning and

Assessment: How do they connect?

• Through Learning Oriented Assessment

• Assessment for learning as well as assessment of

learning, involving feedback and feed forward

© UCLES 2014

Locating LOA within the educational

landscape

“LOA

Ga kind of formative assessment?”

...a kind of summative assessment?”

G?”

© UCLES 2014

Formative assessment G

© UCLES 2014

• is on-going assessment during a period of study

• responds to the evolving needs of the learner

• relates to identified learning objectives

• implies scaffolding learning to help learners

reach identified learning objectives

Formative assessment

© UCLES 2014

Which of these activities can be part of

formative assessment?

A. Observing learners during a speaking activity and

identifying points for further development.

B. Setting regular progress tests.

C. Noting down learners’ mistakes in a writing activity to do

further work on in class.

D. Evaluating learners’ responses to a listening activity.

Which elements are they having difficulty with, what do

they need to work on and what have they understood

well?

E. All of the above.

© UCLES 2014

Formative assessment: pros & cons

• Has a natural affiliation with teaching and

learning

• Emphasises interaction, support and

development

BUTG

• Often based on the teacher’s intuition

• Seen as lacking reliability & validity

© UCLES 2014

Summative assessment

© UCLES 2014

• involves tests at the end of a period of study

(e.g. unit, week, term, course)

• is typically linked to and looks back at the

syllabus

• is an indication of the learner’s ability or

overall proficiency

• often used for certification purposes

Summative assessment

© UCLES 2014

• results can be generalised beyond test

context

• tends to be designed with validity and

reliability in mind

BUT

• is often perceived as just “grading”

• could involve “teaching to the test”

Summative assessment: pros & cons

© UCLES 2014

Evaluating what has happened before

A kind of judgement

Guiding what happens next

A kind of purpose

Formative Summative

Traditional distinction

© UCLES 2014

Multiple functions of assessment

“Every act of assessment we devise or have a

role in implementing has more than one

purpose. If we do not pay attention to these

multiple purposes we are in danger of

inadvertently sabotaging one or more of them '

Assessment has to serve double duty.”

Boud (2000:159)

© UCLES 2014

Provide

evidence/feedback

Support learning =

Learning oriented

assessment

Classroom-based

assessment

Large-scale standardised

assessment

Whatever form assessment takesG

© UCLES 2014

Strengthening the link between

learning, teaching and assessment

“G for all assessments whether predominantly

summative or formative in function a key aim is to

promote productive student learning.” Carless (2009:80)

Defining LOA:

“[LOA] involves the collection and interpretation of

evidence about performance so that judgements

can be made about further language development”

G to promote learning

Purpura (2004:236)© UCLES 2014

Learning Oriented Assessment

• captures the centrality of learning within assessment (not an afterthought)

• challenges the traditional view that exams are external and summative

© UCLES 2014

A Model of LOA

• the macro level – framing educational goals

and evaluating outcomes (policy context)

• the micro level – individual learning

interactions which take place within and

outside of the classroom (learning

environment)

© UCLES 2014

Key features

LOA relies on a systematic approach to

assessment:

• Quality/appropriacy of evidence gathered and

interpretation made

• Appropriacy of feedback/modifications to instruction

• Development of learner autonomy / life-long learning

• An alignment between external measures and

classroom-based assessment

© UCLES 2014

What evidence of learning do you use?

When and how do you collect it?

How do you ‘know’ that it is useful evidence?

Record

Collecting evidence

© UCLES 2014

Collecting evidence

Use multiple sources (triangulation)

• scores (tests/quizzes)

• observation (performance)

• past experiences with similar learners

• learners themselves

• engaging in action research

Evidence tells you something about learner ability

© UCLES 2014

Learning Oriented Assessment

Integrating different

forms of assessment

Basing assessment on

learning objectives

Using assessment to

support learning

Single information

system for:

Individual feedback

Monitoring progress

Student and class profiles

End of course reports

© UCLES 2014

Who benefits from LOA?

• multiple examples of evidence for learning

• clear evidence of progress towards learning objectives

• work at right level

• receive relevant and timely feedback

• become independent learners

• monitor progress towards targets

• valid, reliable and recognised certification

© UCLES 2014

8. Applying assessment

knowledge in your context

1. We have to obtain knowledge and then apply it to make

sense of what we have learnt.

2. One example is through undertaking Action Research.

© UCLES 2014

What is Action Research?

AR involves teachers exploring a specific challenge

G that they have identified

G in their own teaching context

G through several cycles of research

AR is a form of teacher research.

© UCLES 2014

“It is practitioners in their immediate social

situation who are best placed to understand,

examine and innovate in curriculum-related

issues and challenges.”

(Burns, 2011)

Why should teachers do Action

Research?

© UCLES 2014

Examples of AR projects

• Developing reading skills of Arabic students

• Formative assessment in a Web 2.0 environment

• Student attitudes to EAP grammar instruction

• Encouraging students to become independent

learners through self-assessment and reflection

• Using writing rubrics to develop learner autonomy

• Creating a blog for self-assessment

• Introducing learning portfolios

© UCLES 2014

Summary of impact of Action

Research

• teaching/research skills &

knowledge

• professional development

• longer-term impact

• new reputational dimension

• rejuvination of practice

• career options

• programme dissemination

• strengthened practice

• engagement & motivation

• enhanced PD & professionalism

• ‘ripple-effect’

57

Sectoral

Institutional

Individual

© UCLES 2014

9. Find out more: Assessment

resources

1. Self-access materials including webinars and videos and

materials produced by exam boards (see following slides for

Cambridge English examples).

2. Online or F2F courses run by: ALTE www.alte.org/events,

EALTA www.ealta.eu.org/events, CET

www.cambridgeenglishteacher.org/courses or CAN

www.cambridgeassessment.org.uk/events/cppa

3. Join a discussion list or online group for teachers or

researchers, e.g. ESOL-RESEARCH@JISCMAIL.AC.UK, or

try links at http://iteslj.org/links/

4. Talk to your colleagues about assessment.

5. Reflect on your/your learners’ experiences.

6. Find out more about action research, consider doing your

own project on an assessment-related issue.© UCLES 2014

Cambridge English resources

• Webinars for teachers:

www.cambridgeenglish.org/webinars/

• Cambridge English TV:

www.youtube.com/user/cambridgeenglishtv

• Cambridge English Teacher:

www.cambridgeenglishteacher.org/

• Teacher support website:

www.teachers.cambridgeesol.org

• Principles of Good Practice booklet:

www.cambridgeenglish.org/research-and-

validation/quality-and-accountability/

© UCLES 2014

Find out more: Action Research

Some key writers:

• Simon Borg, resources at: http://simon-borg.co.uk/free-

sources-of-language-teaching-research

• Anne Burns, start with her Action Research video at:

http://professoranneburns.com/arvideo.htm

Online resources:

• Action Research in Education course -

www.edu.plymouth.ac.uk/resined/actionresearch/arhome.htm

• Research Notes 44, 48, 53 (reports of Australian projects):

www.cambridgeenglish.org/researchnotes/

• Recent webinar at www.CambridgeEnglishTeacher.org

© UCLES 2014

Find out more: LOA

Online resources:

• Cambridge English approach, resources and FAQs:

www.cambridgeenglish.org/research-and-

validation/fitness-for-purpose/

• Webinar and videos on Cambridge English TV (You Tube)

© UCLES 2014

What have we achieved today?

• Explored some of the key concepts of assessment and

our own experiences of assessment, before thinking

about its impact on teaching/learning and society.

• Thought about how learners and teachers could benefit

from a better understanding and use of assessment,

through a Learning Oriented Assessment approach or

undertaking some Action Research.

• Looked at some of the resources available.

• Laid the groundwork to enable you to think of how

assessment literacy can be applied in your context.

. © UCLES 2014

References 1ALTE Multilingual Glossary of Language Testing Terms. (1998) UCLES /

Cambridge University Press.

Bachman, L and Palmer A (1996) Language Testing in Practice. Oxford: Oxford

University Press. (See Chapters 2 and 4)

Boud, D (2000) Sustainable assessment: rethinking assessment for the learning

society. Studies in Continuing Education, 22 (2) 151-167.

Boud, D (2006) Foreword. In How Assessment Supports Learning: learning-

oriented assessment in action by Carless, D, Joughlin, G, Liu, N F, &

Associates, Hong Kong: Hong Kong University Press.

Burns, A (2010) Doing action research in English language teaching. A guide for

practitioners, New York: Routledge.

Burns, A (2012) Teacher research in a national programme: Impact and

implications, Research Notes 48, 3–7.

References 2Carless, D (2007) Learning-oriented assessment: conceptual issues and

practical implications. Innovations in Education and Teaching International , 44

(1), 57–66.

Carless, D (2009) Learning-oriented Assessment: Principles and Practice and a

Project. In Meyer, L H, Davidson, H, Anderson, R, Fletcher, P M, Johnston &

Rees, M (Eds), Tertiary Assessment & Higher Education Student Outcomes:

Policy, Practice & Research. Wellington: New Zealand.

Hughes, A (1989/2003) Testing for Language Teachers. Cambridge:

Cambridge University Press.

McNamara, T (2000) Language Testing. Oxford: Oxford University Press.

© UCLES 2014

References 3

Purpura, J (2004). Learning-oriented assessments of grammatical ability. In

Assessing Grammar, Cambridge: Cambridge University Press.

Weir, C and Milanovic, M (2003) Continuity and Innovation: Revising the

Cambridge Proficiency in English Examination 1913-2002. UCLES/Cambridge

University Press.

Weir, C J (2005) Language Testing and Validation: An evidence-based

approach. Oxford: Palgrave Macmillan.

© UCLES 2014