Developing assessment patterns that work through TESTA

Professor Tansy JessopTESTA Workshop

University College Dublin10 February 2017

The plan for this session

1. Brief overview of TESTA2. Two case study puzzles3. Assessment patterns which help

students to learn

Mixed methods approach

Programme Team

Meeting

Assessment Experience

Questionnaire(AEQ)

TESTAProgramme

Audit

Student Focus Groups

Growth of TESTAWidespread use

TESTA….

“…is a way of thinking about assessment and feedback”

Graham Gibbs

It enables you to see the whole elephant

Defining the terms

• Summative assessment carries a grade which counts toward the degree classification.

• Formative assessment does not count towards the degree (either pass/fail or a grade), elicits comments and is required to be done by all students.

TESTA Case Study X: what’s going on?

• Plenty of varieties of assessment, no exams• Reasonable amount of formative assessment (14 x)• 33 summative assessments• Masses of written feedback on assignments (15,000 words)• Learning outcomes and criteria clearly specified….looks like a ‘model’ assessment environment

But students:• Don’t put in a lot of effort & distribute their effort across few topics• Don’t think there is a lot of feedback or that it very useful, and

don’t make use of it• Don’t think it is at all clear what the goals and standards are• …are unhappy

Case Study Y: what’s going on?

• 35 summative assessments• No specified formative assessment• Learning outcomes and criteria wordy and woolly• Marking by connoisseurship, tacit, professional judgements….looks like a problematic assessment environment

But students:• Put in a lot of effort and distribute their effort across topics• Have a very clear idea of goals and standards • Are able to evaluate their work and have a good idea of how to

‘close the gap’

Assessment patterns which work

1. Part of principled whole curriculum design 2. Plenty of formative assessment3. Students distribute effort and spend time on task4. Challenging and high expectations5. Feedback connects across modules6. Students interact with and internalise standards

Assessment patterns (n=73 programmes)Characteristic Low Medium High

Volume of summative assessment

Below 33 40-48 More than 48

Volume of formative only Below 1 5-19 More than 19

% of tasks by examinations Below 11% 22-31% More than 31%

Variety of assessment methods

Below 8 11-15 More than 15

Written feedback in words Less than 3,800 6,000-7,600 More than 7,600

Deep and Surface Learning (Marton and Saljo (1976)

Deep Learning• Meaning• Concepts• Active learning• Generating knowledge• Relationship new and

previous knowledge• Real-world learning

Surface Learning• External purpose• Topics• Passive process• Reproducing knowledge• Isolated and

disconnected knowledge• Artificial learning

• High summative on most UK, NZ and Indian degrees (range 12 -227 in UK)

• Pedagogies of control

• Low formative: ratio of 1:8 summative to formative

• Weakly practised and understood

1. Curriculum design question

Does IKEA 101 work for complex learning?

..has led to over-emphasising knowing (Barnett and Coate 2005)

• Knowing is about content• Acting is about becoming

a historian, actor, psychologist, or philosopher

• Being is about understanding yourself, orienting yourself and relating your knowledge and action to the world

Knowing

Being

Acting

The best approach from the student’s perspective is to focus on concepts. I’m sorry to break it to you, but your students are not going to remember 90 per cent – possibly 99 per cent – of what you teach them unless it’s conceptual…. when broad, over-arching connections are made, education occurs. Most details are only a necessary means to that end.

http://www.timeshighereducation.co.uk/features/a-students-lecture-to-rofessors/2013238.fullarticle#.U3orx_f9xWc.twitter

A student’s lecture to her professor

Assessment is mainly sort of the topical knowledge and the topics never relate. We'll never do something again that we’ve already studied, like we learn something and then just move on

(TESTA focus group data).

…and fragmenting content…

2. Plenty of formative but…

“Innovations that include strengthening the practice of formative assessment produce significant and often substantial learning gains”

(Black and Wiliam, 1998, p.40).

Take five

• Which quote strikes you most and why?

• What do students value about formative assessment?

• What are the challenges of doing formative?

It was just a practice and didn’t really matter what we did and we could learn from mistakes so that was quite useful.

He’s such a better essay writer because he’s constantly writing. And we don’t, especially in the first year when we really don’t have anything to do. The amount of times formative assignments could have taken place…

It’s good to know you’re being graded because you take it more seriously.

I would probably work for tasks, but for a lot of people, if it’s not going to count towards your degree, why bother?

Students talking about formative

3 & 4. Effort and challenging tasks

• Chickering and Gamson (1987)

• Time-on-task (Gibbs 2004)

• Arum and Roksa (2011)

The problem is that…

• Summative drives student effort

• ‘Pedagogies of control’

• Too much marking for staff, too little effort by students

• Assess less, learn more

• Measure less, learn more

Typical student effort graph

Almost 50% of students in USA leave with no significant gains in academic writing, complex reasoning and critical thinking skills…

(Arum and Roksa, 2011)

Significant learning gains for students who 1) Read > 40 pages a week of academic writing

2) Write > 20 pages per semester for each unit

From this educational paradigm…

Transmission Model

Social Constructivist Model

ReferencesArum, R and Roksa, J. (2011) Academically adrift: Limited Learning on College Campuses. University of Chicago Press.Boud, D. and Molloy, E. (2013) ‘Rethinking models of feedback for learning: The challenge of design’, Assessment & Evaluation in Higher Education, 38(6), pp. 698–712. doi: 10.1080/02602938.2012.691462.Gibbs, G. & Simpson, C. (2004) Conditions r which assessment supports students' learning. Learning and Teaching in Higher Education. 1(1): 3-31.Harland, T., McLean, A., Wass, R., Miller, E. and Sim, K. N. (2014) ‘An assessment arms race and its fallout: High-stakes grading and the case for slow scholarship’, Assessment & Evaluation in Higher Jessop, T. and Tomas, C. 2016 The implications of programme assessment on student learning. Assessment and Evaluation in Higher Education. Published online 2 August 2016. Jessop, T. and Maleckar, B. (2014). The Influence of disciplinary assessment patterns on student learning: a comparative study. Studies in Higher Education. Published Online 27 August 2014 http://www.tandfonline.com/doi/abs/10.1080/03075079.2014.943170Jessop, T. , El Hakim, Y. and Gibbs, G. (2014) The whole is greater than the sum of its parts: a large-scale study of students’ learning in response to different assessment patterns. Assessment and Evaluation in Higher Education. 39(1) 73-88.Nicol, D. (2010) From monologue to dialogue: improving written feedback processes in mass higher education, Assessment & Evaluation in Higher Education, 35: 5, 501 – 517.O'Donovan, B , Price, M. and Rust, C. (2008) 'Developing student understanding of assessment standards: a nested hierarchy of approaches', Teaching in Higher Education, 13: 2, 205 — 217Sadler, D. R. (1989) ‘Formative assessment and the design of instructional systems’, Instructional Science, 18(2), pp. 119–144. doi: 10.1007/bf00117714.