Client - meeting - _____ - 1

Student perceptions of Student perceptions of assessment at Peninsula assessment at Peninsula

Medical SchoolMedical School

Dr. Karen MattickDr. Karen Mattick

Senior Lecturer, Senior Lecturer, Peninsula College of Peninsula College of Medicine and DentistryMedicine and Dentistry

BackgroundBackground

Modern undergraduate education aims to provide Modern undergraduate education aims to provide an assessment environment that promotes an assessment environment that promotes high high quality learningquality learning

This is not easy to achieve - student perceptions This is not easy to achieve - student perceptions of learning and assessment environments are of learning and assessment environments are hard to predict hard to predict

Experienced educators have introduced changes Experienced educators have introduced changes in assessment designed to promote high quality in assessment designed to promote high quality learning that have had the exact opposite effect!learning that have had the exact opposite effect!

Need to understand the impact of assessment on Need to understand the impact of assessment on what is learnt and how, and use this to promote high what is learnt and how, and use this to promote high quality learning and constructive alignment between quality learning and constructive alignment between teaching and assessment. teaching and assessment.

Great complexity of factors that might influence high Great complexity of factors that might influence high quality learning. Barriers to achieving it almost quality learning. Barriers to achieving it almost inevitable. inevitable.

In 2002, we designed an evidenced based learning In 2002, we designed an evidenced based learning and assessment environment that we believed should and assessment environment that we believed should promote high quality learningpromote high quality learning

Could it stand the ultimate test…?Could it stand the ultimate test…?

Our students…!Our students…!

Our context – learning Our context – learning environmentenvironment

In years 1 and 2, two week case unitsIn years 1 and 2, two week case units PBL sessionsPBL sessions clinical placements and jigsaw sessionsclinical placements and jigsaw sessions biomedical science tutorialsbiomedical science tutorials clinical skills trainingclinical skills training small number of lectures.small number of lectures.

Spiral curriculumSpiral curriculum Emphasis on self-directed learning Emphasis on self-directed learning Two week Special Study Units (SSUs)Two week Special Study Units (SSUs)

research ‘non-core’ topics in depth research ‘non-core’ topics in depth publication-style report. publication-style report.

Our context – assessment Our context – assessment environmentenvironment

Assessment characterised by a ‘little but often’ approach. Assessment characterised by a ‘little but often’ approach. Applied medical knowledge assessed by a ‘progress test’Applied medical knowledge assessed by a ‘progress test’

sat quarterlysat quarterly featuring clinical vignettes and questions at graduation featuring clinical vignettes and questions at graduation

levellevel any topic covered in any test. any topic covered in any test.

Clinical skillsClinical skills regular competency assessmentsregular competency assessments ISCE at the end of the second year. ISCE at the end of the second year.

Team working and professional behavioursTeam working and professional behaviours tutor and peer judgementstutor and peer judgements reflective portfolios.reflective portfolios.

What we didWhat we did

Created a definition of high quality learning for Created a definition of high quality learning for undergraduate medical studentsundergraduate medical students

Interviewed 15 second year medical students in Interviewed 15 second year medical students in 2004 about their approaches to learning, and 2004 about their approaches to learning, and how the PMS learning and assessment how the PMS learning and assessment environment affected thisenvironment affected this

Thematic analysis of interview dataThematic analysis of interview data

Definition of high quality Definition of high quality learninglearning

The latest conceptions of a deep approach to The latest conceptions of a deep approach to studying (learning through interest with an studying (learning through interest with an intention to understand the material covered)intention to understand the material covered)

Combined with broader metacognitive attributes Combined with broader metacognitive attributes that include aspects of the strategic approach that include aspects of the strategic approach described previously (e.g. personal organisation, described previously (e.g. personal organisation, effort monitoring, effort regulation)effort monitoring, effort regulation)

And characteristics perceived as desirable for And characteristics perceived as desirable for trainee doctors (e.g. reflective learning).trainee doctors (e.g. reflective learning).

What we foundWhat we found

Facilitating high quality Facilitating high quality learninglearning

Inhibiting high quality Inhibiting high quality learninglearning

Learning in contextLearning in context Perceived quantity of Perceived quantity of informationinformation

Self directed learningSelf directed learning 1. Perceived lack of feedback 1. Perceived lack of feedback from assessmentfrom assessment

Spiral curriculumSpiral curriculum Disease frameworksDisease frameworks

2. Progress test2. Progress test AnatomyAnatomy

2. Progress test2. Progress test

3. Summative ISCE3. Summative ISCE

1. Perceived lack of assessment 1. Perceived lack of assessment feedbackfeedback

‘‘Little but often’ approach sometimes perceived as an Little but often’ approach sometimes perceived as an absence of useful feedback on progress. absence of useful feedback on progress.

I don’t know if I’m doing enough now. I seem to be doing all I don’t know if I’m doing enough now. I seem to be doing all right, so I think I’m learning enough. But there’s nothing right, so I think I’m learning enough. But there’s nothing that tells you whether you are or not, I find that quite that tells you whether you are or not, I find that quite difficult.difficult. Participant 13; female; school leaver. Participant 13; female; school leaver.

This was despite (per academic year)This was despite (per academic year) 4 progress test results4 progress test results 4 SSU project reports4 SSU project reports numerous skills competencies numerous skills competencies 10 PPD judgements and 2 portfolio analyses scores. 10 PPD judgements and 2 portfolio analyses scores.

So why the mismatch in perception?So why the mismatch in perception? format, timing or level of detail of feedback?format, timing or level of detail of feedback? Importance of feedback within a curriculum that promotes Importance of feedback within a curriculum that promotes

SDL and features reduced contact time with teachers?SDL and features reduced contact time with teachers?

Whilst the perceived lack of assessment feedback Whilst the perceived lack of assessment feedback seemed anxiety-inducing, it deterred a focus on seemed anxiety-inducing, it deterred a focus on assessment.assessment.

I kind of had to change the way I motivate myself… I’ve I kind of had to change the way I motivate myself… I’ve had to do it by myself rather than let other people had to do it by myself rather than let other people motivate me I think. motivate me I think. Participant 14; female; school leaver.Participant 14; female; school leaver.

Participants reported becoming motivated by intrinsic Participants reported becoming motivated by intrinsic factors - a desirable development for lifelong learning factors - a desirable development for lifelong learning

where there will be no formal assessment.where there will be no formal assessment.

2. Progress test – for the majority2. Progress test – for the majority

Most students reported that understanding and application of Most students reported that understanding and application of information was needed due to the clinical vignettes. information was needed due to the clinical vignettes.

Students believed the test rewarded those working Students believed the test rewarded those working consistently. consistently.

The fact that it could test any topic at any time deterred short-The fact that it could test any topic at any time deterred short-term memory strategies and ‘cramming’.term memory strategies and ‘cramming’.

I think that’s what makes progress tests so difficult is I think that’s what makes progress tests so difficult is because you’ve got to be learning constantly, you can’t because you’ve got to be learning constantly, you can’t like have a cram. like have a cram. Participant 5; female; school leaver.Participant 5; female; school leaver.

Progress test already known to reward students with a deep Progress test already known to reward students with a deep approach when starting medical school (Mattick et al. 2004). approach when starting medical school (Mattick et al. 2004).

Students that were highly sensitive to task demands and able Students that were highly sensitive to task demands and able to adapt to the learning environment tended to succeed. to adapt to the learning environment tended to succeed.

2. Progress test – for a 2. Progress test – for a minorityminority

But, when students became excessively anxious about it, the But, when students became excessively anxious about it, the progress test was sometimes a barrier to high quality progress test was sometimes a barrier to high quality learning. learning.

Anxiety seemed to drive students to do more work in terms Anxiety seemed to drive students to do more work in terms of quantity but not necessarily the right kind of workof quantity but not necessarily the right kind of work

If you’re tensed up you need to really rote learn a lot of it. If you’re tensed up you need to really rote learn a lot of it. Participant 8; male; school leaver.Participant 8; male; school leaver.

Previous work indicates that underperforming students often Previous work indicates that underperforming students often adopted or retained undesirable approaches to learning in adopted or retained undesirable approaches to learning in response to poor grades (Mattick, Dennis and Bligh, response to poor grades (Mattick, Dennis and Bligh, unpublished data). unpublished data).

Underperforming students need help and support to become Underperforming students need help and support to become accomplished learners and successful in HE and beyond. accomplished learners and successful in HE and beyond.

Remediation should aim to challenge why, what and how Remediation should aim to challenge why, what and how individuals study.individuals study.

3. Summative ISCE3. Summative ISCE

The ISCE constituted a single exam to pass for The ISCE constituted a single exam to pass for progression to year 3 (i.e. not ‘little but often’). progression to year 3 (i.e. not ‘little but often’).

High stakes and anxiety inducing. High stakes and anxiety inducing. I ended up studying a couple of things that weren’t in the, I ended up studying a couple of things that weren’t in the,

in the ISCE because of the general panic that was out in the ISCE because of the general panic that was out there. It’s hard not to be influenced by that. there. It’s hard not to be influenced by that. Participant Participant 10; male; mature entry.10; male; mature entry.

But, many participants said they liked this - the ISCE was But, many participants said they liked this - the ISCE was a good motivator for learning and consolidating the skills a good motivator for learning and consolidating the skills

learned and practiced over the last two years.learned and practiced over the last two years.

Participants felt a cramming-type approach was an appropriate Participants felt a cramming-type approach was an appropriate way to prepare for the ISCE. way to prepare for the ISCE.

The ISCE is like a final exam, as such, so I think if you The ISCE is like a final exam, as such, so I think if you wanted to you could really cram for that. wanted to you could really cram for that. Participant 8; Participant 8; male; school leaver.male; school leaver.

Was this because:Was this because: it was a practical exam for which they could practice by it was a practical exam for which they could practice by

repetition repetition of the high anxiety it invoked. of the high anxiety it invoked.

Participants perceived the advice from teachers about preparing Participants perceived the advice from teachers about preparing for skills-based examinations (‘for skills-based examinations (‘practice, practice, practicepractice, practice, practice’) to ’) to conflict with that for knowledge-based assessment (‘conflict with that for knowledge-based assessment (‘avoid rote avoid rote learninglearning’). ’).

Questions, reflections.Questions, reflections.

These barriers to high quality learning were interesting These barriers to high quality learning were interesting since they were unintended consequences of the since they were unintended consequences of the curriculum.curriculum.

We don’t think we could predict them without studies We don’t think we could predict them without studies such as these.such as these.

We have now introduced some measures to address the We have now introduced some measures to address the barriers identified (but possibly also introduced more?!).barriers identified (but possibly also introduced more?!).

Any questions about our study? Any questions about our study?

What about your own settings?What about your own settings?