e-AFFECT: What have we learnt?

Dr Anne JonesCentre for Educational Development

Approach to project

after Cooperrider & Whitney (2005)

Phase 1 •3 programme areas•65 staff and 1722 students

Phase 2 •7 programme areas•95 staff and 1482 students

Phase 3 •4 programme areas•95 staff and 1287 students

PROJECT MANAGEMENT

DISSEMINATION

Phas

e 1

Phas

e 2

Phas

e 3

2011-2012 2012-2013 2013-2014

Baseline Plan Develop

Baseline Plan Develop

Baseline Plan Develop

Evaluation/refinement

Evaluation/refinement

Evaluation/refinement

Intervention 1 Intervention 2

Intervention 1 Intervention 2

Evaluation Evaluation

Evaluation Evaluation

On going activity

EVALUATION

EMBEDDING

Participating programmesAHSS EPS MHLS

Business Management Civil Engineering Biomedical SciencesDrama Computer Science Midwifery

English Environmental Planning Pharmacy (baseline only)

Film Studies Psychology

Law

Music

Social Policy

Lessons about Assessment & Feedback business processes

Managing assessment and feedback

• Not all Schools have assessment policies • Sometimes there is one for an area within a School• Some are very detailed • Others are very prescriptive

• Hard copy versus electronic submission of coursework• Variation in the release of feedback and marks on coursework• Variation in the provision of feedback on exams

Lessons about Assessment & Feedback timelines

Assessment and feedback timelines

Represents medium stakes assessment

Represents low stakes assessment

Represents high stakes assessment

F

F

F

Represents low stakes formative assessment

Represents medium stakes formative assessment

Represents high stakes formative assessment

Where students have opportunity to act on one or more formative/summative assessments in a

future a summative assessment

Adapted from the ESCAPE project

2 4 6 8 10 12 week

Assessment and feedback timelines

2 4 6 8 10 12 week

2 4 6 8 10 12 week

2 4 6 8 10 12 week

SEMESTER 1 LEVEL 1

SEMESTER 2 LEVEL 1

2 4 6 8 10 12 week

2 4 6 8 10 12 week

2 4 6 8 10 12 week

2 4 6 8 10 12 week

2 4 6 8 10 12 week

2 4 6 8 10 12 week

F

F

F

F

F

F

F

F

F

SEMESTER 2 LEVEL 2

Level 1

2 4 6 8 10 12 week

2 4 6 8 10 12 week

2 4 6 8 10 12 week

Level 3

2 4 6 8 10 12 week

2 4 6 8 10 12 week

2 4 6 8 10 12 week

F

F F

Level 2

2 4 6 8 10 12 week

2 4 6 8 10 12 week

2 4 6 8 10 12 week

2 4 6 8 10 12 week

Attendance and presentation of weekly

seminar – one chosen at random

Level 3

2 4 6 8 10 12 week

2 4 6 8 10 12 week

F

F

F

F

F

F

Student and staff views

Student and staff questionnaires

Student questionnaire• Study and learning strategies• Extent to which practice is

occurring• About themselves

• Response rate = 20.4%

Staff questionnaire• Assessment and feedback

strategies• Extent to which practice is

occurring• About themselves

• Response rate = 47.3%

Student profileFaculty Year of study Gender Age

AHSS 28.7% Level 1 36.9% Female 74.5% 18-22 years

77.1%

EPS 47.2% Level 2 33.5% Male 25.7% 23+ years

22.9%

MHLS 24.1% Level 3+ 29.6%

Student study and learning strategies

Study and learning strategies

• Amount and distribution of study effort 6 statements• Assignments and learning 8 statements• Quantity and timing of feedback 6 statements• Quality of feedback 6 statements• What students do with feedback 6 statements• The examination and learning 6 statements

Based on Gibbs and Simpson (2004)

Analysis Faculty Year of study Gender Age

Amount and distribution of study effort 6 significant 5 significant 2 significant

Assignments and learning 5 significant 1 significant 2 significant 4 significant

Quantity and timing of feedback 6 significant 2 significant 1 significant 4 significant

Quality of feedback 6 significant 6 significant 4 significant

What you do with the feedback4 significant 5 significant 2 significant 5 significant

The examination and learning 6 significant 2 significant 1 significant 4 significant

Significance p ≤ 0.05

Key findings Faculty Year of

studyGender Age

On this course, it is possible to do quite well without studying much.

Moderately strong ns Moderately

strong ns

In weeks when the assignments are due I put in many more hours. Weak ns Moderate Weak

The assignments are not very challenging.Moderate Weak Strong Weak

The feedback does not help me with any subsequent assignments.

Moderately strong Moderate ns ns

I understand things better as a result of the exam. Weak ns ns Moderate

Student and staff perceptions of practice

• 27 statements based on ATLAB questionnaire (Whitelock and Cross, 2011)• Fully occurring – not in a position to say• Assessment criteria are written and used• All marking criteria are shared with students in advance and

feedback refers to them• Feedback is intended to acknowledge, consolidate and promote

student learning• Students get feedback which corrects errors and supplies further

information• Students have opportunities for cooperative and collaborative

assessment

Analysis

Students Student - Staff

Faculty Year of study

Gender Age

Number of significant statements

17 16 4 6 23

Significance p ≤ 0.05

Students

Faculty Year of study

Gender Age

Students have opportunities for cooperative and collaborative assessment

Moderately strong Moderate ns Weak

Assessment activities are situated within real-world contexts (i.e. they reflect the problems, tasks or competencies a practitioner would face)

Moderate ns ns ns

Students and staff Staff - studentsStudents have opportunities for a feedback dialogue (peer or teacher-student) around assessment tasks on the course

Moderate

Assessment includes the use of a portfolio or similar practice Moderately strong

A variety of communication channels are used to provide feedback to students

Moderately strong

Students are told what type of feedback they can expect and when and how they will receive it

Moderate

Lessons from the methodology

Appreciative Inquiry approach

after Cooperrider & Whitney (2005)

Engendering dialogue – time and space

Bespoke technology workshops

Lessons from activities

Opportunity to act on feedback

Help clarify good performance

Development of self-assessment and reflection

Civil Engineering 1

0-34 35-39 40-49 50-59 60-69 70-79 80-1000.0

5.0

10.0

15.0

20.0

25.0

30.0

35.0

2011-12 2012-13 2013-14

%

Opportunity to act on feedback

Help clarify good performance

Development of self-assessment and reflection

Encourage time and effort on challenging tasks

0-34 35-39 40-49 50-59 60-69 70-79 80-1000

5

10

15

20

25

30

35

40

45

2011-12 2012-13

% a

chie

ving

Civil Engineering 2

also includevideo and audio stimuli, to target

particular phoneticfeatures.

‘Fast feedback’s really important in phonetics. You need to establish good habits right from the start.’

Phonetics

Opportunity to act on feedback

Help clarify good performance

Development of self-assessment and reflection

Encourage time and effort on challenging tasks

Encourage interaction and dialogue around learning0-34 35-39 40-49 50-59 60-69 70-79 80-100

0.00

10.00

20.00

30.00

40.00

50.00

60.00

2011-12 2012-13 2013-14

%

Phase 2 activities• Some of the activities are represented in the Marketplace• Other activities include:• Marking and feedback workshops with students• Students indicating how they have used feedback• The use of Vimeo for uploading films• E-submission/marking/feedback• QuestionMark Perception for formative and summative activities• Peer review using PeerMark

Phase 3 – planned activities

• Development of feedback comments bank• Student generated MCQs using PeerWise• QuestionMark Perception for online objective testing• Vodcasts• Screen capture feedback on draft work • Clarification of assessment criteria and standards – guides and

workshops

The Marketplace 2014

– Collaborate project

A Cross-Institutional Approach to Assessment Redesign:Embedding Work-Integrated Assessments within the Curriculum

Designing Work-Integrated Assessments

Dimensions of a Work-Integrated Assessment

Aligning Technologies to the Model: ‘Top Trumps’ Using a Model for Work-Integrated Assessment:Six Dimensions

School of History and Anthropology: Student generated MCQs

‘The lecturer got us to sign up to a site called 'Peerwise' which involves students creating multiple choice

questions and the other students are able to answer them. I thought this was a brilliant idea as it’s great for

revision.’ Module evaluation, Deviance History, QUB, 2013

Centre for Medical EducationA strategy to help students to actively engage with feedback provided: sharing experience from the undergraduate medical student selected component (SSC) programme

Student responses

I constructed a more balanced argument…

I incorporated more cases to illustrate..

I cut down… to allow more room for….

I put more emphasis on this point….

Examples of feedback

justify reasons for and against this ….

make greater use of relevant cases and sources

to support discussion here….

background should be more focused on…

e-AFFECT

...Shared and jointly owned standards leading

to consistency of feedback

C described his practice in giving feedback when students

previously had had to provide a reflection on the strengths and weaknesses of their work when

they requested feedback. He read their reflective statements

and incorporated these comments into the feedback he had already written. A dialogue

was thus developed.

QUMS – Queens Online assignment tool

Environmental Planning - JingImplementation

1. As a method of delivering formative feedback on visual work to enhance skill development

2. To enhance skill development with Jing Screencast Tutorials

Environmental Planning – Acrobat Pro

Implementation

Annotation over drawings is a standard way of giving feedback for design based modules. Students can access this feedback through their tablets, computers or smart phones.

The module consisted of three stages and Acrobat Pro was used to give feedback for the first of the three submissions in the module.

Environmental Planning - VoiceThreadImplementation

For this module we made four 15 minute videos on a range of planning subjects and asked specific questions that students need to answer. All students can see all the comments that other students and the tutors make.

Drama – VoiceThread for student engagementHow did I use it?

• PowerPoint lectures were put into VoiceThread with my commentary as a voiceover.

• Students were asked to watch the lectures before the class.

• Students were required to add substantive comments or questions to the VoiceThread.

• 15% of the marks were allocated for this activity

Centre for Medical Education – Virtual Patients

Turning flat word-processed patient cases into interactive learning objects

Storyboard

Computer Science - Offline objective marking

105 Level 3 students take a series of assessed practicals and a final assessment in QMP. As the students work through the practicals they complete a worksheet in QMP.

The teaching team used Excel to score the responses

References• Cooperrider, D & Whitney, D 2005 Appreciative Inquiry: a positive

revolution in change Berrett-Koehler Publishers, Inc. San Francisco• Gibbs, G & Simpson, C (2004) Conditions under which assessment

supports learning, Learning and Teaching in Higher Education 1, 3-31 • Nicol, D (2009)

Transforming assessment and feedback: enhancing integration and empowerment in the first year, The Quality Assurance Agency for Higher Education, Mansfield• Whitelock, D. & Cross, S. (2011). Assessment Benchmarking:

Accumulating and accelerating institutional know-how for best practice. International Journal of e-Assessment (IJEA), Vol 1. No. 1 http://journals.sfu.ca/ijea/index.php/journal/article/view/18

Resources

• e-AFFECT Jisc Design Studio http://jiscdesignstudio.pbworks.com/w/page/50671059/e-AFFECT%20Project • e-AFFEECT Resources Jisc Design Studio http://

jiscdesignstudio.pbworks.com/w/page/68656954/e-AFFECT%20Resources • e-AFFECT blog http://blogs.qub.ac.uk/e-affect/ • REAP project http://www.reap.ac.uk/ • Viewpoints Project Jisc Design Studio http://

jiscdesignstudio.pbworks.com/w/page/29227748/Viewpoints%20project and http://wiki.ulster.ac.uk/display/VPR/Home