Welcome Synthesis Report on Assessment and Feedback with Technology Enhancement.
-
Upload
bernadette-george -
Category
Documents
-
view
215 -
download
0
Transcript of Welcome Synthesis Report on Assessment and Feedback with Technology Enhancement.
Welcome
Synthesis Report on Assessment and Feedback with Technology Enhancement
Introduction to the report
Lester GilbertGary Wills
Denise WhitelockVeronica Gale
Synthesis report on assessment and feedback with technology enhancement (SRAFTE)
10th November 2010
Project purpose
Consult the academic community on useful references Seminar series Survey Advisors Invited contributors
Prioritise evidence-based references Synthesise main points For readers:
Academics using technology enhancement for assessment and feedback Learning technologists Managers of academic departments
Kirkpatrick
Level 1Reaction: opinions on look and feel, enjoyment, value, etc
Level 2Learning: able to demonstrate new understanding or skill
Level 3Performance change: new understanding or skill leads to demonstrable improvements in outcomes or outputs
Level 4Impact: improvements in outcomes lead to demonstrable impact upon team, section, department, group, organisation, … Kirkpatrick, D. (1976). Evaluation of training. Training and Development Handbook: A Guide to Human Resource Development, 2, 18-11.
Evidence categories
1a – peer reviewed and experimental method 1b – peer reviewed and effect sizes 2 – peer reviewed and quantified evidence 3 – case studies & frameworks in peer-reviewed journals and conference proceedings 4 – case studies in non-peer reviewed reports and guidance from experienced practitioners
Count
10
7
12
49
40
Total 118
Structure of the report
Whitelock et al, (2006)e-Assessment: Case Studies of Effective and Innovative Practice
Goals
Design
Creation
Testing
Piloting
Delivery
Capturing student responses
Feedback to students
Evaluation
Feedback to stakeholders
Themes
Pedagogy first, an integral part of the learning framework Literature focused on goals and design rather than tools Technology as an enabler:
More effective way of delivering learning design Allows new ways of capturing and comparing for assessments e.g.,
portfolios, group work, peer assessment For tutors providing more detailed feedback and feed forward Prompt delivery of feedback learning gains, encourage self-regulation
and reflection, can save tutor time, human touch Can save HEI time and cost after initial set-up and subject to underlying
drivers Some challenges to negative assumptions about technology Investment needed in staff development and support, creation and
delivery processes and QA
Technology
Large classes Stable content Significant risk reduction Significant cost reduction Lower confines of Bloom’s taxonomy Geographical & temporal dispersion
Success indicators1
1 Gilbert and Gale (2007)
Example contents
Tutors: challenging assumptions about disadvantages of technology use Dermo (2009), “e-Assessment and the student learning experience: A survey of student
perceptions of e-assessment” Evidence category 1b Use of technology did not disadvantage older or female students, was not found to be stressful for
students to use and was trusted by students
Learning technologists: answer matching accuracy Jordan and Mitchell (2009), “e-Assessment for learning? The potential of short-answer free-text
questions with tailored feedback” Evidence category 1a Answer matching demonstrated to be of similar or greater accuracy than specialist human markers
Senior Managers: cost savings and institutional change Ruedel, Whitelock, and Mackenzie (2007), “Key factors for effective organisation of e-assessment” Evidence category 3, patterns of facilitating factors and organisational structures
Main conclusions
Dearth of experimentally valid empirical evidence (category 1a)
Many descriptive case studies (categories 3 & 4)
Need more evidence at Kirkpatrick levels 2, 3, and 4
It’s the pedagogy, not the technology, that makes the difference
Nothing very surprising
Any questions? Comments?
Thanks!