Win Horn by Presentation 130104

8/6/2019 Win Horn by Presentation 130104

1/39

EFFICIENCY ANDEFFICIENCY AND

EFFECTIVENESS INEFFECTIVENESS IN

ASSESSMENTASSESSMENTStrategies for Streamlining AssessmentStrategies for Streamlining Assessment

Win HornbyWin HornbyTeaching FellowTeaching Fellow

The Robert Gordon UniversityThe Robert Gordon University


2/39

ObjectivesObjectives

To look at the changing environment withinTo look at the changing environment within

which assessment now takes placewhich assessment now takes place

To look at a framework for evaluatingTo look at a framework for evaluatingEfficiency and EffectivenessEfficiency and Effectiveness

To look at some empirical data based on aTo look at some empirical data based on a

survey of assessment practice in my ownsurvey of assessment practice in my ownuniversityuniversity

To propose a number of strategiesTo propose a number of strategies

@ Win Hornby 2004


3/39

The Importance of assessment?The Importance of assessment?

Students can with difficulty escape from theStudents can with difficulty escape from the

effects of poor teaching.......effects of poor teaching.......

they cannot ( by definition if they wish tothey cannot ( by definition if they wish tograduate) escape the effects of poorgraduate) escape the effects of poor

assessmentassessment

Boud (1995:35)Boud (1995:35)

@ Win Hornby 2004


4/39

Time spent on Assessment?Time spent on Assessment? RGU Annual Assessment Survey 2003RGU Annual Assessment Survey 2003

55--6 minutes per student6 minutes per student--credit point per annum!credit point per annum!

7575--90 minutes per student per annum taking a90 minutes per student per annum taking astandard 15 credit modulestandard 15 credit module

Gross this number up, university of 10,000Gross this number up, university of 10,000students, 4 year programmes, 120 credits p.astudents, 4 year programmes, 120 credits p.a

Collect in all assessments in one yearCollect in all assessments in one year

Get one member of staff to mark it, working 9 till 5Get one member of staff to mark it, working 9 till 5365 days a year!365 days a year!

It would takeIt would take 35 years35 years to mark!to mark!@ Win Hornby 2004


5/39

Aims of AssessmentAims of Assessment

Assessment has four main roles:Assessment has four main roles:

formativeformative,, to provide support for future learning;to provide support for future learning;

summative

summative,, to provide information aboutto provide information aboutperformance at the end of a course;performance at the end of a course;

certificationcertification,, selecting by means of qualificationselecting by means of qualification

andand

evaluativeevaluative,, a means by which stakeholders cana means by which stakeholders canjudge the effectiveness of the system as a whole.judge the effectiveness of the system as a whole.

@ Win Hornby 2004


6/39

The Changing EnvironmentThe Changing Environment

Changing role of the stakeholdersChanging role of the stakeholders

Changing resource baseChanging resource base

Changing course architectureChanging course architecture

@ Win Hornby 2004


7/39

The Changing EnvironmentThe Changing Environment

How have we adapted to this changingHow have we adapted to this changing

environment?environment? What changes in assessment practices?What changes in assessment practices?

Research Evidence?Research Evidence?

@ Win Hornby 2004


8/39

Rust (2000)Rust (2000)

.although the prevailing orthodoxy in UK highereducation is now to describe all courses in terms of

learning outcomes, assessment systems have not

changed.

Under current arrangements, rather than students

having to satisfactorily demonstrate each outcome

(which is surely what should logically be the case.)

marking still tends to be more subjective with theaggregation of positive and negative aspects of the work

resulting in many cases in fairly meaningless marks

being awardedwith 40% still being sufficient to pass


9/39

Research EvidenceResearch Evidence

Based on a survey of assessment practices in UKBased on a survey of assessment practices in UKuniversities in 1999 by Brown and Glasner theyuniversities in 1999 by Brown and Glasner theyfound:found:

90% of the assessment of a typical British Degree90% of the assessment of a typical British Degreeconsists of examinations PLUS tutor markedconsists of examinations PLUS tutor markedreports or essaysreports or essays

Recent evidence based on 2003 Survey ofRecent evidence based on 2003 Survey ofassessment practices in my own universityassessment practices in my own university

Shows the pattern changing, moving away fromShows the pattern changing, moving away fromtypical modeltypical model

More assessment done by coursework onlyMore assessment done by coursework only

@ Win Hornby 2004


10/39

RGU Survey of AssessmentRGU Survey of Assessment

Methods 2003Methods 2003Exam Only CW & Exam CW Only

WholeUniversity

3.1 43.6 49.7

Faculty ofDesign & Tech

0 50 50

Faculty of Health& Soc Care

6.6 29.5 55.7

Faculty of Mgt /

ABS

1.8 56.4 40


11/39

The Trade OffThe Trade Off

EffectivenessEffectiveness

Encouraging Staff to experiment with alternativeEncouraging Staff to experiment with alternative

assessment modesassessment modes EfficiencyEfficiency

Resource pressuresResource pressures

Coping with the Changing EnvironmentCoping with the Changing Environment

@ Win Hornby 2004


12/39

Effectiveness DefinedEffectiveness Defined

To what extent are the methods usedTo what extent are the methods used

educationally valid?educationally valid?

To what extent are the assessment methodsTo what extent are the assessment methods

used closely linked withused closely linked with desired skills anddesired skills andcompetencescompetences??

Are the assessment methodsAre the assessment methods constructivelyconstructively

alignedaligned to the stated outcomes to use Biggsto the stated outcomes to use Biggs(1997) phrase?(1997) phrase?

Does the assessment method match theDoes the assessment method match the tasktask

and outcomes?and outcomes?@ Win Hornby 2004


13/39

Effectiveness DefinedEffectiveness Defined

Is thereIs there overover--reliance on just one modereliance on just one mode ofof

assessment such as formal unseenassessment such as formal unseen

examinations?examinations?

Are studentsAre students overloadedoverloaded thus encouragingthus encouragingcoping strategies which lead to whatcoping strategies which lead to what

Entwistle has described asEntwistle has described as surface assurface as

opposed to deepopposed to deep learning? (Entwistle 1981) learning? (Entwistle 1981) Do the various stakeholdersDo the various stakeholders understand theunderstand the

criteriacriteria employed in the assessment methodemployed in the assessment method

and what they are designed to assess?and what they are designed to assess?@ Win Hornby 2004


14/39

Are our criteria clear?Are our criteria clear?

yy RGURGU AssessmentAssessment SurveySurvey ResultsResults

yy 6969%% ofof modulesmodules alwaysalways havehave criteriacriteria byby

whichwhich studentsstudents areare assessedassessedyy 2020%% ofof modulesmodules moremore oftenoften thanthan notnot havehave

criteriacriteria

yy 77%% occasionallyoccasionally useuse criteriacriteriayy 44%% nevernever havehave criteria!criteria!

@ Win Hornby 2004


15/39

Efficiency DefinedEfficiency Defined

What are theWhat are the educational and opportunityeducational and opportunity costscosts

of each method in terms of staff time, resourcesof each method in terms of staff time, resources

etc.?etc.?

What are the costs of systems to ensureWhat are the costs of systems to ensure fidelityfidelity ininthe assessment method used?the assessment method used?

What are theWhat are the administrativeadministrative costscosts of differentof different

methods?methods?

What are the costs to ensure assessment isWhat are the costs to ensure assessment isreliable and free from biasreliable and free from bias??

What are the costs of complying with variousWhat are the costs of complying with various

stakeholder demands on transparencystakeholder demands on transparency??@ Win Hornby 2004


16/39

Are we Effective and Efficient in ourAre we Effective and Efficient in our

Assessment?Assessment?

E AT ONA EFFE T E NE

E OURCEEFF CIENCY

i

ow

ow i

@ Win ornby 2004


17/39



E UCATIONA EFFECTIVENE

RE OURCEEFFICIENCY

i

ow

ow i

STAR

@ Win ornby 2004


18/39




RE OURCEEFFICIENCY

i

ow

ow i

STAR

DOGSOr

WOTS

@ Win ornby 2004


19/39




RE OURCEEFFICIENCY

i

ow

ow i

ROLLSROYCE

STAR

DOGSOr

WOTS

@ Win ornby 2004


20/39




RE OURCEEFFICIENCY

i

ow

ow i

ROLLSROYCE

STAR

DOGSOr

WOTS

?

OLDDOUBLEDECKERBUS?

@ Win ornby 2004


21/39


22/39

Consequences of overassessmentConsequences of overassessment

Poor FeedbackPoor Feedback

When it comes to giving feedback some staff stop at When it comes to giving feedback some staff stop atnothing!nothing!

Late feedbackLate feedback

Feedback is like fish it goes off after a week!Feedback is like fish it goes off after a week! Formative Assessment sacrificedFormative Assessment sacrificed

Students cut classes/tutorialsStudents cut classes/tutorials

Students work strategicallyStudents work strategically

Students look for short cuts including plagiarisingStudents look for short cuts including plagiarisingwork and personatingwork and personating

Little meaningful learning takes placeLittle meaningful learning takes place

@ Win Hornby 2004


23/39

Strategies for StreamliningStrategies for Streamlining

AssessmentAssessment1.1. Strategic Reduction of SummativeStrategic Reduction of Summative

assessmentassessment

2.2. FrontFront--end loadingend loading3.3. In Class assessmentIn Class assessment

4.4. Self and Peer assessmentSelf and Peer assessment

5.5. Group AssessmentGroup Assessment6.6. Automated Assessment and feedbackAutomated Assessment and feedback

@ Win Hornby 2004


24/39

Option 1. Strategic Reduction ofOption 1. Strategic Reduction of

Summative AssessmentSummative Assessment

Reduce the instances of assessment?Reduce the instances of assessment?

Exemptions from exams on basis of courseworkExemptions from exams on basis of courseworkperformance?performance?

Assess learning outcomes once only?Assess learning outcomes once only? Combine assessments across modules?Combine assessments across modules?

Abolish resit examinations and reassessAbolish resit examinations and reassessdifferently?differently?

Mechanisms for balancing types of assessmentMechanisms for balancing types of assessmentbetween modules?between modules?

Timetabling assessments?Timetabling assessments?

@ Win Hornby 2004


25/39

Option 2. FrontOption 2. Front--end loadingend loading

Coursework briefing sessions?Coursework briefing sessions?

Discuss, explain and unpack criteria?Discuss, explain and unpack criteria?

Get students to engage with the criteria?Get students to engage with the criteria?

Get students to assess previous cohortsGet students to assess previous cohorts

work?work?

Allow students input into deciding theAllow students input into deciding the

criteria?criteria?

Rust, Price and ODonovan Study (2003)Rust, Price and ODonovan Study (2003)

@ Win Hornby 2004


26/39

OptionOption 3. In Class assessment3. In Class assessment

Marked/graded in class by students?Marked/graded in class by students?

Build in several of these for coursework?Build in several of these for coursework?

Gow (quoted in Hornby (2003) First yearGow (quoted in Hornby (2003) First yearEngineering students @RGUEngineering students @RGU

Reported:Reported:

Improved attendance ratesImproved attendance rates

Improved motivation to learnImproved motivation to learn

Higher retention ratesHigher retention rates

Higher pass rates in the final examination,Higher pass rates in the final examination,hence fewer resits to markhence fewer resits to mark

Improved second year performance becauseImproved second year performance becauseunderpinning first year knowledge betterunderpinning first year knowledge better

@ Win Hornby 2004


27/39

Option 4. Self and Peer assessmentOption 4. Self and Peer assessment

Some scepticism that students can beSome scepticism that students can betrusted to do this for themselvestrusted to do this for themselves

Reliability of Assessment?Reliability of Assessment?

Survey of Research Evidence?Survey of Research Evidence?

Tutor/Student Correlations Low (r=0.21)Tutor/Student Correlations Low (r=0.21)

High Variability between tutor ratings andHigh Variability between tutor ratings andstudent ratingsstudent ratings

Students overStudents over--rate themselves?rate themselves?

@ Win Hornby 2004


28/39

Self and Peer assessmentSelf and Peer assessment

Self Assessment of 48 honours year Accounting StudentsSelf Assessment of 48 honours year Accounting Students Using Grade Related CriteriaUsing Grade Related Criteria

DimensionsDimensions

PresentationPresentation

ResearchResearch Knowledge and UnderstandingKnowledge and Understanding

AnalysisAnalysis

EvaluationEvaluation

Unpacked these with briefing and online discussion forumUnpacked these with briefing and online discussion forum Results?Results?

Tutors, Who Needs them?Tutors, Who Needs them?

@ Win Hornby 2004


29/39


Using GRC and some front end loading weUsing GRC and some front end loading we

found:found:

High Correlation Tutor Students RatingsHigh Correlation Tutor Students Ratings

High Agreement between students ratingsHigh Agreement between students ratings

and tutor ratingsand tutor ratings

No evidence of overNo evidence of over--rating by studentsrating by students Evidence that students found the experienceEvidence that students found the experience

educationally valuableeducationally valuable

@ Win Hornby 2004


30/39

@ Win Hornby 2004


31/39

@ Win Hornby 2004


32/39


The Educational Value of Self AssessmentThe Educational Value of Self Assessment

Students feedback indicated benefits fromStudents feedback indicated benefits fromthe self assessment process but tothe self assessment process but toparaphrase:paraphrase:

Why wasthisnotdoneatanearlierWhy wasthisnotdoneatanearlierstageinthe degree programme!Whystageinthe degree programme!Why

waituntilmyfinalyearto dothis?Wewaituntilmyfinalyearto dothis?Weshould be doingthis kindofthingfromshould be doingthis kindofthingfromDay1Day1

@ Win Hornby 2004


33/39

Option 5. Group AssessmentOption 5. Group Assessment

Potential to be very effective and efficientPotential to be very effective and efficient

But is it reliable?But is it reliable?

What about the hassle factor?What about the hassle factor? The free rider problem?The free rider problem?

@ Win Hornby 2004


34/39

Group AssessmentGroup Assessment

Barnes (quoted in Hornby (2003))Barnes (quoted in Hornby (2003))

Third Year Hospitality Management StudentsThird Year Hospitality Management Students

Reports on using an evaluation instrument for both selfReports on using an evaluation instrument for both selfand peer assessmentand peer assessment

Administered electronically via Question Mark PerceptionAdministered electronically via Question Mark Perception Assesses a number of different dimensions of individualAssesses a number of different dimensions of individual

and group work such asand group work such as

Attendance, ideas generation, contribution to groupAttendance, ideas generation, contribution to groupreport, knowledge and skill acquisition, effortreport, knowledge and skill acquisition, effort

Also an anonymous assessment of the work of othersAlso an anonymous assessment of the work of otherswhich was fed back to each studentwhich was fed back to each student

Not only did student get a grade but also got a score onNot only did student get a grade but also got a score oneach of the identified dimensions and a reporteach of the identified dimensions and a report

@ Win Hornby 2004


35/39

Option 6. Automated AssessmentOption 6. Automated Assessment

and Feedbackand Feedback Automated FeedbackAutomated Feedback

Automated AssessmentAutomated Assessment

@ Win Hornby 2004


36/39

Automated FeedbackAutomated Feedback

Checklists and Statement BanksChecklists and Statement Banks

RGU Case Studies in Streamlining AssessmentRGU Case Studies in Streamlining Assessment

Checklists in Applied SciencesChecklists in Applied Sciences

Statement Banks in the Economics of Tax ModuleStatement Banks in the Economics of Tax Module

Grade related statements on each of the dimensionsGrade related statements on each of the dimensions

Cut and Paste + a bit of personalisationCut and Paste + a bit of personalisation

EE--mail feedback in realmail feedback in real--timetime Quick and dirtyQuick and dirtyvv Clean and slowClean and slow??

Whole Class feedback online via iNETWhole Class feedback online via iNET

@ Win Hornby 2004


37/39

Automated AssessmentAutomated Assessment

Cooper (quoted in Hornby (2003))Cooper (quoted in Hornby (2003))

Bsc (Hons) Sports Science students in the School ofBsc (Hons) Sports Science students in the School ofHealth Sciences.Health Sciences.

Self Assessment linked to video clips of particularSelf Assessment linked to video clips of particular

movements of the body to identify key muscles and jointsmovements of the body to identify key muscles and joints Required to identify muscles producing the movementsRequired to identify muscles producing the movements

and the type of muscle work involved.and the type of muscle work involved.

Stop/ Start/Replay facilityStop/ Start/Replay facility

Multiple choice questions with feedback on each responseMultiple choice questions with feedback on each response

iNET discussion forum to ask questions about the variousiNET discussion forum to ask questions about the variousparts of the syllabus and self assessment assignmentsparts of the syllabus and self assessment assignments

Transferability of the idea to Science and engineeringTransferability of the idea to Science and engineeringareasareas

@ Win Hornby 2004


38/39

How will all this save me time?How will all this save me time?

Some strategies do not involve extra costsSome strategies do not involve extra costs

(e.g. strategic reduction)(e.g. strategic reduction)

Some strategies do involve some initial setSome strategies do involve some initial setup costsup costs

View these like an investment appraisalView these like an investment appraisal

projectproject

Initial capital outlayInitial capital outlay

Reap benefits over timeReap benefits over time

@ Win Hornby 2004


39/39

ReferencesReferences

Quality Enhancement in Teaching, LearningQuality Enhancement in Teaching, Learning

and Assessmentand Assessment

Assessment @RGUAssessment @RGU

http://www.rgu.ac.uk/celt/quality/page.cfm?pge=6201http://www.rgu.ac.uk/celt/quality/page.cfm?pge=6201

@ Win Hornby 2004

Win Horn by Presentation 130104

Documents

Transcript of Win Horn by Presentation 130104