6 July 2011Assessment in Higher Education Conference1 The assessment, marking and feedback inventory...

6 July 2011Assessment in Higher Education

Conference 1

The assessment, marking and feedback inventory (AMFI): a tool forexploring the practitioner perspective

Lin Norton, Bill Norton, Kamel Mansi


Conference 2

Acknowledgments: our thanks to all the lecturers at Liverpool Hope who helped with this research


Conference 3

Research Background

This earlier research only tells part of the story in two respects: 1) it concentrated on ‘new’ lecturers 2) it only looked at their views on assessment design

The current study was designed to look more widely at the whole issue of assessment, marking and feedback with lecturers with a range of experience all working at Liverpool Hope University

Previous research carried out by the Write Now CETL research team focused on newly qualified lecturers’ views about assessment design. We found that certain ‘constraints’ could hinder sound assessment design being put into practice (Norton, Norton & Shannon, in press).


Conference 4

Research questions

1. What do lecturers think about assessment methods, marking and feedback practices?

2. Are there any differences between the subject disciplines ?

3. Can a questionnaire on assessment, marking and feedback be developed that would be useful for both research and practitioner purposes?


Conference 5

Theoretical background: the effects of the discipline

Neumann Parry & Becher (2002) used Biglan’s (1973) original classification of hard- soft and pure –applied and found differences in teaching and research:

hard (pure and applied) - strongly committed to research and less committed to teaching, (generally seen as relatively straightforward and unproblematic), collaborative research and teaching

soft (pure and applied) - greater emphasis on scholarly knowledge that translates readily into teaching, more emphasis on individualistic enquiry and not so much acceptance of joint teaching


Conference 6

But what about assessment…?? Since assessment is so

closely aligned to learning and teaching, might there be similar differences in lecturers’ assessment, marking and feedback practices and attitudes, related to their subject disciplines ??

White & Liccardi (2006) drew some distinctions between the disciplines and assessment methods favoured http://eprints.ecs.soton.ac.uk/12576/2/Harnessing_Insight_into_Disciplinary_Differences_to.ppt

Discipline Favoured assessments

Hard

pure(Natural

Sciences)

Specific & focused exam questions

Objective tests (quantitative nature of knowledge)

Hard applied

(Engineering)

Preference for exam questions, specifically problem-solving

Soft

Pure(Social Sciences & Humanities)

Essay questions, short answer questions, oral presentations, continuous assessment

Soft

Applied(Nursing, Education)

Essays, project-based assignments, peer & self assessments


Conference 7

The concept of signature pedagogies

Lee Shulman, former president of the Carnegie Foundation for the Advancement of Teaching

10 year study to understand how people are prepared for practice in law, engineering, the clergy, teaching, nursing and medicine.

Argued that education for professionals has to include more than knowing, and that understanding the discipline is only part of what we should be teaching

Signature pedagogies are not stable and unchanging and are as likely to change as professional practices themselves


Conference 8

Characteristics of the signature pedagogies Shulman described a mode of teaching that has become

inextricably identified with preparing people for a particular profession, he says there are three defining characteristics:

1. “… it’s distinctive in that profession… 2. … it is pervasive within the curriculum. So that students

learn that as they go from course to course, there are certain continuities that thread through the program that are part of what it means to learn to “think like a lawyer,” or “think like a physician,” or “think like a priest.””

3. (it) … cuts across institutions and not only courses. Signature pedagogies have become essential to general pedagogy of an entire profession, as elements of instruction and of socialization.


Conference 9

So where is assessment in signature pedagogies? Shulman writes about teaching processes but appears silent

on the subject of assessment processes Taking the very broad concept of signature pedagogies as a

proxy for teaching process differences, then there presumably must be assessment process differences…

In our work on assessment design (Norton et al, in press) we found some discipline differences on: ‘desirable assessment practice’ factor

lecturers from soft applied disciplines scored higher on the overall total than lecturers from soft pure disciplines, hard pure disciplines and hard applied disciplines

‘constraints to desirable practice ’ factor lecturers from hard pure and hard applied disciplines scored higher on

the overall total than lecturers from soft pure and soft applied


Conference 10

Development of the Assessment, Marking and Feedback Inventory (AMFI)

A pilot version of the AMFI was developed by Dr Kamel Mansi (former Write Now CETL postdoctoral researcher)

This consisted of four main sections (The items were derived from over 80 interviews with lecturers from 5 UK institutions, as well as from the literature) :

1. methods of assessment (52 item checklist from Bloxham & Boyd (2007))

2. rationale for choice of assessment method (13 statements)

3. marking attitudes and practices (22 statements)

4. feedback attitudes and practices (31 statements) The AMFI was made available online to Hope lecturers. 45 completed inventories from 17 disciplines.


Conference 11

Can signature pedagogies be applied to subjects studied at Hope? We thought it could, given the strong government

agenda that university education should be linked to employability skills BUT Hope is mainly liberal arts; nevertheless we thought subjects that could be counted as professionally-orientated might include Education Music, Drama, Fine Arts Business studies, Marketing Psychology, Sociology, Health

So we were keen to analyse our AMFI results to see if there were any subject differences, but…


Conference 12

…not enough participants Out of the completed inventories, there were not sufficient

numbers to make any meaningful comparisons throughout, as many of the subjects were represented by one or two lecturers.

In subjects where there were more, such as Business (N=5), not all of them completed all the questions.

This left us with the option of assigning the subjects to Biglan’s (1973) broad categories of hard applied, hard pure, soft applied, and soft pure.


Conference 13

Which subject disciplines did the participating lecturers represent?

HARD APPLIED (5)

Computing

Health

SOFT PURE (12)

English

Geography

Music

Psychology

Sociology

Theology & Religious Studies

SOFT APPLIED (24)

Arts

Business Studies

Childhood & Youth Studies

Disability Studies

Drama & Theatre Studies

Education

Marketing

Media

Sport

HARD PURE

No lecturers


Conference 14

Section 1. Methods of assessment 52 methods presented and participants were asked to look at

each one and respond: Have used Would like to use Not Familiar

See handout


Conference 15

Assessment methods continued

Of Bloxham & Boyd’s 52 methods of assessment presented, only one (patchwork texts) had not been used by any of the respondents, although 3 lecturers said they would like to use this method

19 of the 52 methods were unfamiliar to over a third of our lecturers.

Many respondents expressed interest in using methods of assessment they did not currently use..


Conference 16

Most popular assessment methods respondents would like to use

Method N=45

Electronic presentation (web pages) 38%

Writing abstracts 31%

Debate speech 29%

Film or radio programmes 29%

Simulation exercises 24%

Computer based assignment 22%

Annotated bibliography 22%

Fieldwork reports 20%

Review of book, article, website, etc. 20%

Placement reports 20%


Conference 17

Most popular assessment methods usedMethod N=45

Essay 98%

Dissertation 93%

Research project 93%

Presentation 91%

Reflective learning assignments 80%

In-class tests 76%

Examinations (unseen, seen, open-book, case study, take-away) 75%

Portfolio (written) 71%


Conference 18

Differences between the disciplines No real differences in the most commonly used assessment

methods but there was a small difference in the range of methods.

Hard applied lecturers had used 41 methods

Soft pure lecturers had used 47 methods

Soft applied lecturers had used 46 methods


Conference 19

Section 2: Rationale for choice of assessment method 13 items were presented, e.g.

Takes into account students’ preferences Prepares students for employment

Participants were asked to rate how important they felt each item was: Very important Important Not important Completely unimportant


Conference 20

Most important rationales for choice of assessment

Very important/important Overall(N=45)

Provides a sample of what students are able to do with the knowledge they have acquired 98%

Develops students’ writing skills 97%

Prepares students for employment 90%

*Prevents students from plagiarising 89%

Allows students to develop oral as well as written skills 86%

* Preventing students from plagiarising is not a pedagogical rationale, but is commonly cited for using specific methods of assessment such as portfolios (Irons, 2004) and exams (Norton et al, 2006)


Conference 21

Section 3: Marking attitudes and practices This section contained 22 items e.g.

The external examiner system does not ensure grading fairness (attitude)

I have learned to mark myself without any professional training (practice)

Participants were asked to respond to each one by choosing one of the following responses: Strongly agree Agree Disagree Strongly disagree


Conference 22

Section 3. Are there any differences between the subject disciplines in what lecturers think about marking? Hard applied lecturers believed more strongly than soft pure

lecturers that professional training in marking was not needed

Hard applied lecturers agreed more strongly than both soft pure and soft applied lecturers that giving students highly detailed guidelines increased the likelihood of students challenging their grades


Conference 23

Section 4: Feedback attitudes and practices This section of the AMFI contained 31 statements e.g.

Face to face communication with students is the best way to give feedback (attitude)

I seek advice from colleagues for providing clear information to students (practice)

Participants were asked to respond to each one by choosing one of the following responses: Strongly agree Agree Disagree Strongly disagree


Conference 24

Section 4. Are there any differences between the subject disciplines in what lecturers think about feedback?

Hard applied and soft applied lecturers agree more strongly than soft pure lecturers that giving feedback puts an unfair workload on lecturers

Hard applied and soft applied lecturers agree more strongly than soft pure lecturers that feedback sessions in groups were more effective than individual feedback

Hard applied lecturers were more likely than soft applied and soft pure lecturers to agree that lack of time prevented them from giving good feedback


Conference 25

Is there a pattern?

Too small a data set to answer with any certainty, and out of all the items, we only found 5 that were statistically different.

Some indications that hard applied lecturers were ‘more traditional’ in their approach to marking and feedback, they were less keen to spend time on it and used fewer assessment methods than lecturers from the other two disciplines

This lends some support to our earlier finding which showed hard applied lecturers were more likely to be constrained in their assessment design.


Conference 26

Developing the AMFI as a research tool and as a practitioner tool1. Research: Currently refining the items (correlational and

reliability analyses). Would like to try it out on a much larger sample to see if signature pedagogies exist in assessment, and if so what are the implications for engendering change?

2. Practice: Several participants commented that the exercise of completing the AMFI had helped them think more closely about their assessment, marking and feedback practices and attitudes, so it could be used in a similar way to the Assessment Design Inventory as a trigger for debate and discussion ( within departments and/or on university teaching programmes)


Conference 27

A final question…

How easy is it to change assessment in Higher Education? (…if it needs changing) in a sector where generic assessment pedagogy can sometimes be seen as irrelevant to:

Departments (Becher & Trowler, 2001; Knight & Trowler,2000; Fanghanel, 2007)

Disciplines (Biglan,1973; Neumann et al, 2002)

Professions (Shulman, 2005)

6 July 2011Assessment in Higher Education Conference1 The assessment, marking and feedback inventory...

Documents

Transcript of 6 July 2011Assessment in Higher Education Conference1 The assessment, marking and feedback inventory...