Cambridge Exam Research Malta, 2009 © Pollitt & Ahmed, 2009 The Importance of being Valid 1 The...

Cambridge Exam Research Malta, 2009

© Pollitt & Ahmed, 2009 The Importance of being Valid 1

The Importance of being Valid

Alastair Pollitt & Ayesha Ahmed

Cambridge Exam [email protected]

[email protected]



Defining ‘Validity’

There is no true meaning of’ Validity’

Definitions of ‘Validity’ are opinions about what the word should mean

How do we decide what ‘Validity’ should mean?

We need an inclusive concept of ‘Validity’ that is useful to everyone involved in the process of assessment, and not too far from what they normally mean by ‘Valid’.

A useful definition of ‘Validity’



The real world meanings of “valid”

1 offer 1022 ticket 323 licence 254 certificate 255 passport 196 law 157 voucher 14

. . . . . . . . . . . . . . .

18 reason 6

Relative Rank Concordance frequency

Source: Collins Cobuild Corpora (British + American)



What we mean by Validity

We have worked for many years with professionals who:

• Write questions; Review questions; Write mark schemes; Mark tests; Design tests; etc

The key issue for them, and us, is what the students’ minds are/were doing during the test –

in other words,

the Evidence that gives us about achievement.



What we mean by Validity

This is the essence of Validity – the interaction between a test question and the student’s mind.

If this goes wrong then Validity is impossible.

But how do we get it right?

By considering further the concept of Evidence.



Evidence

It is not enough to be Logically Valid.

A “widely used definition of assessment”:

The process of seeking and interpreting evidence for use by …

(Assessment Reform Group, 2002:

quoted by Stobart, 2009)



Importance Statement

A negotiated consensual statement of what is really important in the study of that subject during that phase of education.

Available to teachers, students, parents, examiners, employers, selectors – to everyone with an interest in the subject.



The Importance of Design & Technology

Design and technology prepares pupils to participate in tomorrow's rapidly changing technologies. They learn to think and intervene creatively to improve quality of life. The subject calls for pupils to become autonomous and creative problem solvers, as individuals and members of a team. They must look for needs, wants and opportunities and respond to them by developing a range of ideas and making products and systems.

They combine practical skills with an understanding of aesthetics, social and environmental issues, function and industrial practices. As they do so, they reflect on and evaluate present and past design and technology, its uses and effects. Through design and technology, all pupils can become discriminating and informed users of products, and become innovators.



The Importance of the Importance Statement

1 To ensure that teachers emphasise what is important

2 To ensure that test designers, question and mark scheme writers test what is important

3 To ensure that markers give credit for what is important

4 To ensure that test results are interpreted in terms of what is important

5 The source of Validity



A model of Validity

Validity is about measuring and interpreting what is important in education.

It is not merely about measuring and interpreting what is measured.

Our model describes where Validity comes from, and how it is maximised.



The Bucket Brigade model

Suppose: Y is a house on fire.

Suppose: X is a well full of water.

How do you put out the fire?



A typical assessment professional



The Entropy law of validity

Version 1

During the assessment process, validity always decreases

Version 2

Validity can only be lost, it can never be recovered.



How to maintain validity - 1

Writing the Importance Statement




Specifying the test




Writing the questions and mark schemes




The students




Marking




Grading, etc




Interpreting results



How to maintain validity

1 Writing the Importance Statement

2 Specifying the test

3 Writing the questions and mark schemes

4 Communicating with students

5 Marking

6 Grading

7 Interpreting



Summary

The meaning of test results depends crucially on a proper development of the test, so that it elicits evidence of what is important.

Without meaning, Validity is an empty concept.

The source of Validity must be a clear and shared concept of Importance.



8The importance of being Valid

E

Q

i

M

Idea of a Question

Evaluation

Elicitation

IS

A

Importance Statement

TeachingT

Assessment

Evidence

Cambridge Exam Research Malta, 2009 © Pollitt & Ahmed, 2009 The Importance of being Valid 1 The...

Documents

Transcript of Cambridge Exam Research Malta, 2009 © Pollitt & Ahmed, 2009 The Importance of being Valid 1 The...