Test development cycle Evaluation, item analysis and feedback KAYSERI, 29-31 JANUARY 2014
Aims
! Test development cycle ! Test evaluation ! Constructive feedback ! Item banking
Test development cycle
Plan
Write
Use Evaluate
Revise
Test evaluation
Test evaluation
! How we view statistical data
! Other, less quantifiable factors?? ! commitment, aptitude, attitude to learning,
rapport with teacher and other learners, creativity, intelligence, etc.
objective numerical fact
Assessing a test
Statistical evaluation ! objective results
! based on calculations
! expressed in numerical data ! fact?
! doesn’t recognise external factors
! can only be black-white – or other factors need to be calculated
Holistic evaluation ! subjective impressions
! based on own experience
! expressed as opinion
! takes into account unmeasurable factors
! can be shaded – depending on circumstances
! finding out how difficult our test is ! Facility Value
! finding out how our students performed in comparison to one another (a ‘norm’) ! Discrimination Index
Statistical evaluation: item analysis
see handout
Constructive feedback
Constructive feedback
! clear and direct
! improvements that can be made ! courses of action
! ‘sandwich’ your feedback ! something positive about the subject: „Your test is really comprehensive.” ! your main general criticism: „However, I think it’s too long – maybe you’d like
to cut a few items?” ! specific critical points: „In Task 6, I don’t think you’ll need twelve items – six
would do just as well. Some of them are quite problematic anyway, as there are several possible answers. Maybe you could just keep the unambiguous ones?”
! conclude with something constructive: „I think with the revisions we talked about you’ll cover everything that needs testing at this point.”
Test checklist (adapted from Alderson et al., 1995) 1 Don’t just look! Approach each item as a test taker.
2 To do your test tasks, get someone who doesn’t know them.
3 Nobody writes good tests alone. Ask for others’ feedback.
4 Be prepared to change things. We all write bad items.
5 Revise any item where you get an unexpected response.
6 Check your tests against the syllabus. Anything missing?
7 Assess the test method – is it the best way to get results?
8 Is there a context? Does it feel real?
9 What are your marking criteria? Do students know them?
10 What if your own judgement is different from the test score?
Item banking
Item banking
What? ! test items ! identifying data
! content
! level
! course
! format
! key and scoring ! item analysis: FV and DI ! notes of previous use
Why? ! workload saving
! pre-trialled items
! informed selection of items for a particular use
! more balanced tests
! consistent standards
Top Related