Analysis of Test formats

Authors:Hector Rojas

Daniel BlancoYuly Caraballo

Matching

STRENGTHS

Objective Easy to design Easy to include a

representative example

WEAKNESSES

Easy to guess Only for grammar,

vocabulary, pronunciation

Not good for testing the skills.

Students need to decide on what answers accepted as correct.

Gap-filling

STRENGTHS Objective Easy to correct Provides clear contexts It can be used to

correct different strategies.

WEAKNESSES Extremely difficult to

design an item with only one possible answer (generally solved by providing a list of terms)

Need to think about possible answers (if a list is not provided)

Cloze

STRENGTHS

Easy for the teacher to create

Easy for the students to understand

Easy to correct and design .

WEAKNESSES

Unexpected correct answers are possible

Mainly for vocabulary It can be irritating for students

Completion

STRENGTHS It is easy to construct Requires simple

recognition of information (eliminatting the guessing factor)

It is good to evaluate listening for specific information

WEAKNESSES Depend on knowledge

of language Difficult to grade

depending on evaluation criteria

It can be irritating for students

Multiple Choice

STRENGTHS Easy to correct and grade Scores are less affected

by guessing Good for checking

understanding Usually it requires less

time for test takers to answer

WEAKNESSES Difficult to construct It does not allowed test

takers to demonstrate knowledge beyond the options provided

Emphasize in recognition and guessing rather than in production

True-False

STRENGTHS Easy to write Can cover a wide

range of difficulty levels

Usually it requires less time for test takers to answer

Usually it is easy to grade and score

WEAKNESSES Guessing element The score can not be

reliable It does not allow test

takers to demonstrate broad range of knowledge

Transformations

STRENGTHS

Good for testing some structures

Easy to write and administer

Specifically tackle one grammatical objective

Allow recognition of a connection between “grammar” and “meaning”

WEAKNESSES

Very artificial It can be more than

one possible transformation in some cases

Questions

STRENGTHS

It is good for checking if students comprehend  texts.

It is useful to test the student's ability to analyze a concept.

WEAKNESSES

It is not easy to score It takes more time to

correct and answer.

Recommendations for item design

FORMAT RECOMMENDATIONS

Matching • Give more options than questions.• Each answer should have distracters.• Try meaningful items.• Make sure there’s only one possible selection.• Do not ask to draw lines to match.

Gap filling • No blank at the beginning.• Give enough context.• Avoid clues.• Only one answer • Give line with the same length

Cloze Don’t use nth word, for example : 5th , 3rd

Completion • Avoid the statement becomes ambiguous.• Use blanks of the same length throughout the test so that

the length is not a clue.• If possible, put the blank at the end of a statement rather

than at the beginning. Asking for a response before the student understands the intent of the statement can be confusing and may require more reading time.

Multiple choice

• Be clear and concise in your word and phrase choices.• Make sure that there is only one clearly correct answer

from the options given to the student.• Randomly distribute the correct answer options i.e. A, B, C,

D etc so that there is not a clear pattern that becomes obvious to the student.

True-False • Make sure that the answer is clear and that it could not be either or.• Try not to use negative questions such as “this novel was

not written by....”• Use a random order of true and false responses with your

test questions to avoid creating a pattern.• Use more false questions than true questions as they have

been proven to cater towards higher cognitive level students.

Questions • It has to be easy to understand.• Check how you will judge answers.

Poor & Good items

Multiple choice items

All words should belong to the same grammatical category.

She had to help the ______ old man up the stairs.

A. weak B. slowly C. try D. wisdom

A. weak B. strong C. energetic D. athletic

Adverb Verb Noun

Poor & Good items ii

Grammatical clues make the answer obvius.

She needs to get up earlier so she’s buying an ______ clock.

A. time B. alarm C. watch D. bell

She needs to get up earlier so she’s buying ______ clock.

A. a time B. an alarm C. a watch D. a bell

Deleting the indefinite article would make this item better in terms of objectivity.

Poor & Good items iii

The level of difficulty of the distractors highlights the correct alternative.

They needed lots of training to operate such ______ equipment.

A. easy B. sophisticated C. blue D. wise

“Sophisticated” clearly contrasts with the rest of the options, so students might select it right away.

Poor & Good items iv

Options that collocate with the verb in the sentence. (Differentiators)

She sent the _______ yesterday.

A. letter B. gift C. food D. books

She wrote a _______ yesterday.

A. letter B. gift C. friend D. books

“Post”, “friend” and “courage” are good replacements for “gift”, “food” and “books” that are items that can also be sent or mailed.

Poor & Good items v

Completion

The absence of context will lead to the acceptance of unexpected correct answers.An advanced learner could get to the conclusion of

completing with the letter “y” to form the adjective “nervy”, which means “bold” or “offensive”.

He was a very nerv_____ person.

He was always alert. He was a ________ person.

In this case, given the context, the correct option is “nervous”.

Poor & Good items vi

Words that may take two opposite suffixes lead to ambiguity.

That was a care______ answer.

Enough context will permit the student know which alternative is the most appropiate one.

Yesterday he got on the wrong bus. So today, he was care_______ to find the right one.

Poor & Good items vii

True-FalseColumbus discovered America.

Columbus sailed to the new world in 1942.

The use of trivial information must be avoided . Given the fact that they are more likely to be true than ambiguous information.

If used, they must be presented in a more objective way.

Poor & Good items viii

Since penguins can’t fly, they cannot swim very well either.

Since penguins are unable to fly, they swim badly.

Avoiding the use of negative statements will prevent confusion and contradiction.

Poor & Good items ix

Rattlesnakes live in North America, have fangs, feed on small rodents and crush their prey.

The first three clauses are true, however the last one is not.

The entire sentence must be either true or false, otherwise it will create confusion.

Rattlesnakes crush their preys.

Poor & Good items x

Matching

___1. Sit-and-reach___2. 50-yard dash___3. Pull-up___4. Shuttle run___5. Balke treadmill___6. Dyer backboard volley___7. Disch putting___8. Biopsy

c. Muscle fibersh. Golff.Tennisa.Vo₂maxe.Agilityg.Arm strengthd.Speedb.Flexibility

Items are too heterogeneus, making the answers too obvious.

Contain the same number of items, so the last item could be answered by elimination.

Match column 1 and comlumn 2:

The instructions do not clarify how many times items in column 2 may be used or a basis for matching.

Poor & Good items xi

TransformationChange the sentences to the future

simple:

Doug studied English yesterday.Doug went to the movies with his family.He eats a hamburger.They enjoy the day in the mall.

The activity lacks context

The sentences should be provided in the same tense, to prevent confusion.This exercise requieres more contextualized

instructions, such as:

This is what Doug usually does on the weekend, write about what he is going to do next the weekened.

Item and test analysis.

Item analysis: measure the performance of each item in a test and help us to revise and improve both, items and test as a whole.

Use: -To build reliablity and validity into a test.- More diagnosis information on students.

Features of item analysis

Item difficulty: deals with the number of students who answer correctly. The best items are that are neither too difficult or too hard.

IF = # of students answering the item correctly

Total of students responding that item.

Features of item analysis

Item discrimination : differenciates those test takers with high - abilities from those with low - abilities.

Features of item analysis

Distractor efficiency: deals with the way a distractor lures test takers, especially those with low abilities.

References

Brown, Douglas (2004) Language assessment: Principles and Classroom prractices. Longman, London.

Cunningham, George (1998) Assessment in the Classroom: Constructing and interpreting Tests. Routledge, London.

Morrow, James and others (2010)Measurement and Evaluation in Human Performance. Human kinetics, Illinois.

Madsen, Harold (1983) Techniques in testing. Oxford University Press, New York.