Insights and development of tests

Insights and Insights and Development of Development of

TestsTests

Carlo Magno, PhDCarlo Magno, PhD

De La Salle University, ManilaDe La Salle University, Manila

Activity 1: Assessment Schema Activity 1: Assessment Schema Check-upCheck-up

Answer the following questions as a group. Your answers Answer the following questions as a group. Your answers should reflect your current practices in assessing your should reflect your current practices in assessing your students. Write your answers in a manila paper and post it students. Write your answers in a manila paper and post it after.after.

1. List down the things that you do when preparing to write 1. List down the things that you do when preparing to write your test items? your test items? (procedure)(procedure)2. What are the things that you consider when writing your 2. What are the things that you consider when writing your test items? test items? (concepts)(concepts)3. What further steps do you do after you have scored 3. What further steps do you do after you have scored and recorded the test papers? and recorded the test papers? (procedure)(procedure)4. What other forms of assessment do you conduct aside 4. What other forms of assessment do you conduct aside from paper and pencil tests?from paper and pencil tests?

1. List down the things that you do when 1. List down the things that you do when preparing to write your test items? preparing to write your test items?

Prepare Table of Specifications (TOS)Prepare Table of Specifications (TOS)

Use the Taxonomy of Cognitive skills Use the Taxonomy of Cognitive skills (Bloom’s taxonomy)(Bloom’s taxonomy)

Conduct Item reviewConduct Item review

2. What are the things that you consider 2. What are the things that you consider when writing your test items?when writing your test items?

Learning objectivesLearning objectivesCurriculum/national standardsCurriculum/national standardsNeeds of studentsNeeds of studentsHigher order thinking skillsHigher order thinking skillsTest lengthTest lengthTest instructionTest instructionTest layoutTest layoutScoringScoring

3. What further steps do you do after you 3. What further steps do you do after you have scored and recorded the test papers?have scored and recorded the test papers?

Item analysisItem analysis– Item difficultyItem difficulty– Item discriminationItem discrimination

Distracter analysisDistracter analysis

Reliability analysisReliability analysis

Validity analysisValidity analysis

4. What other forms of assessment do you 4. What other forms of assessment do you conduct aside from paper and pencil tests?conduct aside from paper and pencil tests?

Alternative forms of assessmentAlternative forms of assessment– Performance-based AssessmentPerformance-based Assessment– Authentic assessmentAuthentic assessment– Portfolio assessmentPortfolio assessment

Types of Response FormatTypes of Response Format

Selected-response itemsSelected-response items– Binary-choice itemsBinary-choice items– Matching typeMatching type– Multiple choice typeMultiple choice type

Constructed-responseConstructed-response– Short-answer itemsShort-answer items– Cloze testCloze test– Essay itemsEssay items

Binary TypeBinary Type

1. Avoid the use of “specific determiners” 1. Avoid the use of “specific determiners”

FAULTY: No picture-no sound in a FAULTY: No picture-no sound in a television set may indicate a bad 5U4G.television set may indicate a bad 5U4G.

IMPROVED: A bad 5U4G tube in a IMPROVED: A bad 5U4G tube in a television set will result in no picture television set will result in no picture sound.sound.

Why is the item faulty?


2. Base true-false items upon statements 2. Base true-false items upon statements that are absolutely true or false, without that are absolutely true or false, without qualifications or exceptions.qualifications or exceptions.FAULTY: World War II was fought in FAULTY: World War II was fought in Europe and the Far East.Europe and the Far East.IMPROVED: The primary combat IMPROVED: The primary combat locations in terms of military personnel locations in terms of military personnel during World War II were Europe and the during World War II were Europe and the Far East.Far East.



3. Avoid negative stated items when 3. Avoid negative stated items when possible and eliminate all double possible and eliminate all double negatives.negatives.

FAULTY: It is not frequently observed that FAULTY: It is not frequently observed that copper turns green as a result of oxidation.copper turns green as a result of oxidation.

IMPROVED: Copper will turn green upon IMPROVED: Copper will turn green upon oxidizing.oxidizing.



4. Use quantitative and precise rather 4. Use quantitative and precise rather than qualitative language where possible.than qualitative language where possible.

FAULTY: Many people voted for Gloria FAULTY: Many people voted for Gloria Arroyo in the 2003 Presidential election.Arroyo in the 2003 Presidential election.

IMPROVED: Gloria Arroyo received more IMPROVED: Gloria Arroyo received more than 60 percent of the popular votes cast than 60 percent of the popular votes cast in the Presidential election of 2003.in the Presidential election of 2003.



5. Avoid stereotypic and textbook statements.5. Avoid stereotypic and textbook statements.

FAULTY: From time to time efforts have been FAULTY: From time to time efforts have been made to explode the notion that there may be a made to explode the notion that there may be a cause-and-effect relationship between arboreal cause-and-effect relationship between arboreal life and primate anatomy.life and primate anatomy.

IMPROVED: There is a known relationship IMPROVED: There is a known relationship between primate anatomy and arboreal life.between primate anatomy and arboreal life.



6. Avoid making the true items consistently 6. Avoid making the true items consistently longer than the false items.longer than the false items.7. Avoid the use of unfamiliar or esoteric 7. Avoid the use of unfamiliar or esoteric language.language.FAULTY: According to some peripatetic FAULTY: According to some peripatetic politicos, the raison d’etre for capital punishment politicos, the raison d’etre for capital punishment is retribution.is retribution.IMPROVED: According to some politicians, IMPROVED: According to some politicians, justification for the existence of capital justification for the existence of capital punishment can be traced to the Biblical punishment can be traced to the Biblical statement, “An eye for an eye.”statement, “An eye for an eye.”



8. Avoid complex sentences with many 8. Avoid complex sentences with many dependent clauses.dependent clauses.

FAULTY: Jane Austen, an American FAULTY: Jane Austen, an American novelist born in 1790, was a prolific writer novelist born in 1790, was a prolific writer and is best known for her novel Pride and and is best known for her novel Pride and Prejudice, which was published in 1820.Prejudice, which was published in 1820.

IMPROVED: Jane Austen is best known IMPROVED: Jane Austen is best known for her novel Pride and prejudice.for her novel Pride and prejudice.



9. It is suggested that the crucial elements of 9. It is suggested that the crucial elements of an item be placed at the end of the statement.an item be placed at the end of the statement.

FAULTY: Oxygen reduction occurs more readily FAULTY: Oxygen reduction occurs more readily because carbon monoxide combines with because carbon monoxide combines with hemoglobin faster than oxygen does.hemoglobin faster than oxygen does.

IMPROVED: Carbon monoxide poisoning occurs IMPROVED: Carbon monoxide poisoning occurs because carbon monoxide dissolves delicate because carbon monoxide dissolves delicate lung tissue.lung tissue.


Matching TypeMatching TypeMatch List A with List B. You will be given one Match List A with List B. You will be given one point for each correct match.point for each correct match.

List AList A List BList Ba. cotton gina. cotton gin a. Eli Whitneya. Eli Whitneyb. reaperb. reaper b. Alexander Graham Bellb. Alexander Graham Bellc. wheelc. wheel c. David Brinkleyc. David Brinkleyd. TU54G tubed. TU54G tube d. Louisa May Alcottd. Louisa May Alcotte. steamboate. steamboat e. None of thesee. None of these

•Directions failed to specify the basis for matching•List are enumerated identically•Responses not listed logically•Lacks homogeneity•Equal number of elements •Use of “None of the above”


Matching TypeMatching Type

IMPROVED: Famous inventions are listed in the left-hand IMPROVED: Famous inventions are listed in the left-hand column and inventors in the right-hand column below. column and inventors in the right-hand column below. Place the letter corresponding to the inventor in the space Place the letter corresponding to the inventor in the space next to the invention for which he s famous. Each match is next to the invention for which he s famous. Each match is worth 1 point, and “None of these” may be the correct worth 1 point, and “None of these” may be the correct answer. Inventors may be used more than once.answer. Inventors may be used more than once.

InventionsInventions InventorsInventors__ 1. steamboat__ 1. steamboat a. Alexander Graham-Bell a. Alexander Graham-Bell __ 2. cotton skin__ 2. cotton skin b. Robert Fultonb. Robert Fulton__ 3. sewing machine__ 3. sewing machine c. Elias Howec. Elias Howe__ 4. reaper__ 4. reaper d. Cyrus McCormickd. Cyrus McCormick

e. Eli Whitneye. Eli Whitneyf. None of thesef. None of these

Multiple ChoiceMultiple Choice

1. Salvador Dali is1. Salvador Dali is

a. a famous Indian.a. a famous Indian.

b. important in international law.b. important in international law.

c. known for his surrealistic art.c. known for his surrealistic art.

d. the author of many avant-garde plays. d. the author of many avant-garde plays.

•It is recommended that the stem be a direct question.•The stem should pose a clear, define, explicit, and singular problem.



IMPROVED: With which one of the fine IMPROVED: With which one of the fine arts is Salvador Dali associated?arts is Salvador Dali associated?

a. surrealistic paintinga. surrealistic painting

b. avant-garde theatreb. avant-garde theatre

c. polytonal symphonic musicc. polytonal symphonic music

d. impressionistic poetryd. impressionistic poetry


2. Milk can be pasteurized at home by2. Milk can be pasteurized at home by

a. heating it to a temperature of 130a. heating it to a temperature of 130oo

b. Heating it to a temperature of 145b. Heating it to a temperature of 145oo

c. Heating it to a temperature of 160c. Heating it to a temperature of 160oo

d. Heating it to a temperature of 175d. Heating it to a temperature of 175oo

•Include in the stem any words that might otherwise be repeated in each response.



IMPROVED: The minimum temperature IMPROVED: The minimum temperature that can be used to pasteurize milk at that can be used to pasteurize milk at home is:home is:

a. 130a. 130oo

b. 145b. 145oo

c. 160c. 160oo

d. 175d. 175oo

Multiple ChoiceMultiple Choice3. Although the experimental research, particularly that by 3. Although the experimental research, particularly that by

Hansmocker must be considered equivocal and Hansmocker must be considered equivocal and assumptions viewed as too restrictive, most testing assumptions viewed as too restrictive, most testing experts would recommend as the easiest method of experts would recommend as the easiest method of significantly improving paper-and-pencil achievement significantly improving paper-and-pencil achievement test reliability totest reliability toa. increase the size of the group being tested.a. increase the size of the group being tested.b. increase the differential weighting of items.b. increase the differential weighting of items.c. increase the objective of scoring.c. increase the objective of scoring.d. increase the number of items.d. increase the number of items.e. increase the amount of testing time.e. increase the amount of testing time.

Items should be stated simply and understandably, excluding all nonfunctional words from stem and alternatives.



IMPROVED: Assume a 10-item, 10-minute IMPROVED: Assume a 10-item, 10-minute paper-and-pencil multiple choice achievement paper-and-pencil multiple choice achievement test has a reliability of .40. The easiest way of test has a reliability of .40. The easiest way of increasing the reliability to .80 would be to increasing the reliability to .80 would be to increasedincreaseda. group sizea. group sizeb. scoring objectivityb. scoring objectivityc. differential item scoring weightsc. differential item scoring weightsd. the number of itemsd. the number of itemse. testing timee. testing time


4. None of the following cities is a state 4. None of the following cities is a state capital exceptcapital except

a. Bangora. Bangor

b. Los Angelesb. Los Angeles

c. Denverc. Denver

d. New Havend. New Haven

•Avoid negatively stated items



IMPROVED: Which of the following cities IMPROVED: Which of the following cities is a state capital?is a state capital?

a. Bangora. Bangor

b. Los Angelesb. Los Angeles

c. Denverc. Denver

d. New Havend. New Haven


5. Who wrote Harry Potter and the Goblet 5. Who wrote Harry Potter and the Goblet of Fire?of Fire?

a. J. K. Rowlinga. J. K. Rowling

b. Manny Paquiaob. Manny Paquiao

c. Lea Salongac. Lea Salonga

d. Mark Twaind. Mark Twain

•If possible the alternatives should be presented in some logical, numerical, or systematic order.•Response alternatives should be mutually exclusive.



IMPROVED: Who wrote Penrod?IMPROVED: Who wrote Penrod?

a. J. K. Rowlinga. J. K. Rowling

b. J. R. R. Tolkienb. J. R. R. Tolkien

c. V. Hugo c. V. Hugo

d. L. Carrold. L. Carrol

Multiple ChoiceMultiple Choice6. Which of the following statements 6. Which of the following statements makes clear the meaning of the word makes clear the meaning of the word “electron”?“electron”?

a. An electronic toola. An electronic tool

b. Neutral particlesb. Neutral particles

c. Negative particlesc. Negative particles

d. A voting machined. A voting machine

e. The nuclei of atomse. The nuclei of atoms•Make all responses plausible and attractive to the less knowledgeable and skillful student.



IMPROVED: Which of the following IMPROVED: Which of the following phrases is a description of an “electron”?phrases is a description of an “electron”?

a. Neutral particlea. Neutral particle

b. Negative particleb. Negative particle

c. Neutralized protonc. Neutralized proton

d. Radiated particled. Radiated particle

e. Atom nucleuse. Atom nucleus


7. What is the area of a right triangle 7. What is the area of a right triangle whose sides adjacent to the right angle whose sides adjacent to the right angle are 4 inches long respectively?are 4 inches long respectively?

a. 7a. 7

b. 12b. 12

c. 25c. 25

d. None of the aboved. None of the above

•The response alternative “None of the above” should be used with caution, if at all.



IMPROVED: What is the area of a right IMPROVED: What is the area of a right triangle whose sides adjacent to the right triangle whose sides adjacent to the right angle are 4 inches and 3 inches angle are 4 inches and 3 inches respectively?respectively?a. 6 sq. inchesa. 6 sq. inchesb. 7 sq. inchesb. 7 sq. inchesc. 12 sq. inchesc. 12 sq. inchesd. 25 sq. inchesd. 25 sq. inchese. None of the abovee. None of the above

Multiple ChoiceMultiple Choice8. As compared with the American factory 8. As compared with the American factory worker in the early part of the 19th century, the worker in the early part of the 19th century, the American factory worker at the close of the American factory worker at the close of the centurycenturya. was working long hoursa. was working long hoursb. received greater social security benefitsb. received greater social security benefitsc. was to receive lower money wagesc. was to receive lower money wagesd. was less likely to belong to a labor union.d. was less likely to belong to a labor union.e. became less likely to have personal contact e. became less likely to have personal contact with employers with employers

Make options grammatically parallel to each other and consistent with the stem.



IMPROVED: As compared with the American IMPROVED: As compared with the American factory worker in the early part of the century, factory worker in the early part of the century, the American factory worker at the close of the the American factory worker at the close of the centurycenturya. worked longer hours.a. worked longer hours.b. had more social security.b. had more social security.c. received lower money wages.c. received lower money wages.d. was less likely to belong to a labor uniond. was less likely to belong to a labor unione. had less personal contact with his employere. had less personal contact with his employer


9. The “standard error of estimate’ refer to9. The “standard error of estimate’ refer to

a. the objectivity of scoring.a. the objectivity of scoring.

b. the percentage of reduced error b. the percentage of reduced error variance.variance.

c. an absolute amount of possible error.c. an absolute amount of possible error.

d. the amount of error in estimating d. the amount of error in estimating criterion scores.criterion scores.

Avoid such irrelevant cues as “common elements” and “pat verbal associations.”



IMPROVED: The “standard error of IMPROVED: The “standard error of estimate” is most directly related to which estimate” is most directly related to which of the following test characteristic?of the following test characteristic?

a. Objectivitya. Objectivity

b. Reliabilityb. Reliability

c. Validityc. Validity

d. Usabilityd. Usability

e. Specificitye. Specificity

Multiple ChoiceMultiple Choice10. What name is given to the group of 10. What name is given to the group of complex organic compounds that occur in complex organic compounds that occur in small quantities in natural foods that are small quantities in natural foods that are essential to normal nutrition?essential to normal nutrition?

a. Caloriea. Calorie

b. Mineralsb. Minerals

c. Nutrientsc. Nutrients

d. Vitaminsd. VitaminsIn testing for understanding of a term or concept, it is generally preferable to present the term in the stem and alternative definitions in the options.



IMPROVED: Which of the following IMPROVED: Which of the following statements is the best description of a statements is the best description of a vitamin?vitamin?

Short Answer TypeShort Answer Type

1. Require short, definite, clear-cut, and 1. Require short, definite, clear-cut, and explicit answersexplicit answers

FAULTY: Earnest Hemingway wrote______FAULTY: Earnest Hemingway wrote______

IMPROVED: The Old Man and the Sea was IMPROVED: The Old Man and the Sea was written by _______.written by _______.

Who wrote The Old man and the Sea? Who wrote The Old man and the Sea?



2. Avoid multimutilated statements2. Avoid multimutilated statements

FAULTY: _____ pointed out in ____ the FAULTY: _____ pointed out in ____ the freedom of thought in America was freedom of thought in America was seriously hampered by ___, ____, & __.seriously hampered by ___, ____, & __.

IMPROVED: That freedom of thought in IMPROVED: That freedom of thought in America was seriously hampered by social America was seriously hampered by social pressures toward conformity was pointed pressures toward conformity was pointed out in 1830 by ______.out in 1830 by ______.



3. If several equal answers equal credit 3. If several equal answers equal credit should be given to each one.should be given to each one.

4. Specify and announce in advance 4. Specify and announce in advance whether scoring will take spelling into whether scoring will take spelling into account.account.


5. In testing for comprehension of terms and 5. In testing for comprehension of terms and knowledge of definition, it is often better to knowledge of definition, it is often better to supply the term and require a definition than to supply the term and require a definition than to provide a definition and require the term.provide a definition and require the term.

FAULTY: What is the general measurement term FAULTY: What is the general measurement term describing the consistency with which items in a describing the consistency with which items in a test measure the same thing?test measure the same thing?

IMPROVED: Define “internal consistency IMPROVED: Define “internal consistency reliability.” reliability.”



6. It is generally recommended that in completion 6. It is generally recommended that in completion items the blanks come at the end of the items the blanks come at the end of the statement.statement.

FAULTY: A (an) ________ is the index obtained FAULTY: A (an) ________ is the index obtained by dividing a mental age score by chronological by dividing a mental age score by chronological age and multiplying by 100.age and multiplying by 100.

IMPROVED: The index obtained by dividing a IMPROVED: The index obtained by dividing a mental age score by chronological age and mental age score by chronological age and multiplying by 100 is called a (an) ________multiplying by 100 is called a (an) ________



7. Minimize the use of textbook expressions 7. Minimize the use of textbook expressions and stereotyped language.and stereotyped language.

FAULTY: The power to declare war is FAULTY: The power to declare war is vested in ______vested in ______

IMPROVED: Which national legislative body IMPROVED: Which national legislative body has the authority to declare war?has the authority to declare war?



8. Specify the terms in which the response is to be 8. Specify the terms in which the response is to be given.given.

FAULTY: Where does the Security Council of the FAULTY: Where does the Security Council of the United Nations hold its meeting?United Nations hold its meeting?

IMPROVED: In what city of the United States does IMPROVED: In what city of the United States does the Security Council of the United Nations hold the Security Council of the United Nations hold its meeting?its meeting?

FAULTY: If a circle has 4-inch diameter, its area FAULTY: If a circle has 4-inch diameter, its area is_____is_____

IMPROVED: A circle has 4-inch diameter. Its area IMPROVED: A circle has 4-inch diameter. Its area in square inches correct to two decimal places, in square inches correct to two decimal places, is _____is _____



9. In general, direct questions are 9. In general, direct questions are preferable to incomplete declarative preferable to incomplete declarative sentences.sentences.

FAULTY: Gold was discovered in California FAULTY: Gold was discovered in California in the year ___in the year ___

IMPROVED: In what year was gold IMPROVED: In what year was gold discovered in California?discovered in California?



10. Avoid extraneous clues to the correct 10. Avoid extraneous clues to the correct answeranswer

FAULTY: A fraction whose denominator is FAULTY: A fraction whose denominator is greater than its numerator is a _____greater than its numerator is a _____

IMPROVED: Fractions whose denominator IMPROVED: Fractions whose denominator are greater than their numerators are are greater than their numerators are called _____called _____


ReferenceReference

Magno, C., & Ouano, J. (2010). Magno, C., & Ouano, J. (2010). Designing Designing Written Assessment for student learningWritten Assessment for student learning. . Manila: Phoenix.Manila: Phoenix.

Activity 4: Insights on Activity 4: Insights on AssessmentAssessment

1.1. After the activities, I realized that my test After the activities, I realized that my test ……

2.2. After the activity, now I know that …After the activity, now I know that …

3.3. After the activity, every time I construct After the activity, every time I construct test, I will …test, I will …

Insights and development of tests

Education

Transcript of Insights and development of tests