The Learning Side of Testing Milwaukee Area Technical College Dental Hygiene Program February 25,...

The Learning Side of TestingThe Learning Side of TestingThe Learning Side of TestingThe Learning Side of Testing

Milwaukee Area Technical College

Dental Hygiene ProgramFebruary 25, 2005

Test to LearnTest to LearnTest to LearnTest to LearnLearn to TestLearn to TestLearn to TestLearn to Test

What does this mean? What does it tell us about how What does this mean? What does it tell us about how testing fits into performance-based learning?testing fits into performance-based learning?

Situational Analysis

Plus Minus Interesting

National National Board Board ResultsResults

Testing Testing PracticesPractices

OtherOther

What do we know about National Board Results?What do we know about National Board Results?

0

50

100

150

200

250

2001 2002 2004

MATC

All

MATC performance in comparison to all others.


MATC failure rate.


0.00%5.00%

10.00%15.00%20.00%25.00%30.00%35.00%40.00%45.00%50.00%

2000 2001 2002 2003 2004

April

December




-5

-4

-3

-2

-1

0

1 Anat. Sci

Physo-Bio-Nutr

Micro-Immun

Patho

Pharm

Patient Assess

Radiology

Care Mgt.

Perio

Supp TRT

Comm Hlth

Case-Based

What do we know about testing practices?What do we know about testing practices?


What else might impact the issues?What else might impact the issues?


Testing as Learning

Learning Learning requires

requires practice!practice!

Learning Learning requires

requires practice!practice!

Testing promotes learning.

Testing requires learning.

Sensing MemorySensing MemorySensing MemorySensing Memory

selector

large capacity

short duration

Practice and Learning

Working MemoryWorking MemoryWorking MemoryWorking Memory

processor

powerful

fragile

short duration

small capacity


Long-Term MemoryLong-Term MemoryLong-Term MemoryLong-Term Memory

information storage

large capacity requires retrieval


Five mental processes control information flow:Five mental processes control information flow:

attention (sensingsensing)

encoding (working/long-term)

rehearsal/practice (working)

retrieval (long-term/working)

metacognition (working)

attention (sensingsensing)

encoding (working/long-term)

process/practice (working)

retrieval (long-term/working)

sensing memory attends to the stimulus

Learning is enduring and meaningful only when the:

working memory processes it and sends it to long-term memory

long-term memory stores it


Learners apply Learners apply what they’ve what they’ve learned only learned only when they can when they can retrieveretrieve it from it from long-term long-term memory .memory .

Aha! It’s an

occlusal.


Research has found that providing learners with retrieval practice is much more powerful than giving them the same amount of time to restudy the learning material. Typical improvements range from 30% to 100%.


Will ThalheimerSimulation-Like Questions: How and Why to Write Them

www.work-learning.com

http://www.work-learning.com/

Learners need feedback in order to take responsibility for monitoring and adjusting their own learning.


. . . hmm, the answer to this question isn’t

what I thought it was. What do I need to do

about it?

. . . hmm, the answer to this question isn’t

what I thought it was. What do I need to do

about it?

MetacognitionMetacognitionMetacognitionMetacognition

Feedback is essential when learners answer questions because it helps them correct their misconceptions.

Feedback typically improves learning outcomes by 15 – to 50%.


Will ThalheimerSimulation-Like Questions: How and Why to Write Them

www.work-learning.com

http://www.work-learning.com/

Too much information too fast yieldsToo much information too fast yields

Fried Brain Syndrome


Cognitive Cognitive OverloadOverloadCognitive Cognitive OverloadOverloadCognitive Cognitive OverloadOverload

Fried brain syndrome leads to:Fried brain syndrome leads to:

Cognitive overload slows down or even stops learning!


To avoid cognitive overload:To avoid cognitive overload:To avoid cognitive overload:To avoid cognitive overload:

• break learning into manageable chunks

• clear the working memory with meaningful practice


Performance vs. Practice

• Intended to accomplish a task

• Real thing/big bites• On your own• High stakes• Feedback after the fact• Limited opportunities

to correct• School of hard knocks

• Intended to promote learning

• Protected/chunked• Coached• Low stakes• Concurrent or

immediate feedback• Repetition• Built-in opportunities to

correct

Testing as Learning

ToolTool DescriptionDescription Measure vs.Measure vs.LearnLearn

StakesStakes

Educational Assessment

Process of obtaining evidence about the learners’ knowledge, skills, and attitudes.

Exam Measure/document knowledge or skills for the purpose of credentialing (graded).

Measure High

Test Measure knowledge or skills for the purpose of informing the learner and teacher about the learner’s current level of knowledge or skill. (likely to be graded)

Measure- Promote Learning

Medium

Quiz Measure a learner’s knowledge or skills for the purpose of providing feedback and identifying gaps.

Promote Learning

Low

CAT Provide immediate feedback to teachers and learners to guide adjusts in teaching and learning.

Promote Learning

Very Low

Slide from derived http://questionmark.com

http://questionmark.com/

Testing as Learning

Testing provides practice.

Testing requires practice.

What does this mean?What does this mean?What does this mean?What does this mean?

Testing as Learning

Strategies for Using Testing as LearningStrategies for Using Testing as LearningStrategies for Using Testing as LearningStrategies for Using Testing as Learning

Require learners to “test drive” their knowledge by completing low stakes quizzes at all stages of learning. [Could be self-check.]

Formulate classroom quizzes and tests using the same test formats used in the National Boards.

Present learner with questions that require higher level thinking to make decisions, determine solutions, or draw conclusions.

Testing as Learning

Strategies for Using Testing as LearningStrategies for Using Testing as LearningStrategies for Using Testing as LearningStrategies for Using Testing as Learning

Design questions so that learners apply knowledge in the context of Dental Hygiene practice.

Provide corrective feedback for incorrect answers.

Build in branching to present additional questions after missed questions.

Teach learners early and often how to succeed on tests and quizzes.

What do you know about writing multiple What do you know about writing multiple choice test items?choice test items?

1. What should you do?

2. What makes a good question?

3. What should you avoid?

Question Writing Process

Question Writing Process

• Start with your competencies and learning objectives

• Write question stems

• Write correct answers

• Write distractors

• Edit

Start with target competencies and learning objectives to:

• ensure validity• ascertain significance• check coverage• determine scope of test

Write the Stem

• Derive the question from a competency or learning objective.

• Identify target decision, conclusion, or solution.

• Present a single, definitive problem with a definitive answer.

• Make stem grammatically consistent and parallel in form to options.

• Negatively phrased questions (used in National Boards)

• Absolutes such as “always” and “never”• Unnecessary information• Obvious clues about the answer• Bias (gender, cultural, ethnic, regional)• Humor and trick questions

Write the Stem

What to avoid:

Avoid Grammatical Clues

Option A is the only option that makes grammatical sense.

The best way to increase the reliability of a test is to:

A. Increase the test length. [Correct]

B. Removing poor quality items.

C. Test should be readable for all test takers.

Avoid Association Clues

Option B includes the word “seven.”

Who are the “Magnificent Seven?”

A. A pro basketball team [Correct]B. A group of seven fictional, heroic,

Western figures [Correct]C. A protest group made famous during

the 1960sD. A vegetarian rock group

• Write three to five options. • Make all options grammatically consistent

and parallel in form to stem.• Write all options so they are comparable in

content, length.• Ensure that options are not repetitious.• Order them logically or numerically. • Avoid “All of the above” and “None of the

above.”

Write the Options

What is the cost of an item that normally sells for 9.99 USD that is discounted 25%?

Order Options Numerically

WrongA. 5.00B. 7.50C. 2.50D. 6.66

RightA. 2.50 B. 5.00C. 6.66D.7.50

What are the three important concerns in achievement testing? V=validity R=reliability E=efficiency O=objectivity

Order Options Logically

WrongA. V, R, E B. O, V, RC. R, O, E D. E, O, V

RightA. V, R, O B. V, R, E C. V, E, O D.R, E, O

• Write two to four distractors (giving 3-5 options).• Make all distractors plausible.• Make distractors similar in form or otherwise

harmonious with the answer.

Write the Options - Distractors

Write the Options - Correct Answer

• Be sure that correct answer is the only correct one or clearly the best one.

• Vary the correct answer position in the list of options.

Raising the Level of Thinking

• Evaluation

• Synthesis

• Analysis

• Application

• Comprehension

• Knowledge

Critical Critical ThinkingThinkingCritical Critical ThinkingThinking

definedefinelabellistrecallrecognize

Evaluation

Synthesis

Analysis

Application

ComprehensionKnowledgeKnowledge


explainexplainparaphrasedescribesummarize

Evaluation

Synthesis

Analysis

ApplicationComprehensionComprehension

Knowledge


Evaluation

Synthesis

AnalysisApplicationApplication

Comprehension

Knowledge

applydecidedecidedetermineuse


Evaluation

SynthesisAnalysisAnalysis

Application

Comprehension

Knowledge

classifycomparedescribe structuredetermine cause/effectdetermine cause/effectdetect patternsdiagnose


Evaluation SynthesisSynthesis

Analysis

Application

Comprehension

Knowledge

constructcreatecreatedesigndevelopformulatewrite


EvaluationEvaluation

Synthesis

Analysis

Application

Comprehension

Knowledge

appraiseassessconcludeevaluate evaluate prescribe


How do you help learners practice higher level How do you help learners practice higher level thinking skills?thinking skills?How do you help learners practice higher level How do you help learners practice higher level thinking skills?thinking skills?

make a decision draw a conclusion solve a problem

Continually present questions that require them to:


Tools

• Higher Level Question Worksheet

• Case-Based Question Planner

PerceptionA Web-based Tool for Q&A Assessment

PerceptionA Web-based Tool for Q&A Assessment

Create questions and group into quizzes,

tests, exams and surveys using a web browser or Windows

PC

Specify security

parameters, schedule

assessments and potentially

link to other systems

Deliver assessment using a standard, or secure, browser, or using a Windows PC

Report on the results. Reports include coaching reports, item analysis, gap analysis, and others.

Low Medium High

Consequences to candidate

Legal liabilities

Proctoring standards

Validity & Reliability checking

Planning (testing and itemsignificance)

Psychometrician involvement

Slide from http://questionmark.com

The Dental Hygiene National Boards

High Stakes ExamHigh Stakes ExamHigh Stakes ExamHigh Stakes Exam

http://questionmark.com/


Context and PurposeContext and Purpose

Intended to assess ability to apply knowledge in a problem-solving context

Key to qualifying for licensure

Developed by dental hygiene and science educators

Subject to on-going validity and reliability checking

Administered in highly secure environment with significant consequences for cheating and breach of security

Administered according a consistent set of strict rules

Administered to a large, diverse population across a broad geographic area.


ScopeScope

Discipline-Based (200 questions)Scientific Basis for Dental Hygiene PracticeProvision of Clinical Hygiene Services Community Health Services

Behavioral Sciences (infused into 5%)

Ethics and Risk Management (infused into 5%)

Cased-based items (150Assessing patient characteristicsObtaining and interpreting radiographsPlanning and managing dental hygiene carePerforming periodontal proceduresUsing preventative agentsProviding supportive treatment service


Format – Multiple Choice:Format – Multiple Choice:

Completion

Question

Negative

Paired True-False

Cause-and-Effect

Testlet

Case-Based

Seven Question Types


Format – Format – CompletionCompletionTypical Level – Typical Level – Format – Format – CompletionCompletionTypical Level – Typical Level –

Radiographic intensifying screens are used forExample

A. magnifying lamps.B. increasing detail.C. reducing the exposure time.D. speeding processing time.E. decreasing the processing time.

KnowledgeKnowledge KnowledgeKnowledge


Format – Format – QuestionQuestionTypical Level – Typical Level – Format – Format – QuestionQuestionTypical Level – Typical Level –

Root hypersensitivity diminishes as the tooth forms which of the following?

Example

A. Mantle dentinB. Secondary dentinC. Cellular cementumD. Acellular cementum

Knowledge, Understanding, ApplicationKnowledge, Understanding, Application (all)(all)Knowledge, Understanding, ApplicationKnowledge, Understanding, Application (all)(all)


Format – Format – NegativeNegativeTypical Level – Typical Level – Format – Format – NegativeNegativeTypical Level – Typical Level –

Each of the following characterizes an ideal chemical disinfectant EXCEPT one. Which is the EXCEPTION?

Example

A. Heat-stableB. Long shelf-lifeC. Water solubilityD. Activity against vegetative microbesE. Activity against microbial spores

Knowledge, Understanding, (all)Knowledge, Understanding, (all) Knowledge, Understanding, (all)Knowledge, Understanding, (all)


Format – Format – Paired True-FalsePaired True-FalseTypical Level – Typical Level – Format – Format – Paired True-FalsePaired True-FalseTypical Level – Typical Level –

Protection from excessive exposure to radiation is aided by the use of aluminum filters and a lead diaphragm.

The filters reduce the amount of soft radiation.

Example

A. Both statements are TRUE.B. Both statements are FALSE.C. The first statement is TRUE; the second is FALSE.D. The first statement is FALSE; the second is TRUE.

Understanding, Analysis, EvaluationUnderstanding, Analysis, Evaluation Understanding, Analysis, EvaluationUnderstanding, Analysis, Evaluation


Format – Format – Cause-and-EffectCause-and-EffectTypical Level – Typical Level – Format – Format – Cause-and-EffectCause-and-EffectTypical Level – Typical Level –

Adolescent growth spurts more in the maxilla than in the mandible BECAUSE, at puberty, the lymphoid tissue present in the nasopharynx decreases.

Example

A. Both the statement and the reason are correct.B. The statement is correct, but the reason is NOT.C. The statement is NOT correct, but the reason is

correct.D. NEITHER the statement NOR the reason is correct.

AnalysisAnalysis AnalysisAnalysis


Format – Format – Testlet (Community Health)Testlet (Community Health)Typical Level – Typical Level – Format – Format – Testlet (Community Health)Testlet (Community Health)Typical Level – Typical Level –

Presents a community health scenario followed by five or more multiple-choice items that address its content. Questions are likely to be in the question format, but could be in any format.

For an example see 2005 National Board Dental Hygiene Examination Candidate Guide, p. 40.

Description

Application, Analysis, EvaluationApplication, Analysis, Evaluation Application, Analysis, EvaluationApplication, Analysis, Evaluation


Format – Format – Case-Based (Patient Care)Case-Based (Patient Care)Typical Level – Typical Level – Format – Format – Case-Based (Patient Care)Case-Based (Patient Care)Typical Level – Typical Level –

Present patient histories, dental charts, radiographs (sometimes intra- and extra-oral photographs). Test items, typically in the question format, address functions that a dental hygienist is expected to be able to perform.

Exams will include at least one case regarding the following types of patients: geriatric, adult periodontal, pediatric, special needs, medically compromised.

For an example see 2005 National Board Dental Hygiene Examination Candidate Guide, pp. 41-45.

Description (12-15 on exam)

Application, Analysis, Evaluation, Application, Analysis, Evaluation, SynthesisSynthesis Application, Analysis, Evaluation, Application, Analysis, Evaluation, SynthesisSynthesis

ValidityValidity Test itemsTest items FeedbackFeedback Use of resultsUse of results

Enhancing Your Tests Critique existing tests.Critique existing tests.Critique existing tests.Critique existing tests.

Enhancing Your Tests Critique existing tests.Critique existing tests.Critique existing tests.Critique existing tests.

Strengths Weaknesses Interesting

Enhancing Your Tests

Outline test improvement strategies.Outline test improvement strategies.Outline test improvement strategies.Outline test improvement strategies.

New and Improved!

New and Improved!

New and Improved!

New and Improved!

Create an assessment plan that includes both performance assessment tasks and Q&A tests.

Use competencies and learning objectives to plan Q&A tests: ensure validity, ascertain significance, check coverage and determine scope.



Review and revise existing tests and exams using the Checklist for Q & A Quizzes and Tests.

Link all test questions to a stated learning outcome (ex. competency, learning objective).

Increase the number of higher thinking level questions. Incorporate feedback that corrects misconceptions and

refers learners who answer incorrectly to remedial or further learning for each question.

Incorporate branching to direct learners to remedial, similar, or higher level questions, as needed. (Online delivery)

Increase the variety of questioning formats used in all quizzes and tests.


Consider using a tool like Questionmark to: support online delivery increased security authoring of varied question formats immediate feedback branching for additional testing referral to learning materials ease of scoring analysis of results, and competency/learning

objective driven test development

Maintenance Practice/Rehearsal

ElaborativePractice/Rehearsal

• keeps information in working memory

• clutters working memory

• inefficient

• connects with what learners already know

• applies to solution of problems

• creates something new• efficiently encodes to

long-term memory

Effective practice is elaborative.Effective practice is elaborative.

Enhancing Their Learning Provide effective practice throughout at all phases and levels of the Provide effective practice throughout at all phases and levels of the learning process.learning process.Provide effective practice throughout at all phases and levels of the Provide effective practice throughout at all phases and levels of the learning process.learning process.

•memorization• repetition• recitation• drill & practice

•classification•analysis• skill practice•critique• problem-solving• product creation

Maintenance Practice/Rehearsal

ElaborativePractice/Rehearsal

Effective practice is elaborative.Effective practice is elaborative.

Enhancing Their Learning Provide effective practice throughout at all phases and levels of the Provide effective practice throughout at all phases and levels of the learning process.learning process.Provide effective practice throughout at all phases and levels of the Provide effective practice throughout at all phases and levels of the learning process.learning process.

Make them work for it by using learner-centered, active learning activities at the comprehension phase.

Guide learners to process information using thinking maps and application exercises

Engage learners in collaborative information gathering and processes activities

Enhancing Their Learning

Present Thinking Maps® or other visual learning tools; have learners create visual learning tools.

Engage learners in the process of developing higher level thinking questions and case studies.

Enhancing Their Learning

The Learning Side of Testing Milwaukee Area Technical College Dental Hygiene Program February 25,...

Documents

Transcript of The Learning Side of Testing Milwaukee Area Technical College Dental Hygiene Program February 25,...