Michigan Assessment Michigan Assessment ConsortiumConsortium

Common Assessment Common Assessment Development SeriesDevelopment Series

Putting TogetherPutting TogetherThe Test BlueprintThe Test Blueprint

Developed and Narrated byDeveloped and Narrated by

Bruce R. Fay, PhDBruce R. Fay, PhDAssessment ConsultantAssessment Consultant

Wayne RESAWayne RESA

SupportSupport

The Michigan Assessment Consortium The Michigan Assessment Consortium professional development series in professional development series in common assessment development is common assessment development is funded in part by the funded in part by the Michigan Michigan Association of Intermediate School Association of Intermediate School AdministratorsAdministrators in cooperation with in cooperation with MDE, MSU, Ingham & Ionia ISDs, MDE, MSU, Ingham & Ionia ISDs, Oakland Schools, and Wayne RESA.Oakland Schools, and Wayne RESA.

What You Will LearnWhat You Will Learn

Test blueprints…what they are and Test blueprints…what they are and why you need themwhy you need them

The components of a test blueprintThe components of a test blueprint Criteria for a good test blueprintCriteria for a good test blueprint Test blueprint examplesTest blueprint examples

““If you don't know where If you don't know where you're going, any road will you're going, any road will take you there.”take you there.”

George Harrison (1943 - 2001)George Harrison (1943 - 2001)

"Any Road", Brainwashed, 2002"Any Road", Brainwashed, 2002

Assessment with a PurposeAssessment with a Purpose

Educational assessment is not something Educational assessment is not something incidental to teaching and learning. It is an incidental to teaching and learning. It is an equal partner with curriculum and equal partner with curriculum and instruction. It is the critical “3instruction. It is the critical “3 rdrd leg” through leg” through which both students and teachers receive which both students and teachers receive feedback about the effectiveness of the feedback about the effectiveness of the teaching and learning process in achieving teaching and learning process in achieving desired learning outcomes. Assessment desired learning outcomes. Assessment closes the loop.closes the loop.

Closed–LoopClosed–Loop(Feedback) Systems(Feedback) Systems

Home Heating System(Teaching & Learning)

Desired Temperature(Learning Target)

Actual Temperature(Test Results)

C – I – A AlignmentC – I – A Alignment

Requires thoughtful alignment – Requires thoughtful alignment – ensuring that the items on a test fairly ensuring that the items on a test fairly represent the…represent the… Intended learning targets (intended Intended learning targets (intended

curriculum)curriculum) Actual learning targets (taught Actual learning targets (taught

curriculum)curriculum)

Test what you teach, teach what you testTest what you teach, teach what you test

Target–Level AlignmentTarget–Level Alignment

Relative importance of those targetsRelative importance of those targets Level of cognitive complexity Level of cognitive complexity

associated with those targetsassociated with those targets

Useful feedback requires Useful feedback requires tests that are…tests that are… Reliable (consistent; actually measure Reliable (consistent; actually measure

something)something) Fair (Free from bias or distortions)Fair (Free from bias or distortions) Valid (contextually meaningful or Valid (contextually meaningful or

interpretable; can reasonably support interpretable; can reasonably support the decisions we make based on the decisions we make based on them)them)

Test BlueprintsTest BlueprintsThe Big IdeaThe Big Idea

A simple but essential tool, used A simple but essential tool, used to:to: Design tests that can meet the Design tests that can meet the

preceding requirementspreceding requirements Define the acceptable evidence to Define the acceptable evidence to

infer mastery of the targetsinfer mastery of the targets Build in evidence for validityBuild in evidence for validity

The Test Blueprint orThe Test Blueprint orTable of Test SpecificationsTable of Test Specifications

Explicitly “map” test items to:Explicitly “map” test items to: Learning TargetsLearning Targets Levels of ComplexityLevels of Complexity Relative ImportanceRelative Importance

Provides “common” definition of the Provides “common” definition of the testtest

Learning Targets &Learning Targets &Standards FrameworksStandards Frameworks Standards as structured hierarchical Standards as structured hierarchical

frameworks. Michigan’s is:frameworks. Michigan’s is: StrandsStrands StandardsStandards DomainsDomains Content ExpectationsContent Expectations

Detailed curriculum is usually left to Detailed curriculum is usually left to local districts or classroom teachers.local districts or classroom teachers.

A Simple Taxonomy ofA Simple Taxonomy ofCognitive ComplexityCognitive Complexity

Norm Webb’s Depth Of Knowledge (1997)Norm Webb’s Depth Of Knowledge (1997)(highest to lowest)(highest to lowest)

Extended ThinkingExtended Thinking Strategic ThinkingStrategic Thinking Skill / concept use / applicationSkill / concept use / application RecallRecall

Putting it all together…Putting it all together…A Basic Test BlueprintA Basic Test Blueprint

Table (matrix) format (spreadsheet)Table (matrix) format (spreadsheet) Rows = learning targets Rows = learning targets (one for each)(one for each)

Columns = Depth of KnowledgeColumns = Depth of Knowledge Cells = number of items and points Cells = number of items and points

possiblepossible

Summary InformationSummary Information

Number of items and points possible:Number of items and points possible:

Row Margins = for that targetRow Margins = for that target

Column Margins = for that level of Column Margins = for that level of complexitycomplexity

Lower Right Corner = for the testLower Right Corner = for the test

Example 1 – Basic BlueprintExample 1 – Basic Blueprintfor a test with 5 learning targetsfor a test with 5 learning targets

LearningTargets

Recall# (pts)

Use# (pts)

Strategic# (pts)

Extended#(pts)

Target Totals# (pts)

Target 1 3 (3) 2 (2) 5 (5)

Target 2 1 (1) 2 (2) 2 (4) 5 (7)

Target 3 2 (2) 1 (3) 3 (5)

Target 4 3 (3) 1 (2) 4 (5)

Target 5 2 (4) 1 (4) 3 (8)

Level Totals# (pts)

6 (6) 10 (14) 4 (10) 20 (30)

Is this reasonable?Is this reasonable?Rule of Thumb Criteria…Rule of Thumb Criteria…

At least 3 items per target for reliabilityAt least 3 items per target for reliability Appropriate:Appropriate:

Distribution of items over targetsDistribution of items over targets Levels of complexity for Levels of complexity for

targets/instructiontargets/instruction Distribution of items over levels of Distribution of items over levels of

complexitycomplexity

(all items are NOT at the lowest or highest level)(all items are NOT at the lowest or highest level)

Professional JudgmentProfessional Judgment

Like all things in education, the Like all things in education, the development of assessments and the development of assessments and the use of the results are dependent on use of the results are dependent on professional judgment, which can be professional judgment, which can be improved through…improved through… ExperienceExperience CollaborationCollaboration Reflection on methods Reflection on methods Results Results

Limitations…Limitations…

Shows total points for each target/level Shows total points for each target/level combination, but not how those points combination, but not how those points apply to each itemapply to each item

Doesn’t show item typesDoesn’t show item types Doesn’t indicate if partial credit scoring Doesn’t indicate if partial credit scoring

can/will be used (but may be implied)can/will be used (but may be implied) But…it was easy to construct, is still a But…it was easy to construct, is still a

useful blueprint, and is much better than useful blueprint, and is much better than not making one!not making one!

Add details on item type Add details on item type and format to ensure…and format to ensure…

Appropriate match to learning targets and Appropriate match to learning targets and associated levels of complexityassociated levels of complexity

Balanced use within tests and across tests Balanced use within tests and across tests over timeover time

Specification of test resources, i.e. –Specification of test resources, i.e. – Calculators, dictionaries, measuring tools…Calculators, dictionaries, measuring tools…

Track on same or separate spreadsheetTrack on same or separate spreadsheet

Common item types Common item types include…include…

Selected-response Selected-response Multiple-choiceMultiple-choice MatchingMatching

Constructed-response Constructed-response Brief (fill-in-the-blank, short answer, sort a list)Brief (fill-in-the-blank, short answer, sort a list) Extended (outline, essay, etc.)Extended (outline, essay, etc.)

PerformancePerformance ProjectProject PortfolioPortfolio

Complexity vs. UtilityComplexity vs. Utility

Your test blueprint could get Your test blueprint could get complicated if you try to account for complicated if you try to account for too much in one spreadsheet.too much in one spreadsheet.

Make sure your test blueprint covers Make sure your test blueprint covers the basics, is not a burden to create, the basics, is not a burden to create, and is useful to youand is useful to you

The following example is slightly more The following example is slightly more complicateded, but still workablecomplicateded, but still workable

TargetCode

Item#

Item type SR

Item type CR-B

Item type CR-E

DOKRecall

Pts

DOKUse

Pts

DOKStrategic

Pts

Trgt Tots(pts)

1.1.1.1 1 x 1

1.1.1.1 2 x 1

1.1.1.1 3 x 2 4

1.2.3.4 4 x 3

1.2.3.4 5 x 5 8

etc etc

Col Tots 5 2 2 1 1 7 5 12

Example 2 – BlueprintExample 2 – Blueprintwith Explicit Items and Item Typeswith Explicit Items and Item Types

Beyond the Test BlueprintBeyond the Test Blueprint

Answer key (selected-response items)Answer key (selected-response items) Links to scoring guides & rubricsLinks to scoring guides & rubrics Specs for external test resourcesSpecs for external test resources Item numbering for alternate test formsItem numbering for alternate test forms

ConclusionsConclusions

Destination & Road MapDestination & Road Map Alignment/balance of items/types for…Alignment/balance of items/types for…

learning targets (curriculum /content)learning targets (curriculum /content) size (complexity) of targetssize (complexity) of targets cognitive level of targetscognitive level of targets relative importance of targetsrelative importance of targets Spec or document other aspects of the Spec or document other aspects of the

testtest