Finding CoherenceFocus, Measure, & Connect

The Measure Sessions!January 20th, 2015http://digitallearningforallnow.com

http://www.slideshare.net/jpcostasr

costa@educationconnection.org

Jonathan P. Costa

Three Principles of Coherence

Measure what you

value

Value what you measure

Priority Student Learning

Priority Adult

LearningPriority Systems Learning

Student Learning

Adult Learning

Systems Learning

MeasureFocus Connections

Coherence Pathways

G = Goals P = Practices M= Measures Jonathan P. Costa S= Students A = Adults O = District/Building

Mission

To develop in all children the

knowledge, skills, attitudes and

values...

Theory of Action

FocusMeasureConnect

Student DataDriving

InstructionalPractices &

DecisionMaking

InstructionDriving

ImprovedMeasures

ProfessionalPracticeDriving

ImprovedMeasures

SystemsDriving

ImprovedMeasures

Adult DataDriving

ProfessionalLearning &

DecisionMaking

OrganizationalData

DrivingSystemsDecisionMaking

Coherence Is Supported by Assessment

Jonathan P. Costa

Value

Reliability

Align Assessments With Goals for Learning

Appropriate

Assessment

Instruments

Recall & recognition

based quizzes, tests,

and activities. Multiple

choice, matching, fill

In the blanks, etc.

(SAT/AP/Exams)

Checklists,

analytic rubrics,

or other agreed upon

skill standards

(AP/Smarter B/Exams)

Holistic and,

analytic rubrics,

or other agreed upon

skill standards

(Portfolios, Demonstrations,

Exhibitions, Etc)

Content(Declarative)

Facts

“Define this term”

Content Skills(Procedural)

Discrete Skills

“Factor this equation”

Meta Skills(Contextual)

Applied Understandings

“Skilled Information Consumer”

Type of

Knowledge

Desired

Appropriate

Assessment

Standards/Scales

Take this quiz, test,

or other appropriate

content focused

assessment instrument.

Answering/responding to

classroom or text based

problems, experiments, and

writing assignments.

Complex projects,

real time explorations,

authentic problem based

skill applications.

Appropriate

Assessment

Tasks

Percentages of

right and wrong

answers on an agreed

upon numeric or letter

grade scale.

1-100/A-B-C-D

Specific criteria tied

to an agreed upon

numeric or

descriptive scale.

1-100/Pass-Fail

Insufficient-Excellent

Specific criteria tied

to an agreed upon

descriptive scale.

Insufficient-Sufficient

Proficient-Excellent

Overlap between columns can and does occur.

Reflections on Assessment…

Other data points: Required, by grade, department, standards aligned, type of administration, time required, testing window, vendor, ROI

Reflections on Assessment…

To be Effective, Formative or Summative Assessments Should Be:

1. Generally Reliable (as opposed to statistically so)

2. Valid (reflects what it is intended to measure)

3. Helps the learner improve his/her performance (formative mostly)

4. Can be tracked and analyzed over time (summative mostly)

Building Technical Capacity

• Constructing effective rubrics – measuring what you value

• Scoring protocols – build capacity to judge student work

As a General Rule:

• The goal is the reduce the scoring range – to increase the reliability of the feedback

• Transparency with the standards – sharing standards with the learners and providing them with effective feedback to improve performance.

The ParetoLeverage Principle

VitalFew

Some things are more

important than others.

20/80

Pareto Is Not a New Idea

I served with General Washington in the Virginia Legislature and with Dr. Franklin in Congress. I never heard either of them speak to any but the main point that was to decide the question. They laid their shoulders to the great points, knowing that the little ones would follow themselves.

Thomas Jefferson

The Biggest Challenge

Getting together and building the structures from which to give students the feedback they need to ensure success and accountability in these priority skill areas is the hardest part of this work.

Please write 5 words that describe this winter’s weather in Connecticut.

Keep this shift in mind…

- From -View Judge Explain

- To -Establish a reliable standard; then…

View Analyze Discuss

It’s our natural instinct…Think back to the five words I asked you to

write to DESCRIBE the winter weather in Connecticut…

Pay close attention to the language.

Judgment - Descriptive“reliable” vs. “corroborated by at least

one other source”

“effective” “1st person narrative”

To help this shift…

Narrowing the Range

• As a general rule, descriptive language is preferable to judgmental language for the purpose of narrowing the range of scores.

• The more judgmental the language, the greater the time investment required for calibration and the more dependent on personnel the process.

• The goal is find the balance time – descriptive enough to effectively communicate without being too restrictive or unrealistic.

Getting Some Descriptive PracticeWhat is it – why is it seen that way?

Element

Spring day…

Turkey sandwich…

Student engagement…

Written summary…

Argument with evidence…

More DescriptiveMore Judgmental

Making an Argument Based on Evidence

Item Insufficient Sufficient Proficient Excellent

Stated Position

Rationale

Supporting

Evidence

Closing

Statement

Something to Think About• There is no shortcut to the amount of time

required to use standards and assessment effectively.

• You either have to define the standard from scratch through descriptive language or you will have to define it through working with your peers to develop anchor sets/models of student work.

• The best strategy is probably a combination of both approaches.

Scoring Practice

• Match evidence in the student work to the descriptors in the rubric and scoring guide.

• Scoring decisions must be based on particular evidence in the student work. Be able to point to the evidence in the student work that is relevant to scoring fairly and consistently.

• You must be indifferent to the student – about what is demonstrated not about what you believe was intended.