Data Driven Design for Learning

DEPARTMENT OF EMPLOYMENT, EDUCATION AND TRAINING www.deet.nt.gov.au

Data Driven Design for Learning

Teaching, Learning and Standards Division

June 2008


Session Organisation

0800 – 0845: Overview and framing the inquiry0845 – 1000 and 1030 – 1130: Workshops

Data source Purpose Facilitator

A NTCE Using data to identify priorities and needs at system and school level

Sally Hodgson

B Actual Student Evidence of Learning

Designing for learning from actual evidence – classroom focus

Ellen Herden

C NTCF Reports and MAP Reports

Leading school leaders and teachers to use data to identify priorities and set targets - whole school focus

Linda Dawson

D NTCF Reports – Very Remote Focus

Leading school leaders and teachers to look at data to design for learning – classroom focus

Michelle Massey

E NTCF Reports – Provincial/Remote Focus

Leading school leaders and teachers to look at data to design for learning – classroom focus

Learne Dunne


Changing teacher practice and knowledge

Ref: Julia Atkin, 2007

Inventive and CreativeDon’t notice success, know it will happen

Deliberate and ConsciousExperimentation

Success by design

Expect success because of design

Explicit By plan or recipe

Make sense of successTacit

By chance

Hit and miss success

DesignInquiry

QuestioningEvidence based

Reflection and Evaluation Rules and Procedures

Prescriptions Steps


Setting the context

Describing Expert Teachers

1.Knowledge about content

2. Knowledge about learning and learners

3. Knowledge about teaching 4. Pedagogical

content knowledge

School context and

effectiveness factors

System context and

effectiveness factors

Teacher Professional CharacteristicsSources: Hattie 2003, Killen 2006, Hill 2003


Cueing and retrieving knowledge

Gathering Evidence

Judging and Recording Evidence

Feedback and Reporting

Teaching and Learning

Think of a program, initiative, typical school or teacher support, materials development that you currently are focused on.

Where is the starting point in relation to the columns below? Where does this lead to in relation to the four columns?

What questions are considered in each of these columns?


Comparing existing knowledge with research frames

Gathering Evidence




How will we know when learners have demonstrated the

learning outcomes?

What do learners already know?

How will we inform the learner and

others of the learners progress?


Where do learners need and want to

be?

How best do learners learn?

What resources do we have?

What will constitute the learning

journey?

Who will do what?

The 8 Learning Management Questions (Lynch, 2007)


Planning for Teaching


What do I want learners to learn?

What will constitute the learning journey?

What are the best contexts for teaching?

What resources will I use?

Programming and Planning

Gathering Evidence Teaching and Learning

How will I know when learners have demonstrated learning outcomes?

Assessment Considerations

Where do learners need and want to be?

Who will do what?

Making Judgments


Reporting and Feedback

How will I inform the learners and others of their progress?


Planning for Assessment

Gathering Evidence Teaching and Learning

Gathering Evidence, Making

Judgments, Recording

Reporting

How will I know when learners have

demonstrated learning outcomes?

Assessment Plan

What do I want learners to learn?

Plan linked to given outcomes


Implementing assessment tasks

and marking

How will I inform the learners and others of their

progress?

Reporting

What resources will I use?

What will constitute the learning journey?

What are the best contexts for teaching?

Where do learners need and want to be?

Planning and Teaching


Design for learning using the 8 Learning Management Questions (Lynch, 2007)

Gathering Evidence





learning outcomes?






be?




journey?

Who will do what?


Design for learning using the 8 Learning Management Questions

Learning Management Questions

What? •What do learners already know?

So

What?

•Where do learners need and want to be? •How do learners best learn?

Now What?

•How will we know that learners have achieved the learning outcomes?•What resources do we have at our disposal?•What will constitute the learning journey?•Who will do what? •How will we report progress and achievements?



Gathering Evidence





learning outcomes?

What do learners already

know?



What do learners already

know?


be?




journey?

Who will do what?


Demographics

School Programs

Student Learning

Perceptions

What impacts on student learning? What are the data requirements?

NTCF Reports (CeTool/BIC)

MAP/NAPLAN reports

Attendance

ELA Schools-LLANS

School perception surveys


A system is not the sum of its parts, but rather, the product of the interaction of the parts

Russell Ackoff


Lines of Inquiry

1. Snapshots of measuresDemographics, Perception, Programs, Student

Learning

2. Measures over time

3. Two or more variables within a measureStudent Learning – NTCF and MAP

4. Two or more variables within a measure over time

5. Intersection of two measuresStudent Learning and Demographics – MAP by student groups

6. Intersection of two measures over time

7.Intersection of three measuresStudent Learning and Demographics and Programs – MAP by indigenous learners in AL program

8. Intersection of three measures over time

9. Intersection of all four measures 10. Intersection of all four measures over time

Accountability

Achievement

Improvement

Progress


Why use data

Design for learning

Monitoring impact of professional learning

Close the achievement gap

APIF Process– Annual Operational Plans > data analysis to determine

short term goals and targets – School Performance Reports– 4 year Strategic Improvements Plan

Performance Review


Using data to monitor the impact of teacher learning and development

Professional learning focus

Line of Inquiry What data sources are available?

What other data is needed?

Dimensions of Learning

What is the impact of using evidence based instructional strategies on student achievement in ….?

•Actual Evidence of Learning – pre/post •NTCF Reports – on balance judgments term or biannually

•Student perception•Teacher Level of Use of instructional strategies


Data to close the achievement gap

Ruth Johnson (2002) Using data to close the achievement gap: How to measure equity in our schools in Holcomb, 2004, Getting Excited About Data

Schools with large populations of low income students and students of color mitigate perceived achievement barriers by:

• Continuous inquiry and monitoring through the use of data

• High goals, standards, expectations and accountability for students and adults

• Well qualified and culturally competent teachers

• Curriculum content and rigour

Timperley, Wilson, Barrar, and Fung, 2007, Teacher Professional Learning and Development



Gathering Evidence





learning outcomes?





Where do learners need and

want to be?




journey?

Who will do what?


How? - Process

AnalysisData Browse

Define expected levels

Analysis Data Delve

Line of Inquiry into data sources

Interpretation and Goal Setting

Plan evidence based action


Example: Thinking through data analysis flowchart

Define expected levels for this stageor year level of schooling

Define expected levels for this stageor year level of schooling

Look at data for stage/Year level for past three years

(student learning)

Look at data for stage/Year level for past three years

(student learning)

How many students are at, below,or beyond expected

levels?(student learning + demographics)

How many students are at, below,or beyond expected

levels?(student learning + demographics)

Disaggregate data by demographics to understand what groups of students are at, below above expected level?

(student learning + demographics)

Disaggregate data by demographics to understand what groups of students are at, below above expected level?

(student learning + demographics)

What does perception data tell usabout students, teachers

and parents?(perceptions+ demographics)

What does perception data tell usabout students, teachers

and parents?(perceptions+ demographics)

How are we teaching ____ atthis stage/ Year level of schooling?(school processes by demographics)

How are we teaching ____ atthis stage/ Year level of schooling?(school processes by demographics)

How are these data related? How are these data related?

(Bernhardt, 2004:170)


Defining system expected levels


National Benchmarks for

Reading are within

National Benchmarks for Writing are within:

Yr 3 ESL Level 2/3 ESL Level 2

Yr 5 ESL Level 4/5 ESL Level 3/4

Yr 7 ESL Level 6 ESL Level 5/6

What do National Benchmarks look like for ESL Learners?



Gathering Evidence





learning outcomes?






be?

How do learners best

learn?



journey?

Who will do what?


Influences on student learning

.40.30

.15

0

.50

.60

.70

.80

.90

1.0REVERSE

Developmental

Effects

Typical

Teacher

Effects

ZONE OFDESIREDEFFECTS

Using meta-analysis (synthesis) research > impact measured by effect sizes


Robert Marzano’s 1997 meta-analysis instructional strategies


John Hattie’s meta-analysis on what makes a difference to student learning (2003)

0.40 – 0.80 Moderate effect

0.80 +

Large effect


A teaching and learning framework based on evidence

1. Learning is enhanced when learning opportunities are tailored to individuals’ current levels of readiness

2. Learning is more effective when it leads to deep understandings of subject matter.

3. Learning is more effective when learners are supported to monitor and take responsibility for their own learning.


0.27

0.84

0.42

0.34

0.35

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9

Dimension 5: Ensuring anOrderly & Supportive

Environment

Dimension 4: Promoting &Participating in TeacherLearning & Development

Dimension 3: Planning,Coordinating & EvaluatingTeaching & the Curriculum

Dimension 2: StrategicResourcing

Dimension 1: EstablishingGoals & Expectations

Effect Size

Vivianne Robinson’s meta-analysis on leadership (2007)


NT Teaching and Learning Framework

Thinking and Learning Habits

•Thinking on my own

•Thinking together

Positive Attitudes and Perceptions through

•Classroom Tasks

•Classroom Climate

Engaging With and Acquiring

Knowledge•Link to prior learning

•Explicit teaching•Immersion

•Construct meaning and models

Practicing and DeepeningKnowledge

•Organise and store•Shape and Internalise

•Complex reasoning processes•Questioning•Reflecting

•Sustained Dialogue

Using and ConnectingKnowledge Meaningfully

•Complex reasoning processes•Open ended

learning opportunities•Connecting to world beyond

classroom



Gathering Evidence




How will we know when

learners have demonstrated the learning outcomes?






be?




journey?

Who will do what?


Assessment Design PrinciplesMasters and Forster, ARK Developmental Assessment

Assessment Task

Reliability and comparability

How accurately does the collected evidence reflect what the learner

knows and can do?

Curriculum relevance Do the tasks represent all the valued learning you want the students to have undertaken? Do they assess a sample of all

the important aspects of the learning?

Instructional utility

How useful is the method in yielding insights into students

knowledge and conceptual understandings and can be used to guide teaching and learning?

Fairness

Are the tasks constructed, presented and conducted so that all students have an equal chance of demonstrating their

learning against the NTCF?

Do the assessment tasks require specific knowledge or skills which some students may not have, and which have

not been explicitly taught?

Practical Convenience

Is the method and task easily incorporated into classroom activities?


Assessment FOR learning:

occurs when teachers use inferences about student progress to inform

their teaching.

DET Victoria, Assessment Professional LearningModule 5: Making Consistent Judgments

Assessment AS learning:occurs when students reflect on and monitor their progress to inform their future learning goals.

Assessment OF learning:occurs when teachers use evidence of student learning to make judgements on student achievement against goals and standards.

How will we know when learners have achieved the learning outcomes?


What? Lines of Inquiry into data

Have no line of inquiry Have a line of inquiry

Data browse to look for patterns, stand outs

Data Delve to look for specific details

MAP School Comparison Report

NTCF Reports – Year level summary by strand / elements

MAP Student Raw Score Summary

NTCF Report 1.2b – Year level summary by element

MAP Student Grouping Comparison Report

NTCF Reports – A to E or E/S/C spread

MAP Student Achievement Scores

NTCF Report - Graph – compare school terms – Band and ESC

MAP Student Raw Score Summary

Selected samples / students from classroom data

Moderated Samples compared to classroom samples

NTCF Report – Target setting ESL

Moderated samples Student – teacher writing conferences

Line of Inquiry described Solutions to line of inquiry identified, prioritised and planned


What? Data Sources English - Writing

Snapshot, English-only

Multilevel Assessment Program (>2007)

NTCF Reports

System, cluster, school moderated evidence of

learning

Teacher and student assessed evidence of learning

Ongoing

Most useful to the system and

school

Least useful to the system

Least useful to the classroom teacher

Most useful to the classroom

teacher


What? MAP Reports- Writing

School Comparison Report

•Spread of performance across NTCF levels •Percentages at / above / below expected levels

Data BrowseStudent

Comparison Report

•Which groups area at / above/ below expected levels

Student Achievement Summary

Individual student performance by:•Benchmark level achievement – yes / no•Score•NTCF level Data

Delve Raw Score Summary

(Writing)

Individual student performance by criteria•Score for each criteria•Total score

Patterns across all students in cohort – strengths and gaps

Compare to NTCF Reports at element level


Example: Primary School A Summary by Criteria


Gaps•Vocabulary•Cohesion•Punctuation

Criteria ScoreIdeas Yr 3BM = 2

Yr 4 = 3

Vocab Yr 3BM = 3

Yr 4 = 4

Cohesion Yr 3BM = 3 Yr 4 = 4

Punctuation Yr 3BM = 3

Yr 4 = 4

Structure Yr 3BM = 2 Yr 4 = 3

Spelling Yr 3BM = 3 Yr 4 = 4


NTCF Data


Now What? CeTool planning using monitoring data


Business Intelligence Centre – NTCF and MAP Reports

Data Analysis and Reporting Tool

Log On: nightcliffsch-sec and Password: q1w2e3r4

MAPRET

Log On: nightsch-sec and Password: 371346


Frequently Used NTCF Reports

Who Report number

How Organised Best used to….

School and Teachers

3.1b

3.2b

3.3b

Graphs

3.1 Distribution Band levels –Element – School or Year level or Class

3.2 Distribution Band + ESC –Element – School or Year level or Class

3.3 Distribution A to E – Element – School or Year level or Class

Overview of multi-levelness/ distribution

Number of learning at/above/below expected level

Teachers 1.2b •At Year Level or Class Level •Element•Band or ESL level – ESC – Student•ESL Level – ESC – Student

At /above/ below expected levels

Progress across NTCF levels

1.1C At class level•Student – Strands and Elements – Band and ESC

Profile each student

9.1C At class level•Student – Strands and Elements – ESL level and ESC


If you don’t know where you are going anything you do will get you there

Anonymous in Bernhardt, 2004

So What?


Defining system expected levels


National Benchmarks for Reading are within

National Benchmarks for Writing are within:

Yr 3 ESL Level 2/3 ESL Level 2

Yr 5 ESL Level 4/5 ESL Level 3/4

Yr 7 ESL Level 6 ESL Level 5/6

What do National Benchmarks look like for ESL Learners?


NT DEET System Expected Levels

Year Levels NTCF Document Current National Literacy and Numeracy Benchmarks

System

Expected Level linked to A to E

Transition KGP Key Growth Point 2 Comprehensive

1 Band 1 Key Growth Point 3 Solid

2 KGP 3 Comprehensive / Band 1 Emerging

3 Band 2 Band 1 Emerging Band 1 Solid

4 Band 1 Comprehensive / Band 2 Emerging

5 Band 3 Band 1 Comprehensive/Band 2 Emerging

Band 2 Solid

6 Band 2 Comprehensive / Band 3 Emerging

7 Band 4 Band 2 Comprehensive / Band 3 Emerging

Band 3 Solid

8 Band 3 Comprehensive/ Band 4 Emerging

9 Band 5 Predicted based on 2007 trial

Band 3 Comprehensive Band 4 Solid

10 Band 4 Comprehensive/ Band 5 Emerging


System Expected levels School A - English Writing

Year 5 English Writing

KGP 3 Band 1 Band 2 Band 3 Band 4 Band 5

E S C E S C E S C E S C E S C E S C

ESL Phases and Levels

Phase 1 Phase 2 Phase 3

BL 1

BL 2

BL 3

Level 1

Level 2

Level 3

Level 4

Level 5

Level 6

Level 7

Level 8

Year 4 English Writing




Describing School Expected Levels

Teacher Informed

System expected

levels

Teacher informed

School Expected

Levels

Learner informed

Actual Student

Evidence Progress and Achievement

Learner informed


English Writing: Texts and ContextsTerm 2 and Term 4

1 1

7

1

2 2

0

1

2

3

4

5

6

7

8

KGP3 C 1 E 1 S 1 C 2 E 2 S

Band Evidence (short)

1

2

1

7

4

2

0

1

2

3

4

5

6

7

8

KGP2 S KGP3 E KGP3 S 1 E 1 S 2 E

Band Evidence (short)


NTCF 1.2b Report – a Year level picture


NTCF Report 1.1C – a classroom and student picture


NTCF Report – Student Progress - Compare Terms


Jhye

Criteria Score

Cohesion Yr 3 BM =3

Yr 4 = 4

Generic

Structure

Yr 3 BM =2

Yr 4 = 3

Punctuation Yr 3 BM =3

Yr 4 = 4

Quality

Ideas

Yr 3 BM =2

Yr 4 = 3

Vocab Yr 3 BM =3

Yr 4 = 4Jonas


MAPRET – Progress Monitoring

Jhye

Victoria Jonas

Kiann

Luke


MAPRET Progress Over Time


What is expected?

NTCF Area of Learning: Year Level:



ESL Phases and Levels

Phase 1 Phase 2 Phase 3

BL 1

BL 2

BL 3

Level 1

Level 2

Level 3

Level 4

Level 5

Level 6

Level 7

Level 8


School A: Year 4Setting Year 4 Level targets for improvement in writing

KGP 3 Band 1 Band 2 Band3

E S C E S C E S C E

1 1 8 1 2 2

BL 1 BL 2 BL 3 Level 1 Level 2 Level 3 Level 4 Level 5

2 2 7 14 5 0 2


Which students are at, above and below expected level?

Above School

Expected Level

At School

Expected Level

Below School Expected

School Level

Significantly Below School

Expected Level


SMART Goals

Specific

Measurable

Agreed

Realistic

Time-framed


NT Teaching and Learning Framework





•Classroom Tasks










Connecting and UsingKnowledge Meaningfully

•Complex reasoning processes•Open ended

learning opportunities•Connecting to world beyond

classroom


NT Teaching and Learning FrameworkFor teachers: How best do learners learn?





•Classroom Tasks










Connecting and UsingKnowledge Meaningfully

•Complex reasoning processes for assessment tasks

•Open ended •Connecting knowledges and

• to world beyond classroom


Engaging with and acquiring knowledge

Knowing WhatDeclarative Knowledge

Knowing HowProcedural Knowledge

Facts, Vocabulary > Episodes> Time Sequences > Cause and Effect Relationships >

Generalisations / Principles > Concepts

Skills> Complex Reasoning Processes

Thinking and learning habits


Dimension 2: Acquire and Integrate KnowledgeKnowing What (Declarative Knowledge Patterns)

Event Event Event Event

Topic

Fact

Fact Vocab

Fact

Effect

Episode

person person person

effectcause

Time

Place

Duration

Event Event Event Event

Cause


Knowing What (Declarative Knowledge Patterns)

Generalisation / Principle

Example

Example

Example Concept

Characteristic

Example

Example

Example

Characteristic

Example

Example

Example

Characteristic

Example

Example

Example


Declarative Knowledge

Content and

Information

Construct

Meaning

Organise

Store

Procedural Knowledge

Skills and Processes

Construct Model

Shape

Internalise

Dimension 2: Acquiring and Integrating Declarative Knowledge


Identify patterns in information

Consciously store information in memory

Link new information to prior knowledge.

Construct Meaning

Organise

Store

Dimension 2: Acquiring and Integrating Procedural Knowledge


Procedural Knowledge

Shape

Construct Model

Internalise

Steps

Feedback

Practice

1. Select items to compare. 2. Select characteristics to

base your comparison 3. Explain how the items are

similar and different.

Timely

Specific

Based on Criteria

Massed

Distributive

Focused

Example: Model for Comparison


Eight Complex Reasoning Processes

Comparing

Abstracting

Classifying

Inductive Reasoning

Constructing Support

Deductive Reasoning

Analysing Perspectives

Analysing Errors

Deepening and practicing knowledge


Connecting and using knowledge meaningfully

• Decision Making: Generating and applying criteria to select from among seemingly equal alternates

• Problem Solving: Overcoming constraints or limiting conditions that are in the way of pursuing goals

• Creating: Developing unique products or processes that fulfill perceived needs

• Experimental Inquiry: Generating and testing explanation of observed phenomena

• Investigation: Identifying and resolving issues about which there are confusions or contradictions

• Systems Analysis: Analyzing the parts of a system and the manner in which they interact.


Phase I: Develop a model or language of gathering and using data (assessment) and instruction.

Ensuring Effective Assessment and Instructional Practices (Marzano, 2007)

Phase IV: Monitor the effectiveness of individual teacher’s instructional and assessment practices as a form of teacher feedback.

Phase III: Establish a systematic way for teachers to observe expert teachers and each other using the model of assessment and instruction.

Phase II: Develop a systematic way for teachers to interact about assessment and instruction using the model.

Readiness: Systematically explore and examine assessment instructional strategies.

Data Driven Design for Learning

Documents

Transcript of Data Driven Design for Learning