Saginaw KEYS Data Analysis Training for

Continuous School Improvement

March 20 and 21, 2006

Jacques Nacson jnacson@nea.org

Gary Obermeyer obee@learningoptions.net

Workshop Agenda

• Welcome and Introductions• Review of Agenda and Objectives• Your Expectations and KEYS Experiences • KEYS and Continuous School

Improvement: The Action Research Cycle• KEYS Data Analysis and Interpretations

--Indicators and Predictions--General Data-Related Considerations--Demo Report: Charts and Data Points--Tools and Data Analysis Worksheets--Navigating the Online Data Report and Online Resources

• The NEA KEYS 2.0 Toolkit and Use of the Action and Facilitation Guides

Workshop Objectives

• To examine KEYS data within the larger context of continuous school improvement processes

• To learn how to analyze and interpret KEYS data and how results can be shared with school staff and other stakeholders

• To consider strategies for how KEYS data might be used as a stimulus to drive your targeted school improvement efforts

• To learn how to use KEYS support materials, including online resources, to help you take next steps beyond preliminary analysis of KEYS data

Problem orOpportunity

Identified

Data Collection

Diagnosis & Refinement of

ProblemAction Planning

Action TakingImplementation&

Evaluation

Model for Understanding and Interpreting KEYS 2.0 Data

School completesKEYS 2.0 survey

School receives scores on KEYS 42 indicators of school

quality

Use other data, including student achievement to

confirm, validate or justify

Make inferences about our school’s quality based on

the KEYS results and assessments of other data

School makes decisions about next steps and plans for

improvement

Make judgments based on theories or

hypotheses (if, then) to explain “why”

Implementation of planned intervention

Process & product evaluation including effects on student

achievement

NEA 6 Keysto

School Quality

General Questions and Principles to Consider

Before Looking at Your KEYS 2.0 Data

Recognizing the limits of the bestsurvey instruments

Distinguishing between symptomsand root causes

Are the data surprising orconfirming?

Are you listening to all relevantvoices in your school community?

Moving from data to understandingto knowledge

Without theory, no organizationallearning and improvement canoccur

General Questions and Principles to Consider

Before Looking at Your KEYS 2.0 Data(continued)

Applying systems thinking: thewhole is greater than the sum of itsparts

Looking for patterns within andacross Keys

Slow and steady: the road tocontinuous improvement has noend (PDSA)

Celebrating your victories andlearning from your strengths

KEYS 2.0 – A diagnostic toolleading to a deliberative process,not a rigid, mechanistic, quick-fixprescription

Flow Chart of Suggested Activities/Processes For Analyzing and Interpreting Your Data

School Community Completes KEYS Survey

School Improvement Plan Implemented and Evaluated

Created a School Improvement Team (SIT)

SIT trained in action research, use of quality tools and analysis and interpretation of KEYS Results

SIT conducted preliminary analysis and interpretation of KEYS results and prepared for involvement of whole school community

SIT assisted whole school community to examine results and make preliminary recommendations

SIT considered whole school’s recommendations and begins deliberative process

SIT developed preliminary school improvement plan based on KEYS data and other data and present to school community for feedback

School improvement plan approved and ready for implementation

?No

Yes

No

No

Yes

Yes

?

?

?

?

?

Yes

Yes

No

No

No ?

Yes

Yes

No

Understanding the Graphs for Each Key

Horizontal Axis Measure of quality: 5 point scale Left side Disagree (low value) Right side Agree (high value)

Vertical Axis Indicators (groups of questions that measure the same concept)

KEY 1. Shared Understanding and Commitment To High Goals

Respondents Provide Direct Instruction to Students

Understanding the Graphs for Each Key

School Average All Schools Average

90th Percentile Score• Data Points

• School average (red)• All schools average (38 pilot schools) (Purple)• 90th percentile score of the pilot sample (Yellow)• Length of the horizontal bar (Purple) 1 standard deviation above and below the school average (measure of agreement) The Goals for your school in terms of continuous improvement for each indicator• The school average moving continuously toward the right side (agree – high value of quality for that indicator)• At the same time, reduce the standard deviation (narrow the length of the horizontal bar, meaning greater agreement among respondents)

Standard Deviation

District Reports

5.4 Safe Environment

Worksheet for Analyzing KEYS 2.0 Results (Analysis of Individual Indicators)

Preliminary AnalysisNOTE: THIS WORKSHEET HAS BEEN REDESIGNED

Measures of Quality Degree of Consensus

Low(Wide

>1.3 pt.)

High(Narrow

<1 .3pt.)

VeryHigh

(between 4.1 & 5.0)

High(between3.4 & 4.0)

Low (between2.1 & 2.6)

Very Low

(average scorebetween 1 & 2)

Indicators

Key:_________________ Constituency Group:_________________ Date of Survey:______________________

Worksheet for Analyzing KEYS 2.0 Results (Analysis of Cross Key Indicators)

(Secondary Analysis)

Related Indicators Indicators

Key:_____________________ Constituency Group:____________________

Date of Survey:________________________

score score score score score score score

score score score score score score

score score score score score score

score

score

score

score

scorescore

score

score

scorescorescore

scorescore

score

scorescore

• Action Guide

• Facilitation guide

Tool Box that contains a CD with all materials

Steps a School Might Take Once KEYS Preliminary Analyses are Completed

1. GAPS: Decide on one indicator or a group of indicators where gap (s) exist.

2. RELEVANCE: Reflect with the “team” on the relevance, importance and priority of the selection.

3. DATA COLLECTION-VALIDATION: Consider the need to collect additional data to validate the KEYS findings.

4. DATA COLLECTION-DIAGNOSIS AND REFINEMENT: Examine what is the “root” cause of the problem.

Steps a School Might Take Once KEYS Preliminary Analyses are Completed

5. THEORY OF ACTION: Identify and select the most appropriate solution for your context.

6. ACTION PLANNING: Set achievable goals and develop specific action/project plans.

7. IMPLEMENTATION: Action plans must be implemented for improvement to occur.

8. DATA COLLECTION-EVALUATION: Both process and product evaluations are necessary for learning to happen.

9. BACK TO STEP 1

Objectives for Advanced Training KEYS Data Analysis

• Understand the Value of Data for Making Good Decisions

• Distinguish Between Symptoms and Root Causes for Continuous Improvement

• Learn How to Analyze and Use Data through the Continuous Improvement Cycle (PDCA)

• Use Quality Tools to Improve Process Quality and Student Outcomes

• Understand the Concept of Variation and Use Control Charts to reduce variation and improve outcomes

Agenda for Advanced Training KEYS Data

analysis• Data and General Data Principles• W. Edward Deming – Data and Variation• Continuous Improvement Cycle (PDCA)• Quality Tools

Statistical Process Control-Quality Control ChartsGetting to the Root CauseCollecting, Organizing and Analyzing your DataSelecting the Right Intervention Based on Theory and Hypothesis TestingPlanning and Carrying Out the InterventionEvaluating the success of the Intervention