Iowa Department of EducationIowa Department of EducationTraining WorkshopTraining Workshop

March 30, 2012March 30, 2012

Anita Young, Ph.D.Assistant Professor

&Carol Kaffenberger, Ph.D.

Faculty Associate

Johns Hopkins University

Making DATADATA Work

Have Fun!Have Fun!

Objectives!!1. Discuss the importance and impact of

working systemically2. Practice and apply a four step process

to identify educational issues, collect data, analyze data, and share results

3. Identify a SMART goal that promotes college and career readiness

4. Develop a plan of action

ASCA National ModelASCA National Model

FOUNDATION

Beliefs & PhilosophyMission Statement

Domains: Academic, Career, Personal/Social

ASCA National Standards & Competencies

All students can achieve

Counseling as a foundation

School Counseling Dept Mission Statements

Crosswalking & Connecting to Instruction

Counselors as Leaders

Counselors as Advocates

Counselors as system thinkers

MSCA 2010

ACCOUNTABILITY

Results ReportsSchool Counselor Performance Evaluation

The Program Audit

Results Data

Closing the Gap

Systemic Impact

Program Effectiveness

MSCA 2010

Accountability: What are the purposes for using data?

Monitoring student progress & closing the achievement gap

Program assessment & evaluationDemonstrating counseling program

effectiveness

HART RESEARCH A S S O C I A T E S

Key findings from a national telephone/online survey among 1,507 members of the high school graduating class of 2010

conducted July 29 – August 3, 2011 for

One Year Out

Refer to Slideshow Refer to Slideshow

Your Thoughts???? Your Thoughts????

What does a college and career readiness path mean to you?

Reviewing Data Elements

◦School Report Card◦Brainstorming Worksheet◦Quadrant Worksheet

Examine Manager Middle School’s Data

What is working well at this school?What concerns you about this data? Does an achievement gap exist? Where?What additional information do you need?What should you focus on? What data

should you collect?

What are the school’s strengths?What are you worried about?What else do I need to know to

understand the meaning of this data?How is this data connected to school

counseling programs and services?

Academic Personal/Social Career

1. 1. 1.

2. 2. 2.

3. 3. 3.

4. 4. 4.

Brainstorming ActivityBrainstorming Activity

Quadrant 1:What or who are you worried about in your school? What are the barriers to success for your students? Quadrant 2:What strategies current address the issues? What are you doing

to address these barriers?Quadrant 3: What evidence/data support that you are making a difference in the lives of those studentsQuadrant 4:What do you need to do differently?  

Quadrant ActivityQuadrant Activity

Using DATA to Understand Using DATA to Understand Educational IssuesEducational Issues

1. DESIGN: What is the question?2. ASK: How will you answer the question?3. TRACK: How will you make sense of the

data?4. ANNOUNCE: How will you use the

findings?

Brief Overview

Goal Considerations Goal Considerations

SSpecific

MMeasurable

AAttainable

RRealistic

TTime bound

What do you want to know or understand?What is to be evaluated and why?Does your question align with the school’s mission statement?

State as a question. Examples:•Is the mentor program improving student achievement?•Is the after school homework club increasing attendance?

2.ASK: How will you answer 2.ASK: How will you answer the question?the question?

What information or data will be needed to answer the question?

Does the data or information already exist?

What procedures will you follow?Do data collection instruments need to be

created?What steps do you need to consider

before collecting data?What is your timeline?

Procedures to FollowProcedures to Follow

Timeline Permission (formal and informal)

◦Buy-in from stakeholders◦Written permission

Data collection◦Quantitative (pre-post tests, questionnaires)◦Qualitative (focus groups, interviews, open-

ended questions, observational data)

Demographic DataDemographic DataWhat do you want to know?What do you want to know?

Student demographics: what are the characteristics of our students?

GenderEthnicitySocio-economic status (free/reduced

lunch)Limited English ProficiencyFamily configurationMobility

Achievement DataAchievement Data

Achievement: What does achievement look like at different levels and with different groups of students?

Overall Achievement◦ Grade point average◦ Standardized test scores, SAT, ACT, State tests◦ Passing all subjects

Periodic assessment◦ Semester grades◦ End of course tests

Ongoing classroom assessment◦ Class assignment grades◦ Tests

Suspension rates◦ Impact on achievement

Process DataProcess DataWhat do you want to know?What do you want to know?

“What you did for whom”Evidence that event occurredHow activity was conductedDid the program follow the prescribed

practice?

Perception DataPerception DataWhWhat do you want to know?at do you want to know?What others think, know or demonstrate data.

Measures what students are perceived to have gained in knowledge◦ 89% of students demonstrate knowledge of promotion/

retention criteria◦ 92% can identify early warning signs of violence

Measures competency achieved, knowledge gained or attitudes beliefs of students ◦ Pre-post surveys◦ Every student in grades 9-12 completed a 4 year plan ◦ Every 5th grade student completed an interest inventory◦ 74%of students believe fighting is wrong◦ 29% of students feel safe at school

Results Data - ULTIMATE GOALResults Data - ULTIMATE GOALWhat do you want to know?What do you want to know?

“So WHAT” dataHard dataApplication dataProof your program has (or has not)

positively impacted students ability to utilize the knowledge, attitudes and skills to effect behavior◦Attendance◦Behavior◦Academic achievement

Tips for Creating Tips for Creating QuestionnairesQuestionnaires

Use a simple one-page format with fewest possible relevant questions

Develop questionnaires that are age appropriate

Use parallel language for all questionsConsider using one open-ended questionAdminister pre-tests to assess knowledge

and post-tests to evaluate learningTest your questionnaires with others

SurveysSurveys

About questions• “Is this question worth asking?”• Keep items short as possible• Avoid double-barreled wording

Rate the quality and value of the capstone course

• Focus response choices “Rate the quality of …” Use P-F-G-VG-EX rather than

“The capstone was excellent.” SD-D-A-SA

Activity: Designing a Activity: Designing a QuestionnaireQuestionnaire

Design a 5 item questionnaireAlign items with the ‘burning question’Set up the response scale

Using Technology for DATAUsing Technology for DATA

Don’t reinvent the wheelUsing Data warehouses, student

information systems, Naviance, school reports to gather data

FREE Survey Monkey Tool Create unlimited number of surveys

Allows up to 10 questions per survey  Choose from 15 available question types

Supports any language

MSCA 2010

Focus Groups & Interview TipsFocus Groups & Interview Tips

Focus Groups:◦Select 2-12 stakeholders; co facilitate session;

record or take notes; establish procedures ◦Use an interview guide; ask follow-up questions

to clarifyInterviews:

◦Select key stakeholders to interview; establish procedures

◦Use an interview guide

33..TRACK: How will you make TRACK: How will you make sense of the data?sense of the data?

What can you learn from the data?How will you collate or disaggregate the

data to make it useful?How can you organize the data so that you

can answer your questions and others can understand it?

How will you present your data?Would charts be useful?

Page

Basic Ways to Analyze DataBasic Ways to Analyze Data

Use simple statistics - averages and percentages

Disaggregate – take apart by meaningful wholes

Aggregate – condense statistics to meaningful representative numbers

Cross-tabulate – put data into a chartLongitudinal data – look at data over time

Using Charts: Bullying DataUsing Charts: Bullying Data

Pre Test Post Test

I know what do when I am bullied

1.9 3.4

I would tell a teacher if I was bullied

1.4 2.9

I can resist a bully 2.0 3.8

I believe an adult can help me

1.7 3.9

Using Charts: Bullying DataUsing Charts: Bullying Data

Track: Student Reasons for Track: Student Reasons for Being AbsentBeing Absent

Are Programs Working?Are Programs Working?

Reasons for Student of the Month Selection

Who was Selected for Student of the Month

MSCA 2010

Evaluating the Effectiveness of a Evaluating the Effectiveness of a ProgramProgram

MSCA 2010

4.ANNOUNCE: How will you 4.ANNOUNCE: How will you use your findings?use your findings?

So what do these results mean?What are the recommendations?How will you use your findings?How will you present your findings and

recommendations?Who will you share them with?What are the implications?

Sharing with StakeholdersSharing with Stakeholders

Who are your stakeholders?◦ Principal◦ Parents◦ Superintendent◦ School Board

What do you want them to know?◦ What you have done◦ What others know◦ How this makes a difference

How to communicate the information?◦ Charts, tables, and stories◦ Newsletters

Using the FindingsUsing the FindingsShare and explain your results to stake

holders (administrators, faculty, parents, community).

Be sure to include how the entire school can implement change; not just the counseling department.

DATA is one format but there are other formats to consider (GRIP, MEASURE, SOARING)