TNTP 2013 Student Survey Proctor Orientation TNTP Academy and the Assessment of Classroom...

© TNTP 2013

Student Survey Proctor OrientationTNTP Academy and the Assessment of Classroom Effectiveness

January 2016

/ 2

Contents

Why We Care About Effective Teaching

Understanding ACE

Adding It Up: ACE Scoring

Student Survey Parameters

Scheduling Survey Sessions

Survey Administration

Accommodations

Troubleshooting

Compensation

Closing

/ 3

Through ACE, we strive to create the fullest possible picture of teacher performance using multiple measures.

• Research shows classroom observations are valid predictors of teacher effectiveness.

• You will be observed at least 4 times, with your performance rated according to the ACE Instructional Framework.

Classroom Observati

ons

• Principals will rate your performance relative to other new teachers they have worked with throughout their careers.

Principal Ratings

• The survey uses research-tested questions predictive of teacher effectiveness.

• If you work with very young students (grades K-2) or with students in moderate-to-severe special education settings, you will most likely not administer student surveys.

Student Surveys

• The best measure of your effectiveness is the academic growth of your students.

• Where available, we will incorporate data from state-produced TVAAS score reports to analyze your actual impact on student learning over the course of the school year.

Student Achievement Data

/ 4

Measure 3: Student Surveys

WHY?• Research from the

Gates Foundation’s Measures of Effective Teaching (MET) study demonstrates that student surveys are a reliable indicator of teacher effectiveness.

• Our student survey uses questions shown to be predictive of teacher impact on student learning.

HOW?• Students in grades 3-

12 will take paper-based surveys. For all teachers, a proctor will administer surveys.

• Teachers working with very young students (K-2) or in moderate-severe settings will not administer the survey.

• Surveys and consent forms will be available in Spanish and English, with potential translation into other languages.

• All surveys will be anonymous and must be kept completely confidential.

WHEN?• Teachers must

collect parent consent forms before students complete surveys.

• Student surveys will be administered between January 25 and March 18.

• Sites will be administering on a staggered schedule – Amy will communicate with proctors directly regarding the survey window for your site.

“The average student knows effective teaching when he or she sees it.”(“Learning about Teaching: Initial Findings from the Measures of Effective Teaching“Gates Foundation, December 2010)

/ 5

How the student survey process works

Students complete an anonymous, paper-based

survey. The survey administration takes

approximately one class period and will be

administered by proctors to ensure confidentiality.

1. ADMINISTER

SURVEY Survey questions are grouped into three broad categories: Care, Control,

and Challenge, with responses rated on a 5-

point scale.

Student responses will be averaged to generate a

composite score in each of these categories, and those three category scores will then be

averaged to generate the overall survey score.

2. GENERATE SCORE

Survey Score

Teachers receive formative score reports detailing their students’ perceptions. This

data provides valuable insights into strengths and weaknesses as seen by the students in our teachers’

classrooms every day.

3. IMPROVE PERFORMANCE

Student Survey

1 – 5

/ 6

Calculating the survey score.

Teacher’s overall survey scores map to the performance levels below.

Skillful Proficient Developing Minimally Effective Ineffective4.50 – 5.00 4.00 – 4.49 3.50 – 3.99 3.00 – 3.49 2.99 or lower

STUDENT COMPOSITE SCORES BY CATEGORY

Care Control ChallengeHarry Potter 4 4 2.75Hermione Granger 3.75 4 2.5Ronald Weasley 4.25 3.5 3Cho Chang 4 3.75 2.75Neville Longbottom 4.5 4.25 3.25Lavender Brown 3.75 3.75 2.75Padma Patil 4.25 3.25 2.75AVERAGE CATEGORY SCORES 4.07 3.79 2.82

TEACHER’S OVERALL SURVEY SCORE: 3.56

/ 7

Past participants valued student survey feedback, indicating that it would help them improve their teaching.

• “I was glad to see my students recognized the value of the homework I assigned, that they believe that I want them to explain their thinking, and that they knew that I believed that they could get a good grade if they tried.“

CHALLENGE

• “I deeply cared for my students, but I know now that I didn't show them or tell them enough. This is definitely something I will focus on next year.“

CARE

• “[After reviewing survey results,] I am going to give the kids more input into the behavior system in my classroom, in an effort to give them ownership of their own actions as individuals and as a class.”

CONTROL

/ 8

Contents


Understanding ACE





Accommodations

Troubleshooting

Compensation

Closing

/ 9

Your performance on each ACE measure will be mapped to the performance scale below, earning you between 1 and 5 points for each ACE component.

Each component will be weighted and combined to calculate your final ACE rating.

ClassroomObservations

Principal Ratings

Student Surveys

Student Achievement

Ineffective 1.00 – 1.99 points

Much worse than

1 point

2.99 or lower1 point

Ineffective1 point

Minimally Effective

2.00 – 2.79 points

Worse than2 points

3.00 – 3.492 points

Minimally Effective2 points

Developing 2.80 – 3.59 points

Same as3 points

3.50 – 3.993 points

Developing3 points

Proficient 3.60 – 4.29 points

Better than4 points

4.00 – 4.494 points

Proficient4 points

Skillful 4.30 – 5.00 points

Much better than

5 points

4.50 – 5.005 points

Skillful5 points

/ 10

Adding it up: Four Scoring Scenarios

Scenario 1: Classroom Observations + Principal Ratings

Observations

(60%)

PrincipalRating(40%)

0.60 * 3 points = 1.80 points (Observations)+ 0.40 * 4 points = 1.60 points (Principal Rating)

3.4 points ( Final ACE Score)

Scenario 2: Classroom Observations + Principal Ratings + Student Surveys

Student

Surveys

(30%)

Observations

(40%) PrincipalRating(30%)

0.40 * 3 points = 1.20 points (Classroom Observations)+ 0.30 * 3 points = 0.90 points (Principal Rating)+ 0.30 * 2 points = 0.60 points (Student Surveys)


/ 11

Adding it up: Four Scoring Scenarios, cont’d

0.25 * 3.15 points = 0.79 points (Observations)+ 0.20 * 3 points = 0.60 points (Principal Rating)+ 0.20 * 2 points = 0.40 points (Student Surveys)+ 0.35 * 4 points = 1.45 points (Achievement)


StudentAchievem

ent(35%) Principal

Rating(20%)

Observations

(25%)

Student

Surveys

(20%)

Scenario 3: Classroom Observations + Principal Ratings + Student

Surveys + Student Achievement Data

Scenario 4: Classroom Observations + Principal Ratings + Student

Achievement Data

Observations

(35%)

StudentAchievem

ent(35%)

Principal Rating(30%)

0.35 * 3.15 points = 1.10 points (Observations) + 0.35 * 4 points = 1.40 points (Student Achievement) + 0.30 * 3 points = 0.90 points (Principal Rating)


/ 12

There are 3 possible outcomes at the end of the first year of teaching.

PASS2.75 – 5.00 points

EXTENSION2.50 – 2.74 points

REMOVAL1.00 – 2.49 points

• If a teacher passes ACE and complete all program and state requirements, her or she will be recommended for their state certification.

• If a teacher falls short of passing, but is on the right trajectory to meet that standard with additional progress, he or she may be granted an extension year to continue building skills.

• If a teacher struggles and shows limited prospect of improvement, her or she will not advance to another year of teaching.

• While we anticipate this outcome to be relatively rare, it is necessary. Allowing a chronically struggling teacher to remain in the classroom isn’t good for anyone.

/ 13

Contents


Understanding ACE





Accommodations

Troubleshooting

Compensation

Closing

/ 14

Not all teachers will be eligible for student surveys.

• A valid survey instrument has not yet been developed for this age group.

Pre-K through 2nd

grade

• A valid survey instrument has not yet been developed for this population.

Moderate to severe SPED

• 15 is the minimum number of students our analysts required to produce reliable scores on student surveys for ACE. Remember that we must get 15 students in total – not just in one class period/course section!

Teachers with fewer than 15

students*If you find that you’ve been assigned a teacher or course sections fitting

any of these categories, please let us know immediately!

/ 15

Teacher will administer surveys in up to two course sections.

• If teachers work in a self-contained setting, they will administer just to that class (one “course section”). If they teach in a departmentalized setting, they’ll administer to two course sections.

Elementary (3-5)

• If teachers work in a secondary setting (typically teaching 1-2 subjects), they’ll administer surveys to two course sections/class periods of their choosing.

Secondary (6-12)

• Some teachers completed the Class Period Selection Tool incorrectly and only selected one – so we only printed surveys for one class period.

Exceptions

*The number of class periods for which each of your teachers will administer surveys will be clearly marked on your caseload!

/ 16

What does the survey actually look like?

• The surveys consists of 20-25 questions (more for secondary).

• Surveys are individually coded for each teacher.

• You’ll have plenty of extras! We won’t provide blanks because of the unique ID’s.

• Shred the leftovers – but not until you’ve sent everything back to the Brooklyn office!

/ 17

ContentsWhy We Care About Effective Teaching

Understanding ACE





Accommodations

Troubleshooting

Compensation

Closing

/ 18

Student Survey Windows

Site Window Opens (Start Surveying)

Window Closes By when should I ship all surveys back?

Arizona February 24 March 18 March 18 (preferably earlier)

Baltimore February 24 March 18 March 18 (preferably earlier)

Charlotte February 24 March 18 March 18 (preferably earlier)

DC February 24 March 18 March 18 (preferably earlier)

Fort Worth February 24 March 25 Return to site for shipping

Indianapolis February 24 March 18 March 18 (preferably earlier)

Louisiana January 25 March 18 March 18 (preferably earlier)

Nashville February 3 March 4 March 4 (preferably earlier)

/ 19

Overview of what teachers have done so far…

Teachers have already given us their course selections via the Class Period Selection Tool in Survey Monkey. These data were used to print and ship surveys for each teacher to your site.

Teachers have also included information about the number of foreign language and large print surveys they need, as well as whether students in their selected classes will require a read-aloud accommodation.

Teachers are now in the process of collecting permission forms from students in order to have them take the survey – this is hard work, and you’ll want to be careful not to visit a teacher if they haven’t collected enough forms yet!

Some teachers on your caseload have not given us any information about their course selections yet, so we’ve guesstimated how many surveys they’ll need. You should reach out to them directly for specific course meeting times.

/ 20

Scheduling Survey SessionsReceiving your caseload from the site

o You’ll access your caseload via a dynamic spreadsheet in Google Drive after receiving an introductory email from us.

o If you’re Arizona, Baltimore, Charlotte, DC, Indianapolis, Louisiana, Michigan, or Nashville, you’ll be reporting directly to me/Janet. I’ll be sharing your caseload and asking you to report in regularly via a Google doc.

Accessing teacher course selections and planning your scheduleo Once you’ve received your caseload via the Google doc, you’ll see that it catalogs

everything you need to know to schedule with your teachers - including the time each selected class period meets. Tips and tricks for scheduling include: Allowing time for rescheduling – try to front-load your window Emailing teachers without class period meeting times early and following up

strongly Using www.batchgeo.com to see your geographic layout Sending a confirmation email 1-2 days before your visit!

Confirming appointments with teacherso Once you’ve planned your schedule, add these dates to the Google doc.o Email teachers directly using the email templates provided on the wiki!

Tracking and reporting your scheduleo Please think of the Google doc as a living record – keep this up-to-date with

daily revisions to confirmed meeting times, reschedules, # of surveys collected, etc..

http://www.batchgeo.com/

/ 21

Receiving Surveys

You’ll receive your packet of surveys via Fed Ex to your home

address. Louisiana, Nashville, Indy, and Forth Worth teachers will

receive surveys no later than the end of next week. Arizona,

Baltimore, Charlotte, Michigan, and DC proctors will receive them closer to the start of their survey windows.

You’ll return your completed surveys to our Brooklyn office using the pre-paid mailer, which will also arrive at your doorstep before the

end of the survey window.

Receiving Surveys

Before Surveys Start

1-2 Weeks

/ 22

Contents


Understanding ACE





Accommodations

Troubleshooting

Compensation

Closing

/ 23

Survey administration will happen in three segments.

Scheduling Administration

Report and Return

/ 24

Survey Administration Workflow

1• Arrive at the school at least 15 minutes early. Carry your TNTP identification (if provided).

2• Check in at the front desk.

3• Proceed directly to the classroom.

4• Introduce yourself to the teacher and walk them through the survey process.

5• Introduce yourself to the class and distribute the surveys. Ensure that students keep the

survey upside down until you give them permission to start.

6• Read the survey script word for word before allowing students to turn over the survey and

begin answering the questions.

7• Have students turn the survey over when they’re done.

8• Collect the surveys. Thank the teacher on your way out.

9• Update the Google doc with the total number of surveys collected that evening.

/ 25

Consent Forms

Proctor administers

surveys ONLY to approved

students

Teacher notifies Proctor

of which students are

eligible

/ 26

Survey Script

Please pay close attention while I share important information about the survey that you are about to take.

TNTP, a national nonprofit organization with more than 15 years of experience conducting education surveys across the country, wants to hear from you about your experiences in this class. The reason you are taking this survey is to learn what is working well and what needs improvement within your classroom.

There are no right or wrong answers. This is your chance to provide your honest feedback.

Your answers will be used to better understand your teacher’s strengths and weaknesses from students’ point-of-view. You may choose to skip any questions, but I hope that you will take this chance to share as much of your thoughts as possible.

Your responses are completely confidential. Your responses will be combined with responses from other students in your class before being shared with your teacher. Your teacher will not see how you or any other individual student in this class responded, so please feel safe to answer these questions honestly.

When you are completing the survey, please fill in the circles completely that represent your answer. I cannot answer any questions about individual items on the survey, but please pick the answer that you think best represents the way you feel about your experience in this classroom with your teacher. This is not a long survey, so please take your time in completing it so that you can share your honest and thoughtful responses.

Once you have your copy of the survey, you may begin. Please note that you are not being asked to share your name, so please do not write in your name on the form. Once you have finished, please hold onto your survey form until everyone in the class is finished. I will collect them after all students have completed the survey.

/ 27

Survey Script – Additional Language for Co-Teachers

When two ACE participants from the same classroom are surveyed:

First Administration:• This class has two teachers, so you will take the survey two times – one time for

each teacher. When you take the survey today, only think about insert teacher’s name. Tomorrow you will be able to fill out a survey just for about insert other teacher’s name.

Second Administration: • This class has two teachers, which is why you are filling out this survey again.

When you took the survey yesterday, you thought about insert teacher’s name. When you take the survey today, only think about insert other teacher’s name.

When only one co-teacher from the same classroom is surveyed:

• Even though this class has two teachers, this survey is only for insert teacher’s name. When you take the survey today, only think about insert teacher’s name.

*These scripts will be posted to the wiki for easy reference, along with all email templates, key dates, and other resources!

/ 28

Contents


Understanding ACE





Accommodations

Troubleshooting

Compensation

Closing

/ 29

AccommodationsYou may be required to provide accommodations for one or more students during

survey administration. Accommodations can take the following forms:

Any required accommodations will be noted in your schedule. If you are required to read aloud to a student, you should take him/her to a quiet corner of the room so as

not to disturb other students. If the majority of students require read-aloud accommodation, you may consider reading to the full class.

Large Print

Language Other Than

EnglishRead

Aloud/Scribe

/ 30

Professional Responsibility for Proctors

Surveys are a key component of our teachers’ summative

evaluation (ACE) and directly impact their ability to be

certified.

You are responsible for maintaining the integrity of these surveys, both before and after administration, until they are returned to

our Brooklyn office.

Please return all surveys at once - you will receive

multiple mailers, but send them all at the same time. Be sure to include a cover

sheet for each mailer.

Please schedule return pick-up of your surveys for the day after your last survey

session. Timeliness is essential!

High-Stakes Student Surveys

/ 31

Contents


Understanding ACE





Accommodations

Troubleshooting

Compensation

Closing

/ 32

Some Key DO NOTS:

DO NOT administer surveys to students without permission forms.

DO NOT mix survey types for one teacher (i.e., a teacher cannot give the elementary survey to a 5th grade section and a secondary survey to a 6th grade section – he/she must choose all secondary or all elementary sections).

DO NOT use blank copies of the survey or copy one of your existing surveys if a class section is short materials- reach out to Amy for extra surveys that are individually coded.

DO NOT allow the teacher to circulate or answer questions during survey administration!

/ 33

What If….

• If the substitute or co-teacher is willing to have you administer the survey, then proceed, If not, then reschedule.

The teacher isn’t teaching that day?

• You cannot help a student interpret a question – say “there’s no right or wrong answer, so please just answer the question as best you can.”

A student asks me for help on a question?

• If this is a self-contained elementary classroom, you’ll need to reschedule until the teacher has collected enough permission forms or enough students are present. If this is a secondary classroom, ensure that you’ll be able to survey 15 students total between the two classes.

The class has fewer than 15 students?

• We cannot survey students without a permission slip. These students should not leave the room, but they should not be given a survey. The teacher is responsible for indicating which students have permission and which do not.

Some students don’t have permission slips

on file?

/ 34

What If….(cont.)

• This should not happen – we print surveys for the total number on the roster, not the number of permission forms teachers report collecting!

I don’t have enough surveys for all students with a permission slip?

• Please try to let the school know from the road if you’ll be late. Only administer surveys if you’re confident students will have enough time to complete the survey thoughtfully and without rushing.

I’m running late and class has already

started?

• Teachers have confirmed dates that fit their schedule, so this should happen very rarely. If the teacher attempts to turn you away, insist on continuing or call Amy/Janet for back-up.

The teacher tries to turn me away?

• Students can answer as much or as little of the survey as they’d like, but encourage them to complete the survey fully.

Students opt not to answer all of the

questions?

/ 35

Contents


Understanding ACE





Accommodations

Troubleshooting

Compensation

Closing

/ 36

Billable Hours

Billable Non-Billable

• Hours in the classroom proctoring surveys

• Time spent in the Google doc, planning your schedule, and communicating with teachers

• Drive time IN BETWEEN schools

• Time spent planning for rescheduled survey sessions

• Drive time to your first school or home from the last school

• Lunch breaks

• Extended breaks between survey sessions due to inefficient scheduling or rescheduling

/ 37

Replicon

/ 38

Contents


Understanding ACE





Accommodations

Troubleshooting

Compensation

Closing

/ 39

Questions?

TNTP 2013 Student Survey Proctor Orientation TNTP Academy and the Assessment of Classroom...

Documents

Transcript of TNTP 2013 Student Survey Proctor Orientation TNTP Academy and the Assessment of Classroom...