Understanding and Administering the School-Wide

Evaluation Tool (SET)

Agenda for today• Understanding of the SET

and its purpose• Process for scheduling and

completing• Administering the 4 parts of

the SET• Sign up for dates

In your packet…

• Slides• Supporting Handouts• Blank SET Documents• Activities

Please make note of time limits and watch your clocks!

• Trainer will raise his/her hand.

• Finish your thought/comment.

• Participants will raise a hand and wait quietly.

Attention Signal

Be Responsible• Return promptly from breaks

• Be an active participant

• Use laptop computers appropriately

Be Respectful• Maintain cell phone etiquette

• Listen attentively to others

• Limit sidebars and stay on topic

Be Kind• Enter discussions with an open mind

• Respond appropriately to others’ ideas

Participant Expectations

Understanding and Planning for the School-Wide Evaluation Tool

(SET)

Understanding and Planning: What is the SET?

• The SET is an empirically validated tool used to assess level of PBS implementation.

• It was developed at the University of Oregon for the PBIS National Technical Assistance Center as a research tool.

• SET scores are the national standard used to measure PBS implementation (universals).

Understanding and Planning: Why Administer the SET?

• Provides objective information to schools about effectiveness of implementation

• Gives teams specific areas to focus on for improvement

• Establishes a baseline to compare implementation over time

• Allows WCPSS PBS team to analyze trends in implementation in order to modify strategies of support

• Helps WCPSS to understand discrepancies in school outcomes

Understanding and Planning: How is the SET Administered?• There are four components of

the SET– Administrator interview

– Staff and student interviews

– Observations in all settings

– Document review and scoring

• Takes about 2 hours to complete the SET

• Usually completed between 9:30 am and 11:30 am.

Understanding and Planning: Who administers the SET?

• SET is completed in pairs.

• Primary interviewer does the majority of the interviewing and is responsible for completing all required tasks.

• Secondary interviewer’s role is to check for agreement on scoring, and to discuss any discrepancies with the primary interviewer.

• Secondary interviewers can ask follow up questions if they need more information to document the answer.

Understanding and Planning: Scheduling Process

The SET Scheduling Coordinator will:

• Create and maintain master schedule

• Notify all schools of the assigned date and obtain confirmation

• Re-confirm close to the SET date

• Handle all re-schedules

• Troubleshoot any day-of challenges

• Ensure that all materials are returned to the assigned PBS coach

Understanding and Planning: Scheduling Process

• Master schedule is on google docs, on laptops.

• Sign up for dates and choose primary or secondary interviewer.

• In order to meet our goal, 4 SETS must be completed every day between March 22rd and April 20th.

Understanding and Planning: Process for Completing the SET• Once you have dates, Javier will be

your main contact.

• Day of SET:– Show up – Complete the Administrator Interview first.

– Complete the staff and student interviews while touring and doing observations.

– Review documents and score on site.

• After SET– Write a brief plus/delta of observations

you would like to share with the school.

– Return all documents and notes to Javier.

– Accept our gratitude!

Administering the SET

Administrative InterviewStudent and Staff Interviews

ObservationsDocument Review and Scoring

Administering the SET: Administrator Interview

• The administrator interview must be completed prior to staff and student interviews.

• Get clear answers to each question as staff interviews are scored based on agreement with administrator answers.

• Interview typically takes 20-30 minutes.

Administering the SET: Administrator Interview Tips

• At the beginning of each section, it is helpful to orient the administrator to the types of questions coming.

• It is ok to ask follow up questions if you need more information or to re-word if they don’t understand what you are referring to.

• If the administrator answers no to certain questions you will be prompted to skip ahead.

Administering the SET: Administrator Interview Tips

• Question #5: Find out how staff would refer to the procedure

• Question #6: If unclear, reference the acronym

• Question #11: Focus on the reinforcers that are most frequent, and find out how staff and students refer to them

• Question #12: If yes, follow-up with when and how procedures were reviewed with staff.

• Question #20: Key word in this question is goals. Improving or sustaining behavioral practices must be part of a goal

• Question #21: If yes, follow-up with how

Activity 1: SET Matching Exercise

• On your own, complete the SET matching exercise sheet by matching the interview questions with the evaluation questions.

• Compare answers with a partner.

• Refer to answer key if needed.

Administering the SET: Student and Staff Interviews

• Prior to beginning additional interviews and observations, fill in the blanks on the top of the interview and observation form based on administrator answers.

• Write the administrator’s answers to questions 4, 5 and 17 on the interview and observation form in the corresponding column.

• If you skipped questions on the administrator interview, you need to agree on which additional interview questions you can complete.

Administering the SET: Student and Staff Interviews

• Move throughout the school to conduct interviews of staff members and students.

• Strive to obtain a representative sample of students across grade level, sex, and race.

• Interview staff members with various roles in the school.

• Four of the staff interviews need to be PBS team members so that they can answer additional questions.

Administering the SET: Student and Staff Interviews• When you approach staff, you may have to remind

them of your purpose and ask if they have time to answer a few questions

• Staff Interviews:

– 7 questions with 3 additional for PBS team members

– Interview one person at a time

– Try to locate staff in various grade levels

– Attempt to minimize interruption to direct instruction or supervision

– Record number of expectations known(3/4, 1/3, 5/5, etc.)

– Circle Y or N for yes/no questions

– Put “+” when staff response matches administrator and a “-” when it does not.

Administering the SET: Student and Staff Interviews• Tell the student that you are conducting a

survey and would like to ask them 2 questions. Move on if the student is hesitant.

• Student Interviews:– Interview one person at a time. If students are

standing in a group, ask only one student to answer or ask the student to move over to the side with you.

– Attempt to minimize interruption to direct instruction or supervision

– Record number of expectations known(3/4, 1/3, 5/5, etc.)

– Circle Y or N for yes/no question

Activity 2: Role Play

• Form groups of two or three.• Practice approaching both

“staff” and “students” in role play.

• Note any issues or trouble spots.

Administering the SET: Observations

While interviewing staff and students, complete the required observations in 10 areas.

• Look for posted rules and expectations that are specific to the area you are in

• Ask for documentation of the crisis procedures in the required areas (notebook, red bag containing procedures, cards in nametag, etc.)

Administering the SET: Document Review and Scoring

• After completing the observations and student and staff interviews take all notes and documents to a space on-site.

• Complete the scoring guide with the secondary interviewer.

• Review documents as needed

• For items requiring documents to score there will be a P in the “data source” column.

Administering the SET: Document Review and Scoring

• The role of documents is to support sustainable implementation of PBS.

• Documents should contain details clear enough for future training of staff and students.

• There are many different ways to document procedures that meet the criteria.

Administering the SET: Scoring Guidelines• Most items are easy to score and

have clear criteria for 0,1,2.• Some items may require

discussion between the primary and secondary interviewer to determine score.

• Many items require multiple data sources to determine score.

• Use the “data source” column on the scoring guide to assist you.

Administering the SET: Scoring Guidelines

Common Scoring Challenges• SIP• Action Plans• Budget

When in doubt leave an item blank, make copies of necessary documents and send to Javier for scoring

Activity 3: Case Study

• On your own, review all the materials from the provided case study and complete the scoring guide.

• Compare answers with a partner and discuss any discrepancies.