INTER-RATER AGREEMENT IN KANSAS Summer Principals Academy July 22-24, 2014 Abilene, KS.
-
Upload
austen-cole -
Category
Documents
-
view
217 -
download
1
Transcript of INTER-RATER AGREEMENT IN KANSAS Summer Principals Academy July 22-24, 2014 Abilene, KS.
![Page 1: INTER-RATER AGREEMENT IN KANSAS Summer Principals Academy July 22-24, 2014 Abilene, KS.](https://reader030.fdocuments.net/reader030/viewer/2022032803/56649e2d5503460f94b1d20a/html5/thumbnails/1.jpg)
INTER-RATER AGREEMENT IN KANSASSummer Principals Academy
July 22-24, 2014
Abilene, KS
![Page 2: INTER-RATER AGREEMENT IN KANSAS Summer Principals Academy July 22-24, 2014 Abilene, KS.](https://reader030.fdocuments.net/reader030/viewer/2022032803/56649e2d5503460f94b1d20a/html5/thumbnails/2.jpg)
2
What is Inter-Rater Agreement?• Consistent
understanding• Consistent
agreement• Consistent
process• Consistent
credibility
The degree 2 raters using the same scale give the same rating in identical situations.
![Page 3: INTER-RATER AGREEMENT IN KANSAS Summer Principals Academy July 22-24, 2014 Abilene, KS.](https://reader030.fdocuments.net/reader030/viewer/2022032803/56649e2d5503460f94b1d20a/html5/thumbnails/3.jpg)
3
Where is Inter-rater Agreement Useful??
Inter-Rater Agreement is useful in all areas of administrative practice including:
• Kansas Educator Evaluations• Kansas Accreditation • Personnel Decisions• Family Engagement Programs• Budget Management
![Page 4: INTER-RATER AGREEMENT IN KANSAS Summer Principals Academy July 22-24, 2014 Abilene, KS.](https://reader030.fdocuments.net/reader030/viewer/2022032803/56649e2d5503460f94b1d20a/html5/thumbnails/4.jpg)
4
What are the Goals of Inter-rater Agreement?
• The Inter-Rater Agreement Statistical Goal is 75% - agreement with no rating more than one level apart.
• The Goal is to have an administrative Common Frame of Reference for your district such as:• Agreement with no rating more than one level apart. • Consistent understanding• Consistent agreement• Consistent process• Consistent credibility
![Page 5: INTER-RATER AGREEMENT IN KANSAS Summer Principals Academy July 22-24, 2014 Abilene, KS.](https://reader030.fdocuments.net/reader030/viewer/2022032803/56649e2d5503460f94b1d20a/html5/thumbnails/5.jpg)
5
How is Inter-rater Agreement Helpful??
• What is a Common Understanding?
• How does Inter-rater Agreement help administrators??
• Facilitates Quality• Provide Consistent Messaging• Impacts Increased Student Growth• Enhances Trust • Facilitates Quality Feedback• Legally Defensible • Assists Accurate Data Collection• Allows for Increase Cooperation
over Time • Passes Public Scrutiny
![Page 6: INTER-RATER AGREEMENT IN KANSAS Summer Principals Academy July 22-24, 2014 Abilene, KS.](https://reader030.fdocuments.net/reader030/viewer/2022032803/56649e2d5503460f94b1d20a/html5/thumbnails/6.jpg)
6
District-wide Agreement about External Criteria
• Agreed Upon External Criteria for:• Observations• Artifacts• Units of Study• Processes• Program Development• Time Limits • Walk-Throughs• Protocols
![Page 7: INTER-RATER AGREEMENT IN KANSAS Summer Principals Academy July 22-24, 2014 Abilene, KS.](https://reader030.fdocuments.net/reader030/viewer/2022032803/56649e2d5503460f94b1d20a/html5/thumbnails/7.jpg)
7
Pragmatics/ConsiderationsPotential Landmines
• Discuss with an Elbow Partner Possible Issues that may cause Rater Error• Leniency: Benefit of the
Doubt• Central Tendency: Safety in
the Middle• Comparing One with
Another: Stick to the Rubric• Relying on Gut too Much:
Stick to the Rubric• Evaluating Absent
Evidence: Stick to What is Observed
![Page 8: INTER-RATER AGREEMENT IN KANSAS Summer Principals Academy July 22-24, 2014 Abilene, KS.](https://reader030.fdocuments.net/reader030/viewer/2022032803/56649e2d5503460f94b1d20a/html5/thumbnails/8.jpg)
8
How to Train Administrators in using Inter-rater Agreement
• Inter-Rater Agreement Training – Things to Consider: • Synchronous and Asynchronous • Experience of the Participants• How to Ensure Fidelity • Use of Video Examples• Use of Case Studies• Bias Exists• Single versus Multiple Observations• Professional Judgment• Degree of Familiarity
![Page 9: INTER-RATER AGREEMENT IN KANSAS Summer Principals Academy July 22-24, 2014 Abilene, KS.](https://reader030.fdocuments.net/reader030/viewer/2022032803/56649e2d5503460f94b1d20a/html5/thumbnails/9.jpg)
9
Getting Started in YOUR District
• Getting Your Bearings• Developing a Guidebook• Developing a Training Protocol
![Page 10: INTER-RATER AGREEMENT IN KANSAS Summer Principals Academy July 22-24, 2014 Abilene, KS.](https://reader030.fdocuments.net/reader030/viewer/2022032803/56649e2d5503460f94b1d20a/html5/thumbnails/10.jpg)
10
I. Getting Your BearingsPilot with Administrative Team1. Pilot Approach: Start with Safe Example:
• Use a video from YouTube• Assess Specific Area of Teaching
• Student Engagement• Teacher Activity• Classroom Environment
2. Review current issue/goal/vision/needs
3. Review current practices/instrumentation• Rubrics• Protocol• Training
![Page 11: INTER-RATER AGREEMENT IN KANSAS Summer Principals Academy July 22-24, 2014 Abilene, KS.](https://reader030.fdocuments.net/reader030/viewer/2022032803/56649e2d5503460f94b1d20a/html5/thumbnails/11.jpg)
11
II. Developing a Guidebook• Paper and Electronic: Synchronous and Asynchronous• EXCELLENT Administrative Council Endeavor –
Ultimately to be used during Professional Learning Times with the Staffs
• Pieces and Parts• Terms/Definitions: Frame of Reference/Common Understanding
• “Demonstrates”• “Frequently”• “Usually”
![Page 12: INTER-RATER AGREEMENT IN KANSAS Summer Principals Academy July 22-24, 2014 Abilene, KS.](https://reader030.fdocuments.net/reader030/viewer/2022032803/56649e2d5503460f94b1d20a/html5/thumbnails/12.jpg)
12
II. Developing a Guidebook• Pieces and Parts (continued)
• Examples and Rubrics• Areas to Observe
• Establishes/maintains a culture of safety, respect and rapport
• Knows how to make content accessible to all learners
• Plans standards-based instruction and formative assessments
• Uses a variety of instructional strategies to meet needs
• Deepens understanding through critical thinking, problem-solving and student reflection
• Analyzes student performance and provides feedback
• Collaborates to support student learning
![Page 13: INTER-RATER AGREEMENT IN KANSAS Summer Principals Academy July 22-24, 2014 Abilene, KS.](https://reader030.fdocuments.net/reader030/viewer/2022032803/56649e2d5503460f94b1d20a/html5/thumbnails/13.jpg)
13
II. Developing a Guidebook• Pieces and Parts (continued)
• Exemplars: Video and Artifacts• Master-Coded Videos: Leads to high level of
agreement• Example Artifacts
• Classroom Rules• Lesson Plans• Parent Note Home
• Lesson Plan for Adults: Andragogy
![Page 14: INTER-RATER AGREEMENT IN KANSAS Summer Principals Academy July 22-24, 2014 Abilene, KS.](https://reader030.fdocuments.net/reader030/viewer/2022032803/56649e2d5503460f94b1d20a/html5/thumbnails/14.jpg)
14
III. Developing a Training Protocol• Step-by-Step Procedure
1. Initial Training: Goals/Terms• Practice Interpreting Rubrics• Differentiate between Different Performance Levels with Examples
2. Calibration of the Players• Video and Artifacts
3. Review of the System
4. Adjustments (If needed)
5. Implementation
6. Evaluate/Reflect/Adjust
7. Re-Calibrate…
![Page 15: INTER-RATER AGREEMENT IN KANSAS Summer Principals Academy July 22-24, 2014 Abilene, KS.](https://reader030.fdocuments.net/reader030/viewer/2022032803/56649e2d5503460f94b1d20a/html5/thumbnails/15.jpg)
15
• Start Slowly• Start with
“Engagement” with a Small Group
• Get Agreement• Gain Confidence
Over Time• Write and Refine in
Teams
Take Aways – To Do List
![Page 16: INTER-RATER AGREEMENT IN KANSAS Summer Principals Academy July 22-24, 2014 Abilene, KS.](https://reader030.fdocuments.net/reader030/viewer/2022032803/56649e2d5503460f94b1d20a/html5/thumbnails/16.jpg)
16
• Start with the End in Mind• Develop Common
Understandings• Use Groups to Ensure
Buy-In• Build Measurable Steps to
Reach Each Goal
Final Thoughts