Post on 08-Feb-2016
description
Chair:•Steven M. Ross, Center for Research in Educational Policy; Center on Innovation & ImprovementCollaborating Researchers:•Jen Harmon, Center on Innovation & Improvement•Kenneth Wong, Brown University; Center on Innovation & Improvement
Supplemental Educational Supplemental Educational ServicesServices
Approving, Monitoring, Approving, Monitoring, EvaluatingEvaluating
Promising Practice Briefs:Promising Practice Briefs:Approving, Monitoring, and Evaluating Approving, Monitoring, and Evaluating ProvidersProviders
Commissioned by the Office of Innovation and Improvement
To be developed and released in fall 2008
Promising Practice Promising Practice BriefsBriefs
Sources of data• State SES Director Survey Completed by All States• National Meetings• Site Visits to States• Interviews with SES Directors• Authors’ Experiences as SES Consultants and
Researchers
RecruitmentRecruitment
Two-thirds of the states actively (14%) or informally (52%) recruit providers via:
Direct invitationsWeb announcementsDistrict publicityState meetings and other means
Application RequirementsApplication Requirements
Aside from core application information, states include as optional components:
Attendance at informational meetings Recommendations from former clients A detailed plan for communicating with teachers, parents,
and district coordinators In-person interview Demonstration/description of a tutoring lesson Identification of minimum tutor qualifications
Strategies Used in the Approval ProcessStrategies Used in the Approval Process
16 17
12
8
0
5
10
15
20
Allow forappeals
Grant approvalfor 2-3 years
Require re-approval every
year
Probationarystatus
Num
ber o
f Sta
tes
The Most Successful PracticesThe Most Successful Practices
• Application review using independent review teams (f = 19)
• Clear scoring rubrics (f = 9)• Technical assistance to applicants (f = 5)• Requesting curriculum and tutoring descriptions
(f = 2)• Provider interview (f = 2)
ChallengesChallenges
6 6
14
4
0
5
10
15
20
Evaluatingevidence of
effectivenessand alignment
Insufficienttime/resources
Disconnectsbetween
promises anddelivery
Evaluatingfinancialstability
Num
ber o
f Sta
tes
Desired ImprovementsDesired Improvements
Multiple states want to improve their process by:
Requiring submission of lesson plansAdding an interview processStrengthening scoring rubricImproving reviewer training
Increased Federal AssistanceIncreased Federal Assistance
Increased federal assistance is desired in the areas of:
Specific guidance in practices and policiesFacilitating networking and information sharing
between states
Monitoring Focus Monitoring Focus
Nearly all states view the main focus of monitoring to be:
Provider compliance with rules and regulations (93%)
Districts’ implementation of SES (84%)
ApplicationsApplications
• Three-fourths (74%) of the states use a “formal” monitoring process
• Almost 80% use monitoring results formally (38%) or informally (40%) in evaluating providers
Feedback and CapacityFeedback and Capacity
Feedback– 55% of states produce a written report– 23% have face-to-face meetings
Capacity– 45% monitor all providers each year– 75% monitor at least half yearly
Types of Technical AssistanceTypes of Technical Assistance
6 6
23
0
5
10
15
20
25
Workshops or meetings Listserves and postedwebsite information
Email and phonediscussions
Num
ber o
f Sta
tes
On-Site Monitoring Activities (33%)On-Site Monitoring Activities (33%)
• Visits may be announced or random• Includes online and in-home providers• Review of tutoring documents, materials, etc.• Uses checklist, rubric, or rating scale• May be one person or a team• Tutors or students may be interviewed• Most often at school or community site
Desk MonitoringDesk Monitoring
End-of-year fiscal and participation reportQuarterly reportsOn-line implementation trackingProvider self-evaluationParent and student satisfaction surveysComplaints regarding provider complianceComparison of provider vs. district enrollment
data
District MonitoringDistrict Monitoring
Supplementary for some statesThe only monitoring done in other
states
Most Successful PracticesMost Successful Practices
11
7 75
0
5
10
15
Clear andconsistent
processes andcriteria
On-site visits Districtassistance andcollaboration
On-line or otherdata tracking
system
Num
ber o
f Sta
tes
ChallengesChallenges
15
32
4
0
5
10
15
Limited time,resources, staff
Uncertaintyregarding
provider removal
Limited tools andstandards
Districtparticipation
Num
ber o
f Sta
tes
Desired ImprovementsDesired Improvements
4 4
2 2
13
0
5
10
15
Increaseresources and
staff
Increase on-sitevisits
Improvemonitoring
tools/standards
Create datamanagement
system
Increase districtand parent
input
Num
ber o
f Sta
tes
Implementation of Provider EvaluationsImplementation of Provider Evaluations
• 30 states “regularly” evaluate• 15 are still in planning stages• Remainder “informally” evaluate
Is the Provider Evaluation Effective?Is the Provider Evaluation Effective?
17
9 9
0
5
10
15
20
Agree Undecided Disagree
Num
ber o
f Sta
tes
Evaluation ComponentEvaluation Component
2724
19
23 22
0
5
10
15
20
25
30
District SESCoordinator
Survey/Interview
Parent Survey ExternalEvaluator
EvaluationFeedback to
Providers
Principal Survey
Num
ber o
f Sta
tes
Evaluation ComponentEvaluation Component
17
14
1011
10
0
5
10
15
20
Teacher Survey Public Reporting StudentSurvey/Interview
ClassificationRatings
Removal ofProviders
Num
ber o
f Sta
tes
Student Achievement Student Achievement Analysis ApproachesAnalysis Approaches
1412
10 9
0
5
10
15
20
Student gains onprovider tests
"Matched-pair"analyses of state test
Multiple regressionanalyses of state
score
State performancebenchmark gains
Num
ber o
f Sta
tes
Most Successful Most Successful Evaluation Evaluation PracticesPractices
10
32 2
0
5
10
15
Using externalevaluator
On-line trackingsystem
Clearly defined,transparent
system
Solicitingdistrict/parent
input
Num
ber o
f Sta
tes
ChallengesChallenges
9
76
4
2
0
5
10
15
Data limitations
Limited funding/ resources
Small studentsamples
Districtcompliance
Low completion rates/attrition
Num
ber o
f Sta
tes
Desired ImprovementsDesired Improvements
54 4
3
0
5
10
Use of automateddata collection
system
Increasestaffing/resources
Develop long-rangeplan to increase
rigor
Supplement /improve data
Num
ber o
f Sta
tes
Contact InformationContact Information
Sam Redding, sredding@centerii.org Marilyn Murphy, mmurphy@centerii.org Steven Ross, smross@memphis.edu Jen Harmon, jharmon@centerii.org Kenneth Wong, kenneth_wong@brown.edu
Visit our web site at www.centerii.org