P ERFORMANCE M EASURE R EPORTING : TPP AND PREIS G RANTEES OAH/ACYF and RTI International Webinar...
-
Upload
kelley-hensley -
Category
Documents
-
view
214 -
download
0
Transcript of P ERFORMANCE M EASURE R EPORTING : TPP AND PREIS G RANTEES OAH/ACYF and RTI International Webinar...
1
PERFORMANCE MEASURE REPORTING:TPP AND PREIS GRANTEES
OAH/ACYF and RTI InternationalWebinarAugust 29, 2011
2
Reporting System Development Today: Overview of the system
Data to be collected Options for ways to enter or upload data
Next Steps Pilot testing and refinement of the system Training on how to use the system
3
Today’s Topics
Review of data collection and reporting Roles and responsibilities of RTI and
Grantees Introduction to the Web Portal System Tracking data using spread sheets or web Uploading data Reporting on participant-level measures of
perceived impact Reporting on participant-level measures of
actual impact
4
Data Collection and Reporting Schedule
Instrument Data Collection Time Point Reporting
Enrollment Baseline ImplementationPost Program
Later Follow up Semi-Yearly Yearly
Demographics - Student Reach ALL ALLParticipant Level Impact Questionnaire CD TIER 2 CD TIER 2 CD TIER 2 ALL
Perceived Program ImpactQuestionnaire ALL ALL
Student Attendance-Dosage ALL ALL
Other Attendance-Reach and Dosage ALL ALL
Adherence ALL ALL
Fidelity Observation ALL ALL
Partners ALL
Training ALL
Dissemination ALL
Fidelity Process ALL
5
Roles and Responsibilities
Developing and maintaining website (e.g. assigning IDs, troubleshooting)
Training and TA Analyzing data
provided Reporting performance
measures to OAH
Collecting data Setting standards and
tracking implementation Organizing and labeling
data following guidance from RTI
Ensuring quality and security of data
Reporting accurate data on time
RTI Grantees/Evaluators
6
Web Portal Overview
A secure website hosted by RTI for reporting performance measures
Each grantee will have its own page Grantees will receive credentials to access the website Website allows for 3 different modes of data entry
Raw data entry (performance measures computed by the system)
Uploading of data files Direct entry of computed performance measures – special
conditions. Provides dynamic reports tailored to your
specifications
7
Security and Confidentiality of Data
Data stored in a SQL Server database
Database accessible only to analysts assigned to the project
Electronic communications occur via a secure Internet connection
128-bit encryption of transmissions through secure socket layers (SSL)
Reports will present data in aggregate form
RTI will assign credentials to grantees
Individuals granted access to the website will have IDs and passwords
IDs and passwords are not to be shared
No personal identifiers may be sent to RTI for participant level data – youth or parents Numeric IDs No social security numbers
RTI Grantees
8
Web Portal System Home Page
9
Data Reporting Overview
All grantees need to be sure that the program(s) (e.g., BART, Reducing the Risk) you are implementing are the correct ones listed on the system
All grantees need to identify the classes or sections in which the program is being implemented – can upload
Grantees that will use the system for tracking dosage will need to identify the participants on the system
Grantees can track dosage, fidelity, and reach using spreadsheets or other systems
10
Web Portal System Home Page Menu
11
Grantee Data: Adding New Programs
12
Adding Program Sections/Classes
13
Adding Participants
14
Demographic Data
Demographic information needs to be collected when you enroll participants in case they Are not scheduled to take a baseline
measure Miss taking the scheduled baseline
measure Drop out before taking the post-test
Use the web system or use forms that you develop
If you have questions specific to your unique situation, please contact your project officer
15
Entering Demographics on the Web
Participant ID: --------------------------------Gender: o Male
Femaleo Other
Age: o <= 10 yearso 11-12 13-14 o 15-16 o 17-18 o >= 19
Grade: o <= 6 grade 7-8o 9-10o 11-12o GEDo Technical/vocationalo Collegeo Not currently in school
Ethnicity: o Hispanic/Latino Not Hispanic/Latinoo Unknown/not reported
16
Enrolling Participants in Class
17
Adding Class Sessions
18
Uploading Data
19
Adding Demographics by Excel
Participant Gender Age Grade Ethnicity Race
Language
Special Population
1010234 2 4 3 2 3 1 6
ID only 1=male 1 - <= 10 years
1 - <= 6 grade 1 - Hispanic/Latino
1 - American Indian/Alaskan Native 1 - English
1 - Pregnant or parenting teens
2=female 2 - 11-12 2 - 7-8 2 - Not Hispanic/Latino 2 - Asian 2 - Spanish
2 - Youth in Foster Care
3=other 3 - 13-14 3 - 9-10 3 - Unknown/not reported
3 - Black/ African American 3 - Chinese 3 - Homeless youth
4 - 15-16 4 - 11-12
4 - Native Hawaiian/other Pacific Islander 4 - other
4 - Youth in Juvenile Justice system
5 - 17-18 5 - GED 5 - White 5 - other
6 - >= 19
6 - Technical/ vocational 6 - Other 6- none
7 - College 7 - More than one race
8 - Not currently in school
8 - Unknown/unreported
20
Adding a Class/Section using Excel
SectionName Facilitator Location TimeDay
Section01 John Smith Wilson High Mondays @ 2pm
21
Enrolling a Participant in a Class/Section using Excel
Participant
1010234
22
Deriving Performance Measures After basic data are in the system it will
be possible to calculate performance measures Reach Dosage Fidelity
Dosage and Fidelity will require additional data that can be included either using web entry or excel
23
Reach
# of youth served in program Defined as participating in at least one activity Data will be reported by demographic
characteristics obtained at enrollment # of other clients served – parents, sibs
Defined as participating in at least one activity Data will be reported as numbers
Will need attendance data for RTI to compute
24
Dosage
Data obtained from Attendance Logs Youth/other client attendance clients every session
Tracking options RTI will have a web entry system for entering
individual or summary data Grantees can tally data on a spreadsheet and upload
Measures to be computed Median percent of total intended program services
received by youth/other participants % of participants who received at least 75% of the
program
25
Recording Attendance on the Web
Add a SessionIf you have your own system for maintaining session information, please click here to upload data. Otherwise, please use this form to add session information.
Session name/number: ______________Session date: ____________Session time: ______________Number of Activities Scheduled: ______________Number of Activities Completed: ______________
Session participants (check all who attended) Participant 1Participant 2Participant 3Participant 4Participant 5Participant 6NOTE: Prepopulated from enrollees to classSave and add anotherSave and finish
26
Recording Attendance in Excel
SessionName Participant Attended
Session01 1010234 1
1010235 0
1010236 1
Session02 1010234 1
1010235 1
1010236 1
1= Attended 0 = Didn’t Attend
27
Fidelity – Facilitator Measured Adherence to Program Core Components
Defined as the percentage of prescribed activities that are completed
Completed for all sessions with youth and other clients
Activities tracked on Fidelity Monitoring Logs Grantees can enter directly on web Grantees can tally data on a spreadsheet and upload
Measures to be Computed Median % of activities completed of those scheduled Median % of sessions implemented Median % of sessions with a fidelity log
28
Recording Adherence on the Web
Note: Need to assess whether activities were implemented with minor or major changes prior to entering
29
Recording Adherence using Excel
SessionName SessionDate ActivitiesScheduled ActivitiesCompletedSession01 10/1/21 8 8
Session02 10/8/21 10 9
30
Other Information Needed for Fidelity
Total number of prescribed sessions for each program – grantee to determine
For programs where a variable number of sessions can be offered, the prescribed sessions for each section – grantee to determine
The total number of sessions delivered – grantee to determine
31
Recording Sessions Implemented
SectionName
Session01
Session02
Session03
Session04
Session05
Section01 1 1 1 1 1
Section02 1 1 1 0 0
Section03 1 1 0 1 1
Section04 1 0 1 0 1
Section05 1 0 1 1 1
Grantee responsible for setting a target for the total number of prescribed sessions to be delivered and knowing which ones are delivered 1=Implemented 0 = Not Implemented
32
Recording Sessions Implemented with Logs
SectionName
Session01
Session02
Session03
Session04
Session05
Section01 1 1 1 1 1
Section02 1 1 1 0 0
Section03 1 1 0 1 1
Section04 1 0 1 0 1
Section05 1 0 1 1 1
Grantee responsible for knowing which sessions are delivered1= Log Received 0 = Session Delivered but No Log
33
Fidelity Observation
For Adherence (% activities completed) Use the Fidelity Monitoring Logs (i.e., same
fidelity monitoring tool as the facilitator) Enter on web or on excel spread sheets just
as you will enter for facilitator measured fidelity
For Program Quality Use the Observation Form for TPP Grantees Enter on web or excel spread sheets and
upload to web
34
Fidelity Observation Measures Adherence Performance Measures
Total # sessions implemented Total # sessions observed by an
independent observer Median percentage of activities completed
across sessions that were observed Minimum and maximum percentages of
activities completed across sessions observed
All can be calculated from data entered
35
Recording Quality in Excel
Percentage of observed sessions rated either 4 or higher for overall quality
SectionName
SessionName
Qual1
Qual2
Qual3
Qual4
Qual5
Qual6a
Qual6b
Qual6c
Qual6d
Qual6e
Qual7
Section01
Session04
4 5 3 3 4 5 4 4 5 3 4
36
Grantee Summary Measures – All Grantees
Dissemination# manuscripts accepted/published total # titles of manuscripts# manuscripts submitted total # titles of manuscripts# Presentations - National Total # titles and venues national presentations# Presentations - Regional Total # titles and venues regional presentations# Presentations - State Total # titles and venues state presentations
# Partners# organizations/schools with formal agreements Total ## organizations/schools assist with implementation Total ## organizations/schools actively involved but no formal agreement Total #
Partner Retention# organizations/schools with a formal agreement at start of program year still engaged Total #
Training of Facilitators# new facilitators/teachers Trained Total ## facilitators/teachers receive follow up training Total #
37
Grantee Summary Measures – Tier 2/PREIS
Completed and approved pieces of program necessary to package for replication:
Program Component ApprovalDate
LogicModel
CoreComponents
FideltyTools
CurriculumManual
Facilitator Manual
TrainingMaterials
AdaptationGuidance
38
Grantee Summary Measures (cont.)
Fidelity Process Checklist Reported once a year via web based data entry or
uploading excel spread sheet
Process1
Process2
Process3
Process4
Process5
Process6
Process7
Process8
Process9
Process10
Process11
Process13
2 2 2 1 2 1 1 2 0 1 2 1
0= Not at all 1 = Somewhat 2 = Completely
39
Perceived Impact Measures
Data can be entered in 2 ways: Enter data directly into the web-based reporting
system Create subsets of larger datasets that contain only the
variables required for performance measures, and upload (Excel, CSV)
RTI will provide standard variable labels and formats
40
Entering Perceived Impact on the Web
41
Actual Impact Measures
For grantees with rigorous evaluations, evaluators will collect additional participant-level data (actual impact measures) as part of their scheduled data collections
Evaluators will upload data only for those variables specified in the performance measure documentation
Data reported annually—any data collected that year
42
Uploading Participant-Level Data
43
Important Dates
All data must be uploaded no later than November 30 and May 31 of each year
Same dates as your progress reports All grantees who enter summarized data,
must send their detailed data to RTI for verification and must provide documentation of how variable track to RTI variables
Those who enter summarized data must send their progress officer a detailed description of the system for approval
44
Next Steps for RTI
Wrap up programming – target completion date 9/2 Pilot test web reporting system – target completion
date 9/9 Send email to grantees to request information
regarding users of the system (name, role, email) and after receipt to create accounts – target completion date 9/9
Virtual tour via a WebEx for the users who will be on the system and distribute the credentials - 9/12 -9/14
Launch reporting system!!NOTE: ALL DATES ARE APPROXIMATE
45
Questions
Data and reporting questions – RTI Special circumstances regarding
collection – your project officer