OER Degee Initiative Kickoff | Data & Evaluation Services
-
Upload
achieving-the-dream -
Category
Education
-
view
186 -
download
0
Transcript of OER Degee Initiative Kickoff | Data & Evaluation Services
© 2015 SRI International© 2015 SRI International
OER DEGREE INITIATIVE RESEARCH & EVALUATION
Being a Research Partner
OER Degree Kickoff InstituteSan Francisco, June 13-15
© 2015 SRI International© 2015 SRI International
Agenda
• Meet each other• What do we want to learn?• Our approach• Cost analysis• Roles & responsibilities, permissions, & timeline• Questions / concerns?
© 2015 SRI International© 2015 SRI International
Introductions
• Meet the SRI team
• Meet the rpkGROUP team
• Meet each other
3
© 2015 SRI International© 2015 SRI International
Hypotheses to test• Some students do not purchase required textbooks due to high
cost and do poorly or withdraw from courses as a result
• Some students take fewer courses than they otherwise would due to cost of textbooks
• If this cost is removed, students will progress more quickly and successfully towards degree attainment
• OER Degrees are a cost effective way to improve student outcomes and reduce financial burden on students
• Redesigning programs around OER will trigger additional improvements in pedagogy and institutional culture
© 2015 SRI International© 2015 SRI International
Research Questions for the Evaluation – Academic impacts and implementation
RQ 1: Academic outcomes: Do students who take multiple OER degree classes make more progress to degree than similar students who take traditional classes? Are OER degrees more or less beneficial to particular subgroups of students? What are the key moderators of effects on student outcomes?
RQ 2: What are best practices, facilitators, and barriers associated with implementation of OER degree programs?
RQ 3: What impacts do OER degrees have on key stakeholders’ experiences and on institutional culture?
(We’ll get to economic impacts in a minute)
© 2015 SRI International© 2015 SRI International
Our Approach to the Evaluation
Aggregate pass/completion data will enable us to evaluate the overall impact of the OER Degree Initiative - how many OER course sections are offered, how many students are involved
Quasi-experimental studies will enable us to estimate the impact of OER Degrees on key student outcomes
Combine with cost analysis to measure cost effectiveness and sustainability
Implementation research will enable us to provide formative feedback to grantees and broader community
© 2015 SRI International© 2015 SRI International
To put this another way…
Enrollment/completion/ cost reporting; instructor survey – all 34 colleges
Quasi experimental studies – 10-12 Research Partners
Implementation data collection (phone
interviews, student surveys, site visits)Cost deep
dives – 5 colleges
© 2015 SRI International© 2015 SRI International
Research Partner Focus
• Most research on OER effectiveness has focused on course-level outcomes. This project will focus primarily on student progress to degree.
• Work with SRI to identify comparison population of similar students, ideally with no OER course exposure
• Provide agreed data for all students who enroll in OER degrees and comparison groups
8
Why so important?
What are we asking?
© 2015 SRI International© 2015 SRI International
But aren’t course-level outcomes important too?
YES!
• SRI can also provide technical support for selected course-level studies, such as large enrollment gateway courses.
• These should be included in college study plans.
9
© 2015 SRI International© 2015 SRI International
Research Questions for the Cost Evaluation RQ1: How do OER degrees impact costs for students and institutions?
RQ2: What is the “return” on investment (ROI) for students, colleges, and other stakeholders, where “return” refers to financial savings from reduced cost of attendance and time to completion, or increased revenue to colleges from greater persistence and attainment?
RQ3: What does it cost an institution to make the transition to OER degrees, including both start-up costs and lost revenue streams?
RQ4: What is the cost effectiveness of this model in terms of cost per desired unit of impact on student outcomes?
RQ5: Is this model self-sustainable (i.e., can OER degree programs be established and maintained without philanthropy)?
© 2015 SRI International© 2015 SRI International
Approach to the Cost Evaluation Cost evaluation will address the financial impacts that OER Degrees have
on students and institutions. Capture impact of OER on student degree costs, institutional pricing strategies, and institutional revenues from lost
textbook sales and improved student outcomes.
Analyze start-up and ongoing costs associated with developing and delivering OER degrees.
Financial data combined with information on academic outcomes will inform the cost effectiveness of OER degree programs.
Collect information via interviews, SRI instructor surveys, data collection and analysis templates. Capture top level impacts on student costs and institutional revenues from all 34 colleges using a data collection and
analysis template.
Collect additional information from the five cost analysis partners on cost of implementing OER degree courses and programs (activity based cost model).
Provide technical assistance to help grantees report the information requested and to make appropriate data assumptions when needed.
© 2015 SRI International© 2015 SRI International
Cost Evaluation Activities
1. Webinar on cost evaluation approach and data collection
2. Data collected via SRI phone interviews
3. Data collected via 2 SRI instructor surveys
4. Cost partner interviews
5. Data collection template
6. Modeling of economic impacts on students and institutions
7. Formative and summative reporting
© 2015 SRI International© 2015 SRI International
Research Activity
SRI Responsibility Grantee Responsibility
Quasi-experimental studies
• Provide study planning tool
• Draft college study plan
• Complete study planning tool• Provide feedback and sign off on
college-specific study plan
Instructor surveys
• Develop survey instrument
• Conduct online survey
• Provide instructor contact information
• Encourage participation
Student survey
• Develop survey instrument
• Distribute packets of consent forms and surveys
• Conduct online survey for online classes
• Administer gift cards
• Provide student email addresses (for online classes)
• Coordinate distribution of survey packets to instructors (for F2F classes).
• Instructors conduct paper surveys during class time.
Roles & Responsibilities
© 2015 SRI International© 2015 SRI International
Research Activity
SRI Responsibility Grantee Responsibility
Site visits • Coordinate with grantee liaisons
• Conduct visits
• Coordinate with SRI to schedule interviews, focus groups, class observations
Section-level data reporting
• Provide data template• Manage data collection
across sites• Provide secure file server
• Coordinate campus data collection
• Provide section-level data at 4 intervals for all OER courses and, where possible, comparable non-OER courses
Student-level data reporting
• Provide data template• Comparison group
matching & data analysis
• Provide student-level data at 4 or 5 intervals (depending on prior agreement)
Roles & Responsibilities Continued
© 2015 SRI International© 2015 SRI International
Near Term Activities
• SRI liaisons will be assigned to each grantee
• Liaisons will work with grantees to − Sign letters of commitment− Create college study plan, starting with study planning
tool − Determine whether local research permission is
required− If yes, provide inputs to research application− Put in place data sharing agreements
15
© 2015 SRI International© 2015 SRI International
Project Timeline
© 2015 SRI International© 2015 SRI International
Research Challenges
• Contamination from existing OER courses
• Definition of “treatment” cohort for QEDs. Is four OER courses the right threshold?
• Student attrition
• Variation in academic calendars, data availability, data definitions
• Others?
© 2015 SRI International© 2015 SRI International
Questions or Concerns?Thank you!
© 2015 SRI International© 2015 SRI International
What data is required for quasi-experimental studies?• Essential elements. For each
student in the study population:− Baseline indicators (placement
test scores, GPA)− Demographic variables (DOB,
gender, race/ethnicity, Pell status)
− Whether the courses they enroll in are OER/non-OER
− Course level-outcomes (withdraw, course grade)
− Retention / progression− Credits earned for each course − Cumulative credits earned− Credential attainment
• Desirable elements− Whether student enters
program of study− Additional background data
(HS GPA, SAT/ACT score, year of HS graduation, ELL status, first generation status)
© 2015 SRI International© 2015 SRI International
Recommended Campus Teams• Data person• CFO / finance• IT • Librarian?• Counselors / registrar
© 2015 SRI International© 2015 SRI International
When What Who
Summer 2016 Detailed research plan Design college-level quasi-experimental studies
Grantees / SRI
Fall 2016 (Oct-Nov)
SRI conducts first instructor surveySRI conducts phone interviews
SRI, with assistance from grantees
Winter 2017 SRI collects first batch of student-level data (test run) Research partners with OER courses starting fall 2016
Spring 2017 Formative implementation reportFirst set of site visits
SRI, with assistance from grantees
Summer 2017, Winter 2018, Summer 2018, Winter 2019
All grantees submit section level data on OER coursesResearch Partners submit student-level data for spring 2017
All grantees
Fall 2017 Student survey SRI, with assistance from grantees
Throughout grant period
SRI provides technical assistance to grantees for design and implementation of studies, data collection
SRI
Fall 2018 Second instructor survey SRI
Summer/fall 2019 Final reporting SRI
Major Evaluation Components