Post on 29-Jun-2015
description
Assessing OER impact across varied organisations and learners: experiences from the Bridge to
Success initiative
Tim Coughlan
Horizon Digital Economy Research & School of Computer Science,
The University of Nottingham (UK)
Patrick McAndrew & Beck Pitt
Institute of Educational Technology, The Open University (UK)
Nassim Ebrahimi
Anne Arundel Community College, MD, USA.
Overview and Aims
• Introducing the Topic
• Introducing Bridge to Success
• Research Methods Used
• Overview of Users and Pilot Contexts
• Definitions of “Impact”
• Issues in Assessing “Impact”…
• Towards an Effective General Model for Research
Introducing the Topic
• How to understand the impact of OER and of initiatives?
• What is meant by impact?
• What do different stakeholders want to know?
• What are the particular complexities of research in this space?
• How could we overcome them?
What is Bridge to Success?
The project aimed to offer free, open educational resources to prepare adults to successfully and confidently transition to a college environment in the US, to pursue advanced qualifications, or to be successful in their chosen careers…
Next Generation Learning Challenges
Research Methods“What is the impact of Bridge
to Success on student success and educator practice?”
Product Design/Adaptation Process
Contexts of Use
Student and
Institutional Experience
Student Outcomes
Project Impact
Evaluation Research
Surveys
Quizzes
Interviews & Focus Groups
Pre- and Post-Surveys
Analysis of interviews, meetings, emails etc.
Courses underwent accessibility, usability and design analysis
Action Research
Periodic Interviews with keystakeholders and the project team
Quantitative
Analytics of website use
Institutional student data
Who piloted the content?
• A total of 16 US college and non-college institutions (October 2011 – December 2012).
• As at August 2012 a total of 17 out of 26 completed pilots had been conducted with cohorts of low-income students. A further 16 pilots were ongoing at this date (total = 42 distinct pilots).
• As at February 2013 9 US college and non-college institutions were continuing to utilise Bridge to Success materials, in a variety of contexts.
Overview of Users and Pilots
Defining “Impact”
Differences across Stakeholders• Funders
• Policy Makers
• Educational Organisations
• Academic Research
Independent Learning
Problem solving
Collaboration
Communication
Mastery of Subject
Defining “Impact”
Measures of Impact:• Retention / Completion / Persistence.
• Whether a student has “mastered” material.
• “Deeper Learning.”
• Changes to educator practice.
• Increased learner confidence.
Independent Learning
Problem solving
Collaboration
Communication
Mastery of Subject
Assessing “Impact” in Community Colleges• Nine colleges piloted Bridge to Success materials during the
project’s pilot phase (October 2011 – August 2012).
As at August 2012
• Of the completed pilots that have submitted 100% of their data, and as reported by instructors, 88% of students who completed B2S materials persisted to next semester.
• 98% of these students mastered the subject matter and 96% mastered deeper learning.
63% of these college students were low income.
Of these low income students, 99% persisted to the next
semester, 96% mastered the subject matter and 94%
mastered deeper learning.
Assessing “impact” in non-college institutions
“I’m not a computer literate person, I always dealt with
things hands-on, it’s paper and pencil and I deal with things
hands-on … I mean I came out of school in ’88…For me to go
online and learn online, it’s a challenge.” (Bridge to Success
pilot participant, March 2012)
“The National Assessment of Educational
Progress found that 41% of eighth graders in
free and reduced lunch programs have home
Internet access compared to 72% of their
financially better off peers.” (K
aminski, Seel
& Cullen, 2003)
EXAMPLE ONE
International Workforce Development Agency: supported people across Maryland for over 90 years.
This pilot utilised specific units of the B2S maths course until end of August 2012 to enable students to complete a new requirement math pre-test for a Weatherization program.
Students who originally failed the math entrance exam, 80% passed after working with B2S resources for a 1-3 week period.
Assessing “impact” in non-college institutions EXAMPLE TWO Family Support Centre in residential area where:
“85% of families are headed by a single female parent;90% have not completed high school and do not have a GED;95% are unemployed, underemployed or receive welfare or other social services.”(Source: Case Study Two Family Support Centre leaflet 2012)
Piloting both Bridge to Success courses to support preparation for General Education Development (GED) and pre-GED examinations.
Outcome of Spring 2012 pilot with 8 participants: • 2 obtained their GED certificate,
• 2 less advanced learners referred to another programme,
• 2 students sat their GED examinations but need to retake,
• 1 participants is now employed following a job training programme and will take their GED during March 2013,
• 1 dropped out of Waverly programme.
“Challenges that are unique to
our setting and population”
Use of B2S materials to date
Between 1 October 2011 – 17 March 2013:
147,183 Page Views
18,591 Unique Visitors
Bridge to Success (B2S)
exceeded its unique user and
pilot participant ta
rgets: 1822
current/completed participants
and, by September 2012, over
6,000 unique users of B2S
materials.
How should we really measure impact?
• It is inherent in the Open concept that resources are used in different contexts and in varied ways.
• This is part of what makes for great impact
• But it makes it complex to holistically measure or compare the value of an OER or an OER initiative
Research Methods and Outcomes
• Usage Statistics
• Show broad evidence of amount of use, trends, locations of users
• Can show paths of individual users / identified groups
• No data on why many events happen (e.g. did a teacher tell them to stop at that point or did they get stuck?)
• In-built pre / post unit assessments
• Show evidence of improvement in scores
• But only in a sample who took both quizzes
Research Methods and Outcomes
• Observations and ethnographic work
• Provide evidence of impact outside the expected
• Begin to understand why it works or where it doesn’t
• Time consuming, reliant on opportunities, doesn’t scale well
• Institutional Data
• Longer-term, detailed understanding of impact on individuals
• Heterogeneous
• Requires long-term involvement and relationship building
• Can support institutional comparison, so may be wary of sharing
• Hard to find suitable comparator if targeting at (e.g. underachievers)
Research Methods and Outcomes
• Pre / post course surveys
• Can be used to identify learner and their context in the system and link with other data (e.g. institution ID)
• End point of use is hard to know, and unlikely to be when a learner wants to fill in a survey
• Creates an initial barrier to getting on with using the resources
• Surveys / Interviews with Instructors
• Understand context of learning and use of OER
• Elicit instructors judgement on types of impact (e.g. ‘Deeper Learning’ definitions)
• Can be combined with support and advice
• Scope for definitions to be interpreted differently
What is required?
• Understanding the different contexts of learners
• Identifying individuals and learning contexts across data sources
• Operationalising varied impact definitions to be studied at scale
• Being able to adapt and expand to opportunistic assessments
An Effective General Model
Conclusions and Next Steps
• Broadness of use should be acknowledged in OER research
• Methodological and technical innovation needed
• Mixed methods approaches
• Relationship building and long term planning
• More research on use of Bridge to Success content via the Hewlett funded OER Research Hub project
• Reversioning of the courses to take into account user feedback and analysis
THANK YOU!
Tim Coughlan tim.coughlan@nottingham.ac.uk
@t1mc
Beck Pitt Beck.Pitt@open.ac.uk
@BeckPitt
Patrick McAndrew p.mcandrew@open.ac.uk
@openpad
Nassim Ebrahimi nebrahimi@aacc.edu
Resources
• Watch a review of Bridge to Success functionality and the courses: https://www.youtube.com/watch?v=9pHyYO5d01I
• Visit the Bridge to Success website: http://bridge2success.aacc.edu/
• Visit the OER Research Hub website: http://oerresearchhub.org/
Bibliography• Atkins, D. E., Seely Brown, J., Hammond, A. L., A Review of the Open
Educational Resources Movement: Achievements, Challenges and New Opportunities (2007) (http://www.hewlett.org/uploads/files/Hewlett_OER_report.pdf)
• Bailey & Cho Issue Brief: Developmental Education in Community Colleges prepared for The White House Summit on Community Colleges (October 2010)
• Bridge to Success Report One: Adaptation, Integration and Engagement
• Case Study Two Family Support Centre Leaflet 2012
• College Board: Advocacy & Policy Centre Education Pays 2010 (http://trends.collegeboard.org/sites/default/files/education-pays-2010-full-report.pdf)
• College Board: Advocacy & Policy Centre Trends in Student Aid 2012 (http://trends.collegeboard.org/sites/default/files/student-aid-2012-full-report-130201.pdf)
• Kaminski, K. Seel, P. & Cullen K. Technology Literate Students? Results from a Survey (2003, EDUCAUSE Quarterly, No 3)