Assessing OER impact across varied organisations and learners: experiences from the ‘Bridge to...
-
Upload
oer-research-hub -
Category
Education
-
view
100 -
download
1
description
Transcript of Assessing OER impact across varied organisations and learners: experiences from the ‘Bridge to...
![Page 1: Assessing OER impact across varied organisations and learners: experiences from the ‘Bridge to Success’ initiative](https://reader035.fdocuments.net/reader035/viewer/2022070322/5590ee1c1a28ab0a748b4740/html5/thumbnails/1.jpg)
Assessing OER impact across varied organisations and learners: experiences from the Bridge to
Success initiative
Tim Coughlan
Horizon Digital Economy Research & School of Computer Science,
The University of Nottingham (UK)
Patrick McAndrew & Beck Pitt
Institute of Educational Technology, The Open University (UK)
Nassim Ebrahimi
Anne Arundel Community College, MD, USA.
![Page 2: Assessing OER impact across varied organisations and learners: experiences from the ‘Bridge to Success’ initiative](https://reader035.fdocuments.net/reader035/viewer/2022070322/5590ee1c1a28ab0a748b4740/html5/thumbnails/2.jpg)
Overview and Aims
• Introducing the Topic
• Introducing Bridge to Success
• Research Methods Used
• Overview of Users and Pilot Contexts
• Definitions of “Impact”
• Issues in Assessing “Impact”…
• Towards an Effective General Model for Research
![Page 3: Assessing OER impact across varied organisations and learners: experiences from the ‘Bridge to Success’ initiative](https://reader035.fdocuments.net/reader035/viewer/2022070322/5590ee1c1a28ab0a748b4740/html5/thumbnails/3.jpg)
Introducing the Topic
• How to understand the impact of OER and of initiatives?
• What is meant by impact?
• What do different stakeholders want to know?
• What are the particular complexities of research in this space?
• How could we overcome them?
![Page 4: Assessing OER impact across varied organisations and learners: experiences from the ‘Bridge to Success’ initiative](https://reader035.fdocuments.net/reader035/viewer/2022070322/5590ee1c1a28ab0a748b4740/html5/thumbnails/4.jpg)
What is Bridge to Success?
The project aimed to offer free, open educational resources to prepare adults to successfully and confidently transition to a college environment in the US, to pursue advanced qualifications, or to be successful in their chosen careers…
Next Generation Learning Challenges
![Page 5: Assessing OER impact across varied organisations and learners: experiences from the ‘Bridge to Success’ initiative](https://reader035.fdocuments.net/reader035/viewer/2022070322/5590ee1c1a28ab0a748b4740/html5/thumbnails/5.jpg)
Research Methods“What is the impact of Bridge
to Success on student success and educator practice?”
Product Design/Adaptation Process
Contexts of Use
Student and
Institutional Experience
Student Outcomes
Project Impact
Evaluation Research
Surveys
Quizzes
Interviews & Focus Groups
Pre- and Post-Surveys
Analysis of interviews, meetings, emails etc.
Courses underwent accessibility, usability and design analysis
Action Research
Periodic Interviews with keystakeholders and the project team
Quantitative
Analytics of website use
Institutional student data
![Page 6: Assessing OER impact across varied organisations and learners: experiences from the ‘Bridge to Success’ initiative](https://reader035.fdocuments.net/reader035/viewer/2022070322/5590ee1c1a28ab0a748b4740/html5/thumbnails/6.jpg)
Who piloted the content?
• A total of 16 US college and non-college institutions (October 2011 – December 2012).
• As at August 2012 a total of 17 out of 26 completed pilots had been conducted with cohorts of low-income students. A further 16 pilots were ongoing at this date (total = 42 distinct pilots).
• As at February 2013 9 US college and non-college institutions were continuing to utilise Bridge to Success materials, in a variety of contexts.
Overview of Users and Pilots
![Page 7: Assessing OER impact across varied organisations and learners: experiences from the ‘Bridge to Success’ initiative](https://reader035.fdocuments.net/reader035/viewer/2022070322/5590ee1c1a28ab0a748b4740/html5/thumbnails/7.jpg)
Defining “Impact”
Differences across Stakeholders• Funders
• Policy Makers
• Educational Organisations
• Academic Research
Independent Learning
Problem solving
Collaboration
Communication
Mastery of Subject
![Page 8: Assessing OER impact across varied organisations and learners: experiences from the ‘Bridge to Success’ initiative](https://reader035.fdocuments.net/reader035/viewer/2022070322/5590ee1c1a28ab0a748b4740/html5/thumbnails/8.jpg)
Defining “Impact”
Measures of Impact:• Retention / Completion / Persistence.
• Whether a student has “mastered” material.
• “Deeper Learning.”
• Changes to educator practice.
• Increased learner confidence.
Independent Learning
Problem solving
Collaboration
Communication
Mastery of Subject
![Page 9: Assessing OER impact across varied organisations and learners: experiences from the ‘Bridge to Success’ initiative](https://reader035.fdocuments.net/reader035/viewer/2022070322/5590ee1c1a28ab0a748b4740/html5/thumbnails/9.jpg)
Assessing “Impact” in Community Colleges• Nine colleges piloted Bridge to Success materials during the
project’s pilot phase (October 2011 – August 2012).
As at August 2012
• Of the completed pilots that have submitted 100% of their data, and as reported by instructors, 88% of students who completed B2S materials persisted to next semester.
• 98% of these students mastered the subject matter and 96% mastered deeper learning.
63% of these college students were low income.
Of these low income students, 99% persisted to the next
semester, 96% mastered the subject matter and 94%
mastered deeper learning.
![Page 10: Assessing OER impact across varied organisations and learners: experiences from the ‘Bridge to Success’ initiative](https://reader035.fdocuments.net/reader035/viewer/2022070322/5590ee1c1a28ab0a748b4740/html5/thumbnails/10.jpg)
Assessing “impact” in non-college institutions
“I’m not a computer literate person, I always dealt with
things hands-on, it’s paper and pencil and I deal with things
hands-on … I mean I came out of school in ’88…For me to go
online and learn online, it’s a challenge.” (Bridge to Success
pilot participant, March 2012)
“The National Assessment of Educational
Progress found that 41% of eighth graders in
free and reduced lunch programs have home
Internet access compared to 72% of their
financially better off peers.” (K
aminski, Seel
& Cullen, 2003)
EXAMPLE ONE
International Workforce Development Agency: supported people across Maryland for over 90 years.
This pilot utilised specific units of the B2S maths course until end of August 2012 to enable students to complete a new requirement math pre-test for a Weatherization program.
Students who originally failed the math entrance exam, 80% passed after working with B2S resources for a 1-3 week period.
![Page 11: Assessing OER impact across varied organisations and learners: experiences from the ‘Bridge to Success’ initiative](https://reader035.fdocuments.net/reader035/viewer/2022070322/5590ee1c1a28ab0a748b4740/html5/thumbnails/11.jpg)
Assessing “impact” in non-college institutions EXAMPLE TWO Family Support Centre in residential area where:
“85% of families are headed by a single female parent;90% have not completed high school and do not have a GED;95% are unemployed, underemployed or receive welfare or other social services.”(Source: Case Study Two Family Support Centre leaflet 2012)
Piloting both Bridge to Success courses to support preparation for General Education Development (GED) and pre-GED examinations.
Outcome of Spring 2012 pilot with 8 participants: • 2 obtained their GED certificate,
• 2 less advanced learners referred to another programme,
• 2 students sat their GED examinations but need to retake,
• 1 participants is now employed following a job training programme and will take their GED during March 2013,
• 1 dropped out of Waverly programme.
“Challenges that are unique to
our setting and population”
![Page 12: Assessing OER impact across varied organisations and learners: experiences from the ‘Bridge to Success’ initiative](https://reader035.fdocuments.net/reader035/viewer/2022070322/5590ee1c1a28ab0a748b4740/html5/thumbnails/12.jpg)
Use of B2S materials to date
Between 1 October 2011 – 17 March 2013:
147,183 Page Views
18,591 Unique Visitors
Bridge to Success (B2S)
exceeded its unique user and
pilot participant ta
rgets: 1822
current/completed participants
and, by September 2012, over
6,000 unique users of B2S
materials.
![Page 13: Assessing OER impact across varied organisations and learners: experiences from the ‘Bridge to Success’ initiative](https://reader035.fdocuments.net/reader035/viewer/2022070322/5590ee1c1a28ab0a748b4740/html5/thumbnails/13.jpg)
How should we really measure impact?
• It is inherent in the Open concept that resources are used in different contexts and in varied ways.
• This is part of what makes for great impact
• But it makes it complex to holistically measure or compare the value of an OER or an OER initiative
![Page 14: Assessing OER impact across varied organisations and learners: experiences from the ‘Bridge to Success’ initiative](https://reader035.fdocuments.net/reader035/viewer/2022070322/5590ee1c1a28ab0a748b4740/html5/thumbnails/14.jpg)
Research Methods and Outcomes
• Usage Statistics
• Show broad evidence of amount of use, trends, locations of users
• Can show paths of individual users / identified groups
• No data on why many events happen (e.g. did a teacher tell them to stop at that point or did they get stuck?)
• In-built pre / post unit assessments
• Show evidence of improvement in scores
• But only in a sample who took both quizzes
![Page 15: Assessing OER impact across varied organisations and learners: experiences from the ‘Bridge to Success’ initiative](https://reader035.fdocuments.net/reader035/viewer/2022070322/5590ee1c1a28ab0a748b4740/html5/thumbnails/15.jpg)
Research Methods and Outcomes
• Observations and ethnographic work
• Provide evidence of impact outside the expected
• Begin to understand why it works or where it doesn’t
• Time consuming, reliant on opportunities, doesn’t scale well
• Institutional Data
• Longer-term, detailed understanding of impact on individuals
• Heterogeneous
• Requires long-term involvement and relationship building
• Can support institutional comparison, so may be wary of sharing
• Hard to find suitable comparator if targeting at (e.g. underachievers)
![Page 16: Assessing OER impact across varied organisations and learners: experiences from the ‘Bridge to Success’ initiative](https://reader035.fdocuments.net/reader035/viewer/2022070322/5590ee1c1a28ab0a748b4740/html5/thumbnails/16.jpg)
Research Methods and Outcomes
• Pre / post course surveys
• Can be used to identify learner and their context in the system and link with other data (e.g. institution ID)
• End point of use is hard to know, and unlikely to be when a learner wants to fill in a survey
• Creates an initial barrier to getting on with using the resources
• Surveys / Interviews with Instructors
• Understand context of learning and use of OER
• Elicit instructors judgement on types of impact (e.g. ‘Deeper Learning’ definitions)
• Can be combined with support and advice
• Scope for definitions to be interpreted differently
![Page 17: Assessing OER impact across varied organisations and learners: experiences from the ‘Bridge to Success’ initiative](https://reader035.fdocuments.net/reader035/viewer/2022070322/5590ee1c1a28ab0a748b4740/html5/thumbnails/17.jpg)
What is required?
• Understanding the different contexts of learners
• Identifying individuals and learning contexts across data sources
• Operationalising varied impact definitions to be studied at scale
• Being able to adapt and expand to opportunistic assessments
![Page 18: Assessing OER impact across varied organisations and learners: experiences from the ‘Bridge to Success’ initiative](https://reader035.fdocuments.net/reader035/viewer/2022070322/5590ee1c1a28ab0a748b4740/html5/thumbnails/18.jpg)
An Effective General Model
![Page 19: Assessing OER impact across varied organisations and learners: experiences from the ‘Bridge to Success’ initiative](https://reader035.fdocuments.net/reader035/viewer/2022070322/5590ee1c1a28ab0a748b4740/html5/thumbnails/19.jpg)
Conclusions and Next Steps
• Broadness of use should be acknowledged in OER research
• Methodological and technical innovation needed
• Mixed methods approaches
• Relationship building and long term planning
• More research on use of Bridge to Success content via the Hewlett funded OER Research Hub project
• Reversioning of the courses to take into account user feedback and analysis
![Page 20: Assessing OER impact across varied organisations and learners: experiences from the ‘Bridge to Success’ initiative](https://reader035.fdocuments.net/reader035/viewer/2022070322/5590ee1c1a28ab0a748b4740/html5/thumbnails/20.jpg)
THANK YOU!
Tim Coughlan [email protected]
@t1mc
Beck Pitt [email protected]
@BeckPitt
Patrick McAndrew [email protected]
@openpad
Nassim Ebrahimi [email protected]
![Page 21: Assessing OER impact across varied organisations and learners: experiences from the ‘Bridge to Success’ initiative](https://reader035.fdocuments.net/reader035/viewer/2022070322/5590ee1c1a28ab0a748b4740/html5/thumbnails/21.jpg)
Resources
• Watch a review of Bridge to Success functionality and the courses: https://www.youtube.com/watch?v=9pHyYO5d01I
• Visit the Bridge to Success website: http://bridge2success.aacc.edu/
• Visit the OER Research Hub website: http://oerresearchhub.org/
![Page 22: Assessing OER impact across varied organisations and learners: experiences from the ‘Bridge to Success’ initiative](https://reader035.fdocuments.net/reader035/viewer/2022070322/5590ee1c1a28ab0a748b4740/html5/thumbnails/22.jpg)
Bibliography• Atkins, D. E., Seely Brown, J., Hammond, A. L., A Review of the Open
Educational Resources Movement: Achievements, Challenges and New Opportunities (2007) (http://www.hewlett.org/uploads/files/Hewlett_OER_report.pdf)
• Bailey & Cho Issue Brief: Developmental Education in Community Colleges prepared for The White House Summit on Community Colleges (October 2010)
• Bridge to Success Report One: Adaptation, Integration and Engagement
• Case Study Two Family Support Centre Leaflet 2012
• College Board: Advocacy & Policy Centre Education Pays 2010 (http://trends.collegeboard.org/sites/default/files/education-pays-2010-full-report.pdf)
• College Board: Advocacy & Policy Centre Trends in Student Aid 2012 (http://trends.collegeboard.org/sites/default/files/student-aid-2012-full-report-130201.pdf)
• Kaminski, K. Seel, P. & Cullen K. Technology Literate Students? Results from a Survey (2003, EDUCAUSE Quarterly, No 3)