Adaptive courseware vendor selection and engagement
-
Upload
karen-vignare -
Category
Education
-
view
97 -
download
0
Transcript of Adaptive courseware vendor selection and engagement
![Page 1: Adaptive courseware vendor selection and engagement](https://reader034.fdocuments.net/reader034/viewer/2022042604/587873dc1a28ab497b8b6b0d/html5/thumbnails/1.jpg)
Adaptive Courseware Vendor Selection and
EngagementKaren Vignare, PhD, MBA
CEO, KV [email protected]
![Page 2: Adaptive courseware vendor selection and engagement](https://reader034.fdocuments.net/reader034/viewer/2022042604/587873dc1a28ab497b8b6b0d/html5/thumbnails/2.jpg)
Agenda• Introductions• Vendor Engagement• Selection• University team• Project Management
• Setting Expectations• Changing Culture and Building Buy-In• Iterating and Scaling
![Page 3: Adaptive courseware vendor selection and engagement](https://reader034.fdocuments.net/reader034/viewer/2022042604/587873dc1a28ab497b8b6b0d/html5/thumbnails/3.jpg)
Karen Vignare, PhD, MBA• 20 years experience online, emerging technologies in learning• Started online programs, worked with international universities• Managed blended learning, MOOCs, and adaptive learning projects• Extensive research portfolio• More about me at LinkedIn, https://www.linkedin.com/in/karen-
vignare
![Page 4: Adaptive courseware vendor selection and engagement](https://reader034.fdocuments.net/reader034/viewer/2022042604/587873dc1a28ab497b8b6b0d/html5/thumbnails/4.jpg)
Selection process• Short and simple…mission fit• Tools exist to help make choices (Thanks Tyton and EdSurge)• Efficacy Research exists (Thanks SRI)• Many of you already have vendor partners• Multiple tools could be useful if you can make that work
![Page 5: Adaptive courseware vendor selection and engagement](https://reader034.fdocuments.net/reader034/viewer/2022042604/587873dc1a28ab497b8b6b0d/html5/thumbnails/5.jpg)
What We Did Not Have
Source: Tyton Partners
Lists of companies and categorization of tool type
![Page 6: Adaptive courseware vendor selection and engagement](https://reader034.fdocuments.net/reader034/viewer/2022042604/587873dc1a28ab497b8b6b0d/html5/thumbnails/6.jpg)
Source: https://www.edsurge.com/higher-ed/special-projects/dl/orientation
![Page 8: Adaptive courseware vendor selection and engagement](https://reader034.fdocuments.net/reader034/viewer/2022042604/587873dc1a28ab497b8b6b0d/html5/thumbnails/8.jpg)
Adaptive Technology Integration ComparisonsNext Generation Digital Learning Environment
Courseware in Context Framework
Reported pilots
Interoperability and Integration
Latest version compliance LTI
Most report easy interoperability
Personalization Critical within system No clear standard match so reports are more on reported outcome success
Analytics, Advising, and Learning Assessment
Caliper and LTI Needs improvement as data exchange is not widely used (yet)
Collaboration Currently not as important
Accessibility and Universal Design
Critical to scale Major concerns (not just a vendor issue)
![Page 9: Adaptive courseware vendor selection and engagement](https://reader034.fdocuments.net/reader034/viewer/2022042604/587873dc1a28ab497b8b6b0d/html5/thumbnails/9.jpg)
The Situation before Pilots at UMUC• Moving to all OER/free resources• Needed LTI (latest version) for integration with BrightSpace (D2L)• Preferred platform tools over others• Involved in previous grants, Gates Foundation PLN, Breakthrough
Models Incubators, and Digital Courseware
![Page 10: Adaptive courseware vendor selection and engagement](https://reader034.fdocuments.net/reader034/viewer/2022042604/587873dc1a28ab497b8b6b0d/html5/thumbnails/10.jpg)
![Page 11: Adaptive courseware vendor selection and engagement](https://reader034.fdocuments.net/reader034/viewer/2022042604/587873dc1a28ab497b8b6b0d/html5/thumbnails/11.jpg)
Evaluation ProcessInputs
Outputs
Outcomes
Analyze
Iterate
![Page 12: Adaptive courseware vendor selection and engagement](https://reader034.fdocuments.net/reader034/viewer/2022042604/587873dc1a28ab497b8b6b0d/html5/thumbnails/12.jpg)
Research Questions Independent Variable
Dependent Variables/Outcome Measures
Analysis
1. Do completion rates differ by condition?
Condition (Treatment and Control)
Course completion chi-square
2. Does student achievement differ by condition?
Condition (Treatment and Control)
Homework gradesProject gradesMidterm exam gradesFinal exam gradesFinal course grades
t-test
3. Do students view the adaptive learning system as an effective learning tool
Student Survey Descriptives
4. Is there a relationship between time spent studying in the adaptive learning system and student achievement?
Total login timeFinal course grades
Correlation
![Page 13: Adaptive courseware vendor selection and engagement](https://reader034.fdocuments.net/reader034/viewer/2022042604/587873dc1a28ab497b8b6b0d/html5/thumbnails/13.jpg)
Research Questions
Independent Variable
Dependent Variables/Outcome Measures
Analysis
Is there a relationship between the level of faculty engagement and student achievement?
Faculty Engagement Indicators
Final course grades
Correlation
Is there a relationship between response time effort and student achievement?
RTE
Quiz grades
Homework grades
Correlation
![Page 14: Adaptive courseware vendor selection and engagement](https://reader034.fdocuments.net/reader034/viewer/2022042604/587873dc1a28ab497b8b6b0d/html5/thumbnails/14.jpg)
Evaluations• Random Control Trials are the gold standard• But there are always variables that can’t be completely controlled in
education• Repeated findings are important BUT • It may not be that easy (Or it may)• Looking for the silver bullet could deter leveraging the tool in new ways• Most technologies require more data/students to be effectiveResources: SRI, https://www.sri.com/work/projects/adaptive-learning-market-acceleration-program
![Page 15: Adaptive courseware vendor selection and engagement](https://reader034.fdocuments.net/reader034/viewer/2022042604/587873dc1a28ab497b8b6b0d/html5/thumbnails/15.jpg)
University Selection Team• Center for Innovation in Learning & Student Success• Learning Design & Solutions• Information Technology• Undergraduate School representative• Graduate School representative• Procurement
![Page 16: Adaptive courseware vendor selection and engagement](https://reader034.fdocuments.net/reader034/viewer/2022042604/587873dc1a28ab497b8b6b0d/html5/thumbnails/16.jpg)
Timing of Pilot Initiation• Initial vendor discussions fall 2014• Multiple demos, online and on site• Both continuing and new grant activities were concurrent• Goal to start Summer 2015 especially with a design
Search & Demos (Fall, 2014)
Evaluation (Winter,
2015)
Contracting (S/S 2015)
Launch (Summer
2015)
![Page 17: Adaptive courseware vendor selection and engagement](https://reader034.fdocuments.net/reader034/viewer/2022042604/587873dc1a28ab497b8b6b0d/html5/thumbnails/17.jpg)
Pilots Chosen• Mix of those courses identified by analytics as those needing
improvement• Program chairs interest/willingness• Student support/services• Grant obligations
![Page 18: Adaptive courseware vendor selection and engagement](https://reader034.fdocuments.net/reader034/viewer/2022042604/587873dc1a28ab497b8b6b0d/html5/thumbnails/18.jpg)
Vendors Selected at UMUCSummer 2015 LaunchCogbooksRealizeITEdReady
GrantsOLI (CMU started in 2009)OLI (Stanford) potentially with Open EdX (2015)Lumen Learning (2015)
![Page 19: Adaptive courseware vendor selection and engagement](https://reader034.fdocuments.net/reader034/viewer/2022042604/587873dc1a28ab497b8b6b0d/html5/thumbnails/19.jpg)
Project Management• Developed a work flow and estimated types of personnel needed• Used consultant from earlier grant to get documents & sites prepared• Instructional Design team dedicated one person as project manager• Instruction Design team dedicated three designers (not full-time)• Information Technology dedicated one person• Innovation team included three people • This group then broke into the multiple projects which included faculty
and subject matter experts• Established a steering committee
![Page 20: Adaptive courseware vendor selection and engagement](https://reader034.fdocuments.net/reader034/viewer/2022042604/587873dc1a28ab497b8b6b0d/html5/thumbnails/20.jpg)
Project Manager• Led documentation efforts (extensive at UMUC)• Identified with lead ID vendor PM• Set up training/meetings• Updated documentation
![Page 21: Adaptive courseware vendor selection and engagement](https://reader034.fdocuments.net/reader034/viewer/2022042604/587873dc1a28ab497b8b6b0d/html5/thumbnails/21.jpg)
Instructional Designers• Learn technologies• Identify existing content• Work with IT on integration• Setting up pilot processes• Work with faculty to design course map and content flow• Work with vendors ID staff• Work with faculty to improve course design (more content &
activities)
![Page 22: Adaptive courseware vendor selection and engagement](https://reader034.fdocuments.net/reader034/viewer/2022042604/587873dc1a28ab497b8b6b0d/html5/thumbnails/22.jpg)
Information Technology• LTI integration with D2L• Solve back-end issues• Work with IDs whenever vendor meetings included technical
architecture/software discussions• Integrate data into data warehouse
![Page 23: Adaptive courseware vendor selection and engagement](https://reader034.fdocuments.net/reader034/viewer/2022042604/587873dc1a28ab497b8b6b0d/html5/thumbnails/23.jpg)
Center for Innovation in Learning• Build shared knowledge base• Responsible for scope & contracts• Project Owner• Co-manage with ID project progress• Run Steering Committee & shared progress (transparently)• Report to senior academics• Evaluations• Presentations
![Page 24: Adaptive courseware vendor selection and engagement](https://reader034.fdocuments.net/reader034/viewer/2022042604/587873dc1a28ab497b8b6b0d/html5/thumbnails/24.jpg)
Academics• Assigned by program chairs• Generally an adjunct and/or collegiate (full-time)• Adjuncts were paid by contract• Collegiates in lieu of teaching • Build MORE content, assessments
![Page 25: Adaptive courseware vendor selection and engagement](https://reader034.fdocuments.net/reader034/viewer/2022042604/587873dc1a28ab497b8b6b0d/html5/thumbnails/25.jpg)
Vendor• Stay transparent• Agree to a definitions/taxonomy—so you are on the same page• Agree to the schedule but also remember delays happen• Try to find others working with the vendor (create a community)• If you don’t get answers, escalate
![Page 26: Adaptive courseware vendor selection and engagement](https://reader034.fdocuments.net/reader034/viewer/2022042604/587873dc1a28ab497b8b6b0d/html5/thumbnails/26.jpg)
Challenges• Staff consistency• Project manager left university• IDs needed to be transferred as other projects had ebbs & flows
• Vendor challenges (with above & some other issues)• Projects chosen
• Often relied on a single SME/faculty• Projects chosen needed to be high priority instead of having
competing priorities (OER, redesign)• NEED more content than expected
![Page 27: Adaptive courseware vendor selection and engagement](https://reader034.fdocuments.net/reader034/viewer/2022042604/587873dc1a28ab497b8b6b0d/html5/thumbnails/27.jpg)
Setting Expectations• Be Transparent with everyone• Be CLEAR about hypothesis and why this pilot• Meet with team (even virtually) regularly• Meet with vendor regularly• Set evaluation expectation, by this date we expect X• Be clear about the funding, where is it being spent• Keep senior leadership informed• Share your progress with the community
![Page 28: Adaptive courseware vendor selection and engagement](https://reader034.fdocuments.net/reader034/viewer/2022042604/587873dc1a28ab497b8b6b0d/html5/thumbnails/28.jpg)
Iterating and Scaling• Test and Learn• There is often a vicious cycle of not having final evaluations in time to
iterate before the next launch• Plan your schedule to allow for appropriate redesign• Scaling is critical though for testing
![Page 29: Adaptive courseware vendor selection and engagement](https://reader034.fdocuments.net/reader034/viewer/2022042604/587873dc1a28ab497b8b6b0d/html5/thumbnails/29.jpg)
Pisano, G. (2015). You Need an Innovation Strategy. Harvard Business Review. June 2015
Changing the Culture
![Page 30: Adaptive courseware vendor selection and engagement](https://reader034.fdocuments.net/reader034/viewer/2022042604/587873dc1a28ab497b8b6b0d/html5/thumbnails/30.jpg)
Changing the Culture• Universities are not very good at scaling effective practices• What are your innovation/piloting goals—research, student success,
redesigned teaching (no wrong answers)• You need to build the case post grant • You will need to create an innovation pipeline• Leverage the tools in the community to get beyond one off projects
![Page 31: Adaptive courseware vendor selection and engagement](https://reader034.fdocuments.net/reader034/viewer/2022042604/587873dc1a28ab497b8b6b0d/html5/thumbnails/31.jpg)
Lessons LearnedThe future of adaptive learning in higher education depends on the commitment level of universities• Preparation...really take the time to understand the power of the tool, take a class• Skill up faculty, instructional staff and technology team• Pick your course(s) based on solving problems (is content difficult, would more
student practice help, does immediacy help students• Build content maps and you will need more content than you currently have
(unless you use pre-packaged)• Learn to use the dashboard• Iterate probably at least three times....