Evaluating Online Learning: Issues and Strategies Mark Hawkes Dakota State University Innovations,...
-
Upload
randolph-small -
Category
Documents
-
view
216 -
download
0
Transcript of Evaluating Online Learning: Issues and Strategies Mark Hawkes Dakota State University Innovations,...
![Page 1: Evaluating Online Learning: Issues and Strategies Mark Hawkes Dakota State University Innovations, Educating New Generations March 1, 2002.](https://reader030.fdocuments.net/reader030/viewer/2022032607/56649ebb5503460f94bc31f6/html5/thumbnails/1.jpg)
Evaluating Online Learning: Issues and Strategies
Mark HawkesDakota State UniversityInnovations, Educating New GenerationsMarch 1, 2002
![Page 2: Evaluating Online Learning: Issues and Strategies Mark Hawkes Dakota State University Innovations, Educating New Generations March 1, 2002.](https://reader030.fdocuments.net/reader030/viewer/2022032607/56649ebb5503460f94bc31f6/html5/thumbnails/2.jpg)
Presentation Objective
Discuss online learning evaluation approaches in graduate programs at two universities Identify criteria/indicators suitable for the evaluation of online learning environments
![Page 3: Evaluating Online Learning: Issues and Strategies Mark Hawkes Dakota State University Innovations, Educating New Generations March 1, 2002.](https://reader030.fdocuments.net/reader030/viewer/2022032607/56649ebb5503460f94bc31f6/html5/thumbnails/3.jpg)
Distance Learning Literature
Evaluation . . . . . . . No Yes(79%)
(21%)
Focus . . . . . . . . . Training Education
Impact on Learning . . . . . . . No Yes
(87%)(13%)
(58%) (42%)
![Page 4: Evaluating Online Learning: Issues and Strategies Mark Hawkes Dakota State University Innovations, Educating New Generations March 1, 2002.](https://reader030.fdocuments.net/reader030/viewer/2022032607/56649ebb5503460f94bc31f6/html5/thumbnails/4.jpg)
Instructor Student Student Student Student Student
Internet
Login Interface
Content
Resources
AssessmentCommunication
Modes
Support Services System Resources
Managementand Organizational
Information
OnlineLearning Architecture
Institutional
![Page 5: Evaluating Online Learning: Issues and Strategies Mark Hawkes Dakota State University Innovations, Educating New Generations March 1, 2002.](https://reader030.fdocuments.net/reader030/viewer/2022032607/56649ebb5503460f94bc31f6/html5/thumbnails/5.jpg)
Familiar Online Learning Evaluation Targets . . .
interface designinstructional designstudent satisfactiontechnology accessfaculty satisfactioneconomic viabilitydepartmental capacityinterdepartmental collaboration
![Page 6: Evaluating Online Learning: Issues and Strategies Mark Hawkes Dakota State University Innovations, Educating New Generations March 1, 2002.](https://reader030.fdocuments.net/reader030/viewer/2022032607/56649ebb5503460f94bc31f6/html5/thumbnails/6.jpg)
Common Dimensions of Effective Online Learning:
Relevant and challenging assignments Providing adequate and timely feedback through teacher-student interactionFlexibility in teaching and learning Constructing coordinated learning environments Constructing rich environments for student to student interaction
![Page 7: Evaluating Online Learning: Issues and Strategies Mark Hawkes Dakota State University Innovations, Educating New Generations March 1, 2002.](https://reader030.fdocuments.net/reader030/viewer/2022032607/56649ebb5503460f94bc31f6/html5/thumbnails/7.jpg)
DSU’s Educational Technology Program
Students:36 Credit hour MS program80% Education (K12, Technical, Higher Ed.)20% Business/industry90% Online; 10% On campusFemale 68%; Male 32%Project-based curriculum
![Page 8: Evaluating Online Learning: Issues and Strategies Mark Hawkes Dakota State University Innovations, Educating New Generations March 1, 2002.](https://reader030.fdocuments.net/reader030/viewer/2022032607/56649ebb5503460f94bc31f6/html5/thumbnails/8.jpg)
DSU’s ET EnvironmentPervasive technological culture Consistency between program goals and the state/region-wide initiatives Campus-wide faculty support Institutional experience in Web-based instruction delivery Multi-delivery methodsClient: teachers, teacher developers, instructors, trainers, technology coordinators, etc.Predominantly web-based delivery
![Page 9: Evaluating Online Learning: Issues and Strategies Mark Hawkes Dakota State University Innovations, Educating New Generations March 1, 2002.](https://reader030.fdocuments.net/reader030/viewer/2022032607/56649ebb5503460f94bc31f6/html5/thumbnails/9.jpg)
An Evaluation Model . . .
IlluminativeOperation of Components And Subcomponents
IntegrativeHolistic perspective onThe learning experience
Course &Program Design
Components
Infra-structure/System
Work Flow
Interaction Impact
Process Product
Observing and Detecting Focused on Performance Functional Problems Outcomes
![Page 10: Evaluating Online Learning: Issues and Strategies Mark Hawkes Dakota State University Innovations, Educating New Generations March 1, 2002.](https://reader030.fdocuments.net/reader030/viewer/2022032607/56649ebb5503460f94bc31f6/html5/thumbnails/10.jpg)
Infrastructure/System
Input/output devices Network speed and connectivity Network design/Topology Technical support systems and maintenance
![Page 11: Evaluating Online Learning: Issues and Strategies Mark Hawkes Dakota State University Innovations, Educating New Generations March 1, 2002.](https://reader030.fdocuments.net/reader030/viewer/2022032607/56649ebb5503460f94bc31f6/html5/thumbnails/11.jpg)
Course and Program Design
Nature of the Design Situation Based Role of State and National Standards Sequencing/Instructional Strategies Assessment Motivation: Learning vs. Performance
Visualization Tools and MediaUser InterfaceCourse Management
![Page 12: Evaluating Online Learning: Issues and Strategies Mark Hawkes Dakota State University Innovations, Educating New Generations March 1, 2002.](https://reader030.fdocuments.net/reader030/viewer/2022032607/56649ebb5503460f94bc31f6/html5/thumbnails/12.jpg)
Work Flow
Use of discussion toolsSoftware usageMessage redundancy (audio, video, web pages, emails).Progression Do learners progress through their
work tasks in a linear fashion? (novice-like)
Nonlinear opportunistic fashion (expert-like)
![Page 13: Evaluating Online Learning: Issues and Strategies Mark Hawkes Dakota State University Innovations, Educating New Generations March 1, 2002.](https://reader030.fdocuments.net/reader030/viewer/2022032607/56649ebb5503460f94bc31f6/html5/thumbnails/13.jpg)
![Page 14: Evaluating Online Learning: Issues and Strategies Mark Hawkes Dakota State University Innovations, Educating New Generations March 1, 2002.](https://reader030.fdocuments.net/reader030/viewer/2022032607/56649ebb5503460f94bc31f6/html5/thumbnails/14.jpg)
InteractionSocial and instructionalMust account for all of the following relationships:
Instructor Learner
Learners
Content
Learners
Technology
Content
Technology
![Page 15: Evaluating Online Learning: Issues and Strategies Mark Hawkes Dakota State University Innovations, Educating New Generations March 1, 2002.](https://reader030.fdocuments.net/reader030/viewer/2022032607/56649ebb5503460f94bc31f6/html5/thumbnails/15.jpg)
Learner
Instructor
Content Assessment
Learning Resources(Technology)
LearnerRecords
Catalog info
Query Preferences
Performance
Interaction Example
(Metcalf, Snitzer, Austin, 2001)
Learner
![Page 16: Evaluating Online Learning: Issues and Strategies Mark Hawkes Dakota State University Innovations, Educating New Generations March 1, 2002.](https://reader030.fdocuments.net/reader030/viewer/2022032607/56649ebb5503460f94bc31f6/html5/thumbnails/16.jpg)
Online Course InteractionAnnouncementsEmailDiscussion Board (WebBoard Conferencing)Synchronous text chatStreaming media using Real PlayerDesktop VideoFile LoadingOnline assessmentAudio/video clipsAudio-narrated PowerPoint'sRoom-based Video
![Page 17: Evaluating Online Learning: Issues and Strategies Mark Hawkes Dakota State University Innovations, Educating New Generations March 1, 2002.](https://reader030.fdocuments.net/reader030/viewer/2022032607/56649ebb5503460f94bc31f6/html5/thumbnails/17.jpg)
ImpactCourse performanceCollaborative learningRetention/attrition (course and program)Professional relevance and utilityLearner productivity
![Page 18: Evaluating Online Learning: Issues and Strategies Mark Hawkes Dakota State University Innovations, Educating New Generations March 1, 2002.](https://reader030.fdocuments.net/reader030/viewer/2022032607/56649ebb5503460f94bc31f6/html5/thumbnails/18.jpg)
Evaluation Attributes
Multi-sourced data (students, server log files, etc) Internal and external Performance based Comparison and criterion based
![Page 19: Evaluating Online Learning: Issues and Strategies Mark Hawkes Dakota State University Innovations, Educating New Generations March 1, 2002.](https://reader030.fdocuments.net/reader030/viewer/2022032607/56649ebb5503460f94bc31f6/html5/thumbnails/19.jpg)
![Page 20: Evaluating Online Learning: Issues and Strategies Mark Hawkes Dakota State University Innovations, Educating New Generations March 1, 2002.](https://reader030.fdocuments.net/reader030/viewer/2022032607/56649ebb5503460f94bc31f6/html5/thumbnails/20.jpg)
Student Course Ratings
3.16
3.26
3.27
3.29
3.32
3.32
3.42
3.45
3.45
1 2 3 4
1
1=Strongly Disagree; 4=Strongly Agree
The instructor wasaccessible
The instructor wasresponsive
Course was relevant todegree
Communicationtechnologies were effective
Course activities wereapplicable
Ample opportunity fordiscussion
Course format appropriatefor content
I enjoyed the course
Confident in conductingdesign
![Page 21: Evaluating Online Learning: Issues and Strategies Mark Hawkes Dakota State University Innovations, Educating New Generations March 1, 2002.](https://reader030.fdocuments.net/reader030/viewer/2022032607/56649ebb5503460f94bc31f6/html5/thumbnails/21.jpg)
Helpfulness of Technology
3.13
3.19
3.52
3.58
3.61
3.74
1 2 3 4
1
1=Not at all helpful; 4=Very helpful
Personal communicationwith classmates orinstructor
Informational course emails
Electronic chat sessionswith instructor
Electronic chat sessionswith classmates
Course readings on website
Course texts
![Page 22: Evaluating Online Learning: Issues and Strategies Mark Hawkes Dakota State University Innovations, Educating New Generations March 1, 2002.](https://reader030.fdocuments.net/reader030/viewer/2022032607/56649ebb5503460f94bc31f6/html5/thumbnails/22.jpg)
The breadth of this course was:
1 2 3 4 5 6 7 Not nearly enough The right amount Way too much
Compared to a traditional course
1 2 3 4 5 6 7A much narrower range About the same range A much wider range ofof material was covered of material was covered material was covered
Online: 4.61Compared to traditional 4.65 n=32
![Page 23: Evaluating Online Learning: Issues and Strategies Mark Hawkes Dakota State University Innovations, Educating New Generations March 1, 2002.](https://reader030.fdocuments.net/reader030/viewer/2022032607/56649ebb5503460f94bc31f6/html5/thumbnails/23.jpg)
The depth of this course was:
1 2 3 4 5 6 7 Not nearly enough The right amount Way too much
Compared to a traditional course
1 2 3 4 5 6 7Material was covered in Material was covered in Material was covered in much less depth about the same depth much more depth
Online: 4.48Compared to traditional 4.42 n=32
![Page 24: Evaluating Online Learning: Issues and Strategies Mark Hawkes Dakota State University Innovations, Educating New Generations March 1, 2002.](https://reader030.fdocuments.net/reader030/viewer/2022032607/56649ebb5503460f94bc31f6/html5/thumbnails/24.jpg)
The extent of critical thinking required:
1 2 3 4 5 6 7 Not nearly enough The right amount Way too much
Compared to a traditional course
1 2 3 4 5 6 7 Much less About the same Much more
Online: 4.61Compared to traditional 4.94 n=32
![Page 25: Evaluating Online Learning: Issues and Strategies Mark Hawkes Dakota State University Innovations, Educating New Generations March 1, 2002.](https://reader030.fdocuments.net/reader030/viewer/2022032607/56649ebb5503460f94bc31f6/html5/thumbnails/25.jpg)
The amount of effort put into the course:
1 2 3 4 5 6 7 Much less About the same Much more
Compared to a traditional course
1 2 3 4 5 6 7 Much less About the same Much more
Online: 5.65Compared to traditional: 5.26 n=32
![Page 26: Evaluating Online Learning: Issues and Strategies Mark Hawkes Dakota State University Innovations, Educating New Generations March 1, 2002.](https://reader030.fdocuments.net/reader030/viewer/2022032607/56649ebb5503460f94bc31f6/html5/thumbnails/26.jpg)
Evaluation Processes
Mostly formativeMixed data collection and analysis methods (document analysis, student artifact analysis, survey, interview, text analysis, etc.)Course and program evaluation
![Page 27: Evaluating Online Learning: Issues and Strategies Mark Hawkes Dakota State University Innovations, Educating New Generations March 1, 2002.](https://reader030.fdocuments.net/reader030/viewer/2022032607/56649ebb5503460f94bc31f6/html5/thumbnails/27.jpg)
Typical Problems with Online Courses
Facilitating and encouraging collaboration Time managementStudent proficiency with course toolsAmbiguous directionsTimeliness of feedback
![Page 28: Evaluating Online Learning: Issues and Strategies Mark Hawkes Dakota State University Innovations, Educating New Generations March 1, 2002.](https://reader030.fdocuments.net/reader030/viewer/2022032607/56649ebb5503460f94bc31f6/html5/thumbnails/28.jpg)
Factors Beyond ID Control
Student sophistication with technology tools System capacity Learner availability/accessibility Enthusiastic, responsive instructorGood learner supportMotivated learners
![Page 29: Evaluating Online Learning: Issues and Strategies Mark Hawkes Dakota State University Innovations, Educating New Generations March 1, 2002.](https://reader030.fdocuments.net/reader030/viewer/2022032607/56649ebb5503460f94bc31f6/html5/thumbnails/29.jpg)
How to Design and Effective Online Course?
Follow basic ID principalsBuild a climate of disclosure and full participation Institute informal student evaluation and check-in mechanisms Active and intensive instructor participation Build in as much interactivity as possibleCreate visually interesting screens/pagesEnsure instructions are very clearMulti-mode interaction is criticalRedundancy