Introduction To Canvas LMS- Evaluation Planning

7/29/2019 Introduction To Canvas LMS- Evaluation Planning

1/10

EvaluationPlanning

Document

JeffToorongianEDTECH512,Fall2011

Prof.K.DianeHall


2/10

Evaluation Criteria for WBI: An Introduction to the Canvas Learning Management

System

Evaluation Criteria Explanation Data Sources

Effectiveness:

Mastery of WBIlearning goals as

indicated throughcompletion of Module

lessons and activities

WBI modules will have video

tutorials and demonstrations.Effectiveness will be demonstrated

through successful completion ofmodule activities

-Web-based

assessments -Completed module

learning objects andproducts

Appeal: Self-paced

WBI is used and wellreceived. Learners

work through alltraining modules.

Content is engagingand well received.

Video tutorials wellreceived.

As self-paced instruction, the WBI

will need to attract and retain learnerinterest to keep them engaged

through the entire course. Instructionwill need to establish itself as a

valued resource for all instructorsusing the new LMS for their teaching

activities.Modules will feature video tutorials

reviewing important material in eachmodule. These resources will keep

learners engaged and provideimportant visual learning

opportunities.

-Learner comments and

feedback.

-Course facilitatorcomments and

feedback.

-Usage data onparticipant progress

through course content(did learners complete

all modules, or start andquit?)

-General feedback fromlearners, facilitators,

and SME, regarding

course content, deliverymethods, and levels oflearner engagement and

overall impressionswith course content,

structure, and deliverywithin Canvas LMS.

Efficiency: WBI

provided a short,engaging, and content

rich experience for

busy instructors

Participants are busy. The goal of

the WBI is to deliver self-paced,efficiently presented information that

they need to get up and running with

a new LMS product. By deliveringthe instruction within the LMS theyreceive additional hands-on, practical

experience that will further supporttheir successful completion of

learning goals.

-Feedback from course

evaluation will providedetails on time invested

by learners. Open-

ended comments inevaluation survey willallow learners to

comment further ondelivery methods,

course structure, andcontent.


3/10

Evaluation Matrix for Formative Evaluation Questions for WBI: An Introduction to

the Canvas Learning Management System

Evaluation Criteria andCategories

Questions Methods and Tools

Effectiveness/Goals Are the learning goals alignedwith instructional materials?

Are the learning goals alignedwith instructional activities and

assessments?

Are the activities and assessments

achievable in a time that isreasonable for a self-pacedcourse?

Do the learners have feedback

mechanisms available to them ifneeded?

End-user survey, Instructor

SME review, Designer expertreview.

Effectiveness/Content Do the video tutorials provideadequate content to achieve

module learning objectives?

Are the instructional materialsaccurate and comprehensiveenough to support learning goals?

Are the instructional materials

aligned with learning activitiesand goals?

End-user survey, InstructorSME review, Designer expert

review, Extant data.

Effectiveness/Technology Are learners able to access allinstructional materials needed?

Does the course navigation

function properly?

Do the video tutorials open and

play properly?

Do other learning resources(links) work when accessed?




4/10

Does downloadable content open

properly in the target application?

Effectiveness/Message

Design

Is the course structure and

navigation clear, intuitive, andfunctional within the LMS?

Are the instructional video

tutorials well designed andreadable?

Is the audio quality of the tutorials

conducive to learning withoutmajor distractions or technical

problems?

Is the video quality of the tutorialsconducive to learning without

major distractions or technicalproblems? Are visuals clear and

easy to read?

Are the instructional materialsclearly and consistently labeled

and linked?

Are the lesson activities andassignments clearly described and

easily comprehended?



Appeal/Goals Do the goals encourage

participation from the learners?

Are the learning objectivesappropriate for the most important

skills needed by learners?



Appeal/Content Is the web-based instruction

engaging to the learner?Do the learning goals provide for

active learning opportunities forparticipants?

Does the instruction encourage


SME review, Designer expertreview, Extant data.


5/10

learners interact with the LMS asthey develop new skills?

Appeal/Technology Is the instructional content easyand intuitive to navigate?

Are there any grammatical or

typographic errors that distractfrom the learning objectives?

Do media files and external links

work properly?

End-user survey, Designerexpert review.

Appeal/Message Design Is the instructional messagealigned to the intended learnergroup (university instructors)?

Is the instructional web

environment intuitive, clear, andconducive to learning?

Is navigation through course

content easy and intuitive?

Are graphical elements within theweb environment attractive,

effective and not distracting to the

learner.

End-user survey, Designer

expert review.

Efficiency/Goals Are learning goals clearly andsimply stated?

Are the learning activities inalignment with course objectives?



Efficiency/Content Is the instruction presented clearlyand without distractions?

Does each learning module

represent unique learningobjectives?

End-user survey, InstructorSME review, Designer expert

review, Extant data.

Efficiency/Technology Does the web-based instruction

function effectively and withoutdistraction or complications to the

learner?




6/10

Do the video tutorials coverimportant and necessary

information to help learnersachieve goals?

Is the audio in instructional mediaof high quality and support thelearner and facilitate a transfer of

information?

Efficiency/Message

DesignAre the learning modules unified

in design and overall learninggoals?

Are the learning objectives

presented logically and in a

manner appropriate for learnersuccess?

End-user survey, Designer

expert review.

Formative Evaluation Plan

Who are the Stakeholders? Primary Stakeholders:

-Course trainer/facilitator will have responsibilityfor developing course content in instructional

modules, designing course layout within the CanvasLMS, and facilitating course participants during

course.

-eLearning support staffwill be responsible forcourse design (in conjunction with course trainer),

support content development and delivery, providetechnical support to course participants, and facilitate

course administration for course participants.

-Course participants are the focus of the WBI.Their success in achieving learning objectives and

gaining a strong working knowledge of their newlearning management system will be the critical

factor that will determine the success of the trainingeffort.


7/10

Secondary Stakeholders:

-IT staffwho support the server-side LMS

installation, back end support, and campus

application integrations are responsible for keepingthe LMS functional.

-Senior Administrators, including CIO,Library/eLearning Director, and executive team

members are ultimately responsible for the selectionand support of the LMS, and providing the necessary

resources (staff and funding) to provide training toinstructors.

What is being evaluated? - Course Objectives: Are the objectives that havebeen developed for the training course appropriate to

assure that participants will have sufficient skills withthe new LMS to begin developing their course in it?

- Course Assignments and Exercises: Do the

assignment and exercises developed and presented inthe WBI sufficient to assure learner mastery of the

skill(s) in question?

- Course Assessments: Do the assessments toolsadequately measure achievement of the learning

objectives?

- Design/Navigation: Is the navigation within thecourse clear, intuitive, and consistent? Are design

elements and the graphical display of course content(as experienced within the LMS) appealing,

appropriate to the content and learner audience, andconvey course information effectively?

- Course Content: Are the course materials of highquality, free of errors (grammatical and technical),and robust in breadth and depth of required content?

- Delivery Methods: Do the instructional delivery

technologies work properly? Does all linked contentfunction? Do all media files play as intended within

or external from the LMS, as designed?


8/10

Who are the Evaluators and

Reviewers?Evaluator/Designer: An instructional designer, onstaff in the eLearning group will provide expert

review of the WBI, identify any areas of designweakness, suggestions to improve the design. This

instructional designer has more than seven years of

practical experience supporting faculty on campus inthe design of their course elements within the LMS.They also have extensive technical expertise with

troubleshooting and user LMS technical issues.

Expert Reviewer/Instructor: An AssociateProfessor in the university chemistry department will

provide technical expertise relevant to the Canvaslearning management system. He has piloted several

of his online and campus-based courses in the CanvasLMS over the last two semesters. His practical

knowledge of Canvas will provide invaluable insightinto the WBI content from the viewpoint of an

instructor.

End-User reviewers: A small group (no more than10) recent participants in the Canvas WBI course will

complete an online user-feedback survey. Thisreviewer group will be university instructors of

various ranks. Their feedback will offer importantinformation about course content, organization, and

design from a user perspective. Follow-up interviews

with a subset of these reviewers may be conducted iftime and resources permit, and are deemed necessary.

What are the Evaluation

Methods and Tools?-Expert reviews from designer and instructor via

checklist or written comments.

-End-user survey data.

-Extant data(see Evaluation Matrix for detailed breakdown each

criteria)


9/10

When and how should formative evaluation take place?

The instructional designer and instructor reviewer will conduct initial review ofWBI design plans. At this early stage their input will likely involve observations and

suggestions related to course content, organization, and identified learning objectives.

An analysis of instructional activities and learner motivational strategies may also bereviewed. As course development nears completion, the instructional designer andinstructor reviewer will review course objectives, content, and message design issues to

find areas of weakness and identify errors that need addressing.

Due to a very small e-Learning support staff and the relatively short format of thisweb-based course, a limited try-out cycle will be conducted, consisting of a small-group

evaluation. This group of up to ten university instructors or other individuals withteaching experience and responsibilities, such a graduate students, will be asked to

engage the course content as it is designed and delivered within the Canvas learningmanagement system. The try-out cycle with end-user reviewers will also provide

important feedback on the technical quality of the instructional content and theapproximate time it takes to complete all course materials.

What decisions need to be made as the WBI design plan and prototype are

developed and revised?

Results from formative evaluation will primarily be qualitative data and will becompiled from survey results and evaluation comments. These findings will guide the

design team in their decision-making as they identify the areas of concern and resultantchanges to the course materials. Instances where there are divergent opinions regarding

needed changes, as identified in the evaluation results, may require consultation withstakeholders for guidance. Comments and completed course assignments from end-user

reviewers in the small group try-out cycle will provide important feedback regardingattitudes and newly developed skills. This information should prove valuable in helping

to determine needed changes to course assignments, course instructional content, andcourse design and navigation.

Preliminary Plan for Summative Evaluation

This is a new web-based training course, so no preexisting data from an original

course is available. Results from participant end-user surveys accompanied by theircompletion of course activities will be the primary methods of confirming success of the

instruction in preparing participants to use the Canvas LMS. Preliminary summativeevaluation plan criteria, questions, and related data sources are provided in the table

below. Results of the summative evaluation findings will be presented in a written reportthat will be submitted to the Director of the Center for Teaching and Learning and the

designated primary course instructor/facilitator. Information on the structure of theevaluation process, data sources, results and any recommendations will be included in the

report.


10/10

Evaluation Criteria Main Questions Data Sources

Effectiveness - Did course help learnersto achieve learning

objectives?

- Were the activities andassessments achievable in a

time allocated for the self-paced course?

- Do learner perceptions

indicate they are preparedto use the Canvas LMS

upon completion of the

training course?

Course assessmentactivities

End-user survey results,and LMS student tracking

data.

End-user survey results

Efficiency - Did each learning moduleprovide unique learning

objectives as part of thetraining course?

Expert reviewer responses,end-user survey results

Appeal - Were course instructional

materials effective andengaging to the users?

- Did the web course design

and navigation supportparticipant learning

objectives?

- Do course objectivesencourage learning

participation in WBI?

End-user survey results

Expert reviewer responses,

end-user survey results

Expert reviewer responses,end-user survey results

2011JeffToorongian

Introduction To Canvas LMS- Evaluation Planning

Documents

Transcript of Introduction To Canvas LMS- Evaluation Planning