FORMATIVE AND SUMMATIVE EVALUATION REPORTS: THE … · FORMATIVE AND SUMMATIVE EVALUATION REPORTS:...

FORMATIVE AND SUMMATIVE EVALUATION REPORTS: THE RETENTION CHALLENGE OF ONLINE STUDENTS 1

Formative and Summative Evaluation Reports

Matt DeLaire, Ann-Marie Peirano, and Jennifer Watson

The University of Alabama


Formative Evaluation Report

Purpose

A previously conducted needs assessment report revealed that effective communication

in online undergraduate courses is a significant factor in the positive retention of online students.

Using this data, an instructional design plan was developed in order to provide guidance to

online instructors on how to successfully incorporate communication into the online courses they

teach. After the completion of the instructional design plan, this formative evaluation report was

undertaken in order to assess the effectiveness and quality of the instructional design plan. The

evaluation specifically focused upon the following issues: understandability of stated objectives,

appropriateness of the instruction for the target audience, alignment of performance objectives

with content activities, thoroughness of the instruction, adequate opportunities to practice usage

of the content, and adequate feedback on products (Rothwell & Kazanas, 1998). This

information will be used to make improvements and any necessary changes to the content of the

instructional design plan before the plan is finalized and potentially put into action (Morrison,

Ross, Kalman, & Kemp, 2011). The audience for this formative evaluation report is the

instructional design team who created the initial plan; they will undertake plan revisions to the

plan as required.

Methods

Data for the formative evaluation was gathered by administering a survey (see Appendix

A) to three members of the target population who could potentially receive the planned

instruction but who are also experienced online instructors. These three subject matter experts

(SMEs) each currently teach at least one online undergraduate course, and all three have been


teaching online courses for four academic years. After the completion of the initial instructional

design plan, the three survey participants were emailed the “Formative Evaluation Instrument”

(Appendix A) and one complete learning episode, termed module three (see Appendix B), from

the six episode instructional design plan. Module three was chosen for evaluation because it was

representative of the larger instructional design plan. Participants were asked to review the

learning episode and electronically complete and return the evaluation survey within three days,

and all three surveys were returned within this time frame. The data collected was analyzed

qualitatively with an explanation of the researchers’ interpretation of extant data.

Analysis

The formative evaluation instrument (Appendix A) consisted of 11 questions dealing

with the effectiveness and quality of the planned instruction. Each question allowed the SME to

answer “yes,” “no,” or “not applicable,” and for seven of the 11 questions, all of the SMEs gave

an answer of “yes” (Appendices C, D, and E). An answer of “yes” indicated that the content

pertaining to the question did not need to be revised in the opinion of the SME. The survey also

provided an area for comments for each question, and these comments provided the most

valuable feedback on the learning episode under review and the larger instructional design plan.

On the evaluation instrument, all of the SMEs reported that the performance objectives

were clearly stated and that the instructional materials support those objectives. The three SMEs

also answered “yes” when asked if the content of the instruction was complete and up to date,

and if the learners were given adequate feedback on real-world products created during the

training. In addition, all three survey respondents stated that the instructional methods were

appropriate for the learners and the instructional methods support the objectives. The four

remaining questions on the evaluation instrument received an answer of “no” or “not applicable”


from one or more of the SMEs.

All of the SMEs answered negatively or felt the question was not applicable when asked

about the alignment of instructional materials to learner characteristics in the learning episode. In

addition, two of the SMEs commented that they did not answer “yes” simply because they did

not know the specific learners who would be completing the instruction. The instructional design

team concluded that more information about the learners should have been provided to the SMEs

before administering the formative evaluation survey, but that the negative SME responses in

this area do not warrant adjusting the instructional deign plan. The other questions that received

negative responses dealt with giving learners adequate opportunities to receive information about

the content, and giving learners adequate opportunities to practice or apply what they have

learned. On the evaluation instrument, all three of the SMEs answered “no” to one or both of

these areas. Comments from the survey participants indicated that more time was needed than

was allotted in the learning episode to effectively present initial information. Likewise, SMEs

commented that more time was needed for participants to produce real world products, such as

revising a lesson plan, than was allotted in module three.

Conclusions

Due to the positive SME responses concerning the performance objectives and how the

instructional methods support those objectives, the instructional design team is confident that the

objectives themselves need no further revision. In addition, comments from the SMEs indicating

that the content of the instructional design plan was valid and that participants would receive

adequate feedback indicated to the instructional design team that the plan does not need to be

adjusted in these areas. However, SME concerns about the time allotment for the presentation of

instruction and the production of work products signals to the design team that modifications to


the plan must made in the area of timing. Therefore, in the instructional design plan, revisions

have been made to extend the time for the presentation of stimulus material and learner guidance

by 30 minutes. In addition, during instances in which learners are being asked to elicit a

performance, such as revising a course outline or lesson plan, the time given to complete these

tasks has been extended by 45 minutes.


Summative Evaluation Plan

Purpose/Goals

The summative assessment will utilize the Kirkpatrick four-level model to evaluate the

effectiveness of the coursework provided. This plan will outline the instruments that will be

used after the implementation of the training that is focused on improving communication in

online learning in order to increase retention. The course objectives that will be evaluated in the

Kirkpatrick model are as follows:

Instructors will understand and describe the basic principles of communication for online

coursework.

o Type of communication

o Importance of communication

o Synchronous verses asynchronous

Instructors will analyze and evaluate best practices in online communication and apply these to

real world products.

o Improved student satisfaction through communication

o Establishing reasons to revisit the course site

o Communication focused lesson plans

Instructors will prepare students in their courses for online learning.

o Accessibility

o Clear expectations and technical guidelines

The goal of the summative evaluation plan is to benchmark the following:

Satisfaction of the participants

Learning take-aways


Behavioral changes in course design

Positive effects on the retention rates of online students resulting from the training

Methods

Before beginning the online instruction modules, students will complete a pre-test to

determine their baseline level of knowledge related to the topic of communication in the online

medium (see Appendix G). At the conclusion of the entire online instruction modules, students

will once again take the described pre-test as a post-test to assess the impact of instruction as

indicated in level two of Kirkpatrick’s four levels of evaluationAlso, to address the level one

issue of student reactions, after completing the course, students will complete a course evaluation

(see Appendix F) that allows them the opportunity to rate their experience in the course and also

the effectiveness of the content and delivery medium.

The level three summative evaluation tool (see Appendix H) is designed to be given to senior

faculty in supervisorial roles. This evaluation utilizes a Likert scale and points will be assigned

to the scale and an overall rating for each instructor’s course will be evaluated. Scoring on the

Likert scale is as follows:

Strongly disagree = -2

Somewhat disagree = -1

Neutral = 0

Somewhat agree = 1

Strongly agree = 2

Scores greater than zero will be viewed as favorable and the possible correlation between

lower scores and retention rates will be investigated further. Additional communication

enhancing instruction will be recommended for lower scoring courses. The level four summative


evaluation tool (see Appendix I) included in our plan is a trend analysis of retention rates before

and after the introduction of our communication training. Data gathered over time will likely

draw a correlation between retention rates and our training course. This will provide

management with supportive data to implement future similar programs or expansions to the

existing program.


References

Morrsison, G. R., Ross, S.M., Kalman, H.K., & Kemp, J.E. (2011). Designing effective

instruction. New Jersey: John Wiley & Sons, Inc.

Rothwell, W.J., & Kazanas, H.C. (1998). Mastering the instructional design process. San

Francisco: Josey-Bass/Pfeiffer.


Appendix A

Formative Evaluation Instrument

Answer each question by placing an “X” next to yes, no, or not applicable. Please provide any additional comments that you feel will enhance the effectiveness of the instruction.

Question Yes No Not Applicable

Comments

1. Do instructional materials clearly state the performance objectives expected?

2. Do the instructional materials appear to match the learner characteristics?

3. Do the instructional materials support the performance objectives?

4. Is the content of the instruction complete?

5. Is the content of the instruction up to date?

6. Are learners given adequate opportunities to receive information about the content?

7. Are learners given adequate opportunities to practice or apply what they learn?

8. Are learners given adequate opportunities to receive feedback on how well they applied what they learned?

9. Are the instructional methods appropriate for the targeted learners?

10. Do the instructional methods support the performance objectives?

11. Are there other issues that you noticed that could improve the instruction? If yes please describe in the comments box.

Source: Rothwell, W.J., & Kazanas, H.C. (1998). Mastering the instructional design process. San Francisco: Josey-Bass/Pfeiffer.


Appendix B

The document below represents the instructional design planning sheet for learning episode three (module 3) from the instructional design plan. The performance objectives are included at the end of this document.

Learning Episode #3 (Module 3)

Terminal Objective 3 - Instructors will analyze and evaluate best practices in online communication and apply these practices to real world products.

Design Sequence Description Objective(s) Time Instructional

Strategy

Media

(Technology, Worksheets, etc.)

Gain Attention Pre-instructional strategy

3.1

3.2

3.3

3.4

3 minute Participants will view a Camtasia screencast that will display and orally present statistics and literature findings concerning the correlation between effective communication in online courses and the retention of online students. This information is designed to enlighten participants on the importance of involvement and communication in online courses. The following information will be displayed:

In a survey of 1,406 higher education students, Drouin (2008) discovered that students reported higher levels of satisfaction and sense of community membership when there was increased communication with the instructor and with other students enrolled in the course.

These findings were supported by Harrell (2008) who found that one of the key factors causing online graduate students to drop out was a sense of isolation.

Camtasia screencast


Inform Learners of Objectives

Pre-instructional strategy

3.1

3.2

3.3

3.4

5 minutes Using the Camtasia screencast and accompanying PDF handout participants will be provided with a written and oral summary of the importance of self-reflection upon the level of satisfaction that their own online students have experienced. Also, an overview of the process for evaluating existing lesson plans in order to incorporate opportunities for enhanced communication will be provided.

Camtasia screencast, PDF attachment/handout

Stimulate Recall of Prior Learning

Review Terminal objective 1

5 minutes The Camtasia screencast and accompanying PDF handout will continue and will briefly review the following concepts from previous modules: basics in communication for online coursework, advantages in communication for online courses and its importance, types of communication common in online courses, asynchronous communication, and synchronous communication.


Present Stimulus Material

Recall 3.1 5 minutes The Camtasia screencast will continue and will visually and orally explain to participants Herbert’s (2006) survey concerning online student satisfaction and retention.

Camtasia screencast

Provide Learner Guidance

Recall 3.1 N/A The Camtasia screencast explaining Herbert’s (2006) survey will be accompanied by a PDF file that provides participants with a copy of the survey that can be downloaded or printed.

PDF attachment/handout

Elicit Performance

Elaboration 3.2 10 minutes Participants will take Herbert’s online student satisfaction survey from the point of view that they are a student in an online course that they currently teach. Participants will be asked to answer each question as honestly as possible. The survey will be provided in the module as a PDF file download.


Provide Feedback Integration 3.2 5 minutes Participants will post questions to a discussion board created for the online instructional module. The questions will be based upon the results of the satisfaction survey and the practical implications of the survey to the teaching of online courses. The instructional design team will address these questions.

Discussion Board


Present Stimulus Material

Recall 3.3 15 minutes The Camtasia screencast will continue with an explanation of the elements needed to create lesson plans with effective communication in online courses. Participants will be provided with a PDF document that outlines these elements and also gives an effective lesson plan template.


Provide Learner Guidance

Recall 3.3 N/A Participants will be provided with a PDF document that gives several examples of lesson plans that utilize best practices in online communication.


Elicit Performance

Elaboration 3.4 45 minutes Participants will use the lesson template provided in the Camtasia screencast lecture to evaluate and revise a personally created existing lesson plan to incorporate opportunities for enhanced communication.


Provide Feedback Recall 3.3 5 minutes Participants will post questions to a discussion board created for the online instructional module concerning the most effective methods for creating lesson plans that maximize communication. The instructional design team will address these questions.

Discussion Board

Assess Performance

Elaboration 3.1

3.2

3.3

3.4

10 minutes

Participants will have two weeks to offer feedback to other participants.

Participants will post responses to a discussion board created for the online instructional module concerning how they plan to improve communication in the areas in which they were weak on the survey. In addition, participants will post the revised lesson plan for feedback and suggestions from other participants and instructors.

Discussion Board


Enhance Retention and Transfer

Recall 3.1

3.2

3.3

3.4

N/A Participants will be provided with a quick reference PDF sheet that summarizes the assessment of online communication and the elements needed for an effective lesson plan.


Learning Objectives

Terminal Objective 3

3.0 Instructors will analyze and evaluate best practices in online communication and apply these practices to real world products.

3.1 Instructors will recall the importance of student involvement and student satisfaction regarding online course experiences.

3.2 Instructors will evaluate through self-reflection the level of involvement and satisfaction previous students have expressed regarding online course experiences.

3.3 Instructors will identify ways in which lesson plans can provide opportunities for enhanced communication in online courses.

3.4 Instructors will evaluate and revise existing lesson plans to incorporate opportunities for enhanced communication.

3.5 Instructors will identify meaningful opportunities for students to visit the online course’s site on a regular basis and the tools to provide these opportunities.


Appendix C

Subject Matter Expert Review #1 of Learning Episode

Formative Evaluation Instrument Answer each question by placing an “X” next to yes, no, or not applicable. Please provide any additional comments that you feel will enhance the effectiveness of the instruction.


Comments


x


x I needed more info. about who will be completing the course to answer this question.


x


x


x


x The time listed for “lecture type” material seems a little short to explain the information.


x


x


x


x

11. Are there other issues that you noticed that could improve the instruction? If yes please describe in the comments box.

X Forty-five minutes does not seem long enough for someone to completely revise a lesson plan.



Appendix D




Comments


X


X Don’t know the learner characteristics


X


X


X


X


X I only put no because it will take most people longer than ten minutes to take the survey, and longer than 45 min. to revise a lesson plan.


X I like that both instructors and other participants will provide feedback.


X


X

11. Are there other issues that you noticed that could improve the instruction? If yes, please describe in the comments box.

X I really like the activities where participants get to interact with one another. It would be valuable to work more of this interaction into the larger plan.



Appendix E




Comments


x


x


x


x It’s complete for the given module but I did not review the entire product.


x


x


x The learners would benefit from revising more than one lesson plan; perhaps they could revise a whole unit or topic of a course.


x


x


x

11. Are there other issues that you noticed that could improve the instruction? If yes, please describe in the comments box.


FORMATIVE AND SUMMATIVE EVALUATION REPORT: THE RETENTION CHALLENGE OF ONLINE STUDENTS 18

Appendix F

Level I Summative Evaluation Tool

Incorporating Communication into Online Course Design - Course Evaluation

Directions: Using the Likert scale provided, please rate the design of the online course in

regards to communication. In the comment section please include any ideas that could help

improve clarity of expectations or communication aspects of the course.

Course Evaluation

Strongly Agree

Somewhat Agree

Neutral Somewhat Disagree

Strongly Disagree

1. The information presented was relevant to my job as an online instructor.

2. The course provided useful information on communication in online courses.

3. I feel more comfortable communicating with my online students.

4. The course objectives were clear.

5. The instructional materials supported my learning.

6. I would recommend this course to a friend.

7. The pace of the course lessons was appropriate.

8. I felt like I had the required background for the content covered in this course.


9. The course achieved its desired

objectives.

10. I feel confident I can incorporate the skills learned into my teaching practice.

What did you like best about the course?___________________________________________________-_____________________________________________________________________________________

_____________________________________________________________________________________

_____________________________________________________________________________________

What would you like to see changed about the course? _______________________________________-_____________________________________________________________________________________

_____________________________________________________________________________________

_____________________________________________________________________________________

Will you use any of the communication techniques in your courses? Which ones? __________________ _____________________________________________________________________________________-_____________________________________________________________________________________

_____________________________________________________________________________________

Please write any additional comments regarding the course. ____________________________________ _____________________________________________________________________________________-_____________________________________________________________________________________

_____________________________________________________________________________________


Appendix G

Level II Summative Evaluation Tool

Incorporating Communication into Online Course Design - Pre-Test / Post-Test

Corresponding Objective

Pre-Test / Post-Test Question

All Communication in online courses should be handled in exactly the same way as face to face courses.

True False

1.5 Provide two examples of asynchronous communication used in online courses:

_____________________________ _____________________________

1.5 Provide two examples of synchronous communication used in online courses:

_____________________________ _____________________________

3.2 Name three important factors in student retention in online courses

_____________________________ _____________________________ _____________________________

3.5 List three examples of meaningful reasons in which a student will visit an online course platform:

_____________________________ _____________________________ _____________________________

2.5 Email replies to students in online courses should be completed within this time frame:

12 Hours 24 Hours 3 Days 2 Weeks


2.4 Tone is always inferred uniformly in written communication, such as online course

content.

True False


Appendix H

Level III Summative Evaluation Tool

Incorporating Communication into Online Course Design Evaluation Form

Directions: Using the Likert scale provided, please rate the design of the online course in

regards to communication. In the comment section please include any ideas that could help

improve clarity of expectations or communication aspects of the course.

Objective 1: Instructors will understand the basics in communication for online coursework.

Strongly Agree

Somewhat Agree


Strongly Disagree

1. Communication features are readily accessible in the course.

2. Communication features of the course are simple and easy to use.

3. The course design clearly outlines and identifies the expectations on when to use the different types of communication mediums.

4. The course makes good usage of asynchronous communication.

5. The course makes good usage of synchronous communication.

6. Communication was a clear fundamental consideration in the design of this course.

Comments: __________________________________________________________________________

____________________________________________________________________________________

____________________________________________________________________________________


Objective 3: Instructors will implement best practices in online communication.

Strongly Agree

Somewhat Agree


Strongly Disagree

7. Substantial communication enhancements exist in this course design compared to previous versions of this course or most other online course experiences.

8. Communication between the instructor and student is the key focus of this course compared to previous versions of this course or most other online course experiences.

9. Increased student to student communication was a key focus compared to previous versions of this course or most other online course experiences.

10. Lesson plans have clear communication objectives built into them.

11. Reoccurring student visits to the course site are more likely compared to previous versions of this course or most other online course experiences.

12. An increase in student satisfaction was perceived due to a focus on communication.

Comments: __________________________________________________________________________

____________________________________________________________________________________

____________________________________________________________________________________

Objective 2: Instructors will prepare students in their courses for online learning, and make themselves available, communicate technical guidelines and clear expectations.

Strongly Agree

Somewhat Agree


Strongly Disagree


13. Course design promotes instructor

availability and willingness to help.

14. Virtual office hours or other arrangements for one on one communication are clearly outlined.

15. Reasonable response time for online correspondence is clearly indicated.

16. Technical assistance and/or proper links to contacts are available.

17. Technical guidelines for the course are accessible and clear.

18. The course has clearly stated centrally located expectations set for the students.

Comments: __________________________________________________________________________

____________________________________________________________________________________

____________________________________________________________________________________


Appendix I

Level IV Summative Evaluation Tool

Incorporating Communication into Online Course Design Evaluation Form

Directions:

Step 1: Gather Data

Data will need to be gathered from online course registration data and will need to be compiled

into a summary table that looks like the following:

Year/Semester Instructor Retention Rate Course completion

2011/Fall Dr. Bob 78% N

2011/Summer Dr. Bob 85% Y

2011/Spring Dr. Bob 82% N

2010/Fall Dr. Bob 74% N

2011/Fall Dr. Benson 100% Y

2011/Summer Dr. Benson 99% Y

The data will need to be filtered to include only those professors that not only attended the

course but that scored above a zero on the Level III evaluations showing that they were able to

successfully incorporate communication enhancements into their course design behaviors.

Step 2


Continued data collection over time for several semesters after the training course will allow a

proper assessment of the trending. A minimum of three data points, pre and post training, will be

required, but collecting more data over time is suggested.

Step 3

Compile a trend analysis to see if retention rates increase. This will be done for instructors who

have positively altered their behavior as it relates to communication in online courses. This will

be submitted to management as part of an impact report and will look like the following:

FORMATIVE AND SUMMATIVE EVALUATION REPORTS: THE … · FORMATIVE AND SUMMATIVE EVALUATION REPORTS:...

Documents

Transcript of FORMATIVE AND SUMMATIVE EVALUATION REPORTS: THE … · FORMATIVE AND SUMMATIVE EVALUATION REPORTS:...