Blended Learning ReportEvidence of Learning Gains

AZUSA PACIFIC UNIVERSITY

Office of Innovative Teaching & Technology in the Center for Teaching, Learning, & Assessment

May 2015

Blended Learning Research Team

i

Dr. Ann Kwinn, Director of Instructional Strategy, Office of Innovative Teaching and Technology

Dr. Laurie Schreiner, Chair and Professor, Department of Higher Education

Dr. Mike Truong, Executive Director, Office of Innovative Teaching and Technology

Executive Summary

ii

“I really felt like [the blended course] was overall very effective in keeping [students] up with the reading and the online materials - way more so than my teaching the class in the traditional way.”

-Faculty Participant

Technology affords learning in ways unconstrained by time and place.  As more education happens online, institutions that do not develop a purposeful approach to blended learning that integrates classroom-based and online instruction will fail to attract the next wave of students.

Three APU faculty were selected for this study, based on their ability to teach the same course in face to face and online formats in consecutive semesters to sufficient numbers of undergraduates.  The data showed that students’ academic performance was significantly greater in two of the three blended learning sections.  In other words, students seemed to learn just as much, and in two of three sections they learned more, than in the traditional class format.  However, when we examined students’ perceptions of their learning gains, it was clear that students did not believe they had learned more.

The BLP research team recommends that APU proceed with caution to implement blended learning courses in the undergraduate and graduate curricula.  We recommend that the approach be introduced over the course of an academic year and offered to a limited number of faculty on a voluntary basis at first, with sufficient training and resources provided for the experience to result in positive learning outcomes for students.

iii

Chapter 1

An Overview of Blended Learning

APU defines blended learning as combining in-class instructional activities with online or fieldwork activities that can be either asynchronous or synchronous, resulting in a percentage reduction of in-class time

Although considerable variations exist across different institutional contexts, blended learning typically includes a combination of face-to-face (F2F) and online learning (Graham, Woodfield, & Harrison, 2013).  Also known as hybrid learning, blended learning represents applying active pedagogies to both the face-to-face and online learning environments.   At APU, we prefer the term “blended learning” in order to designate that these best practices from both types of learning environments are being interwoven to produce the best possible learning outcomes for a particular course.  We define blended learning at APU as combining in-class instructional activities with online or fieldwork activities that can be either asynchronous or synchronous, resulting in a percentage reduction of in-class time.

It is important to note that blended classes are different from flipped classes; the former reduces F2F time and the latter simply shifts lectures/instruction to an online environment without any reduction in F2F time. As Figure 1.1 illustrates, there are many ways to blend a course.

For example, in the fixed model, a traditional course that meets F2F Tuesdays and Thursdays every week might reduce the F2F meeting to just Tuesdays, and students spent Thursdays online or in the field.  In the flex model, the F2F meetings are determined by the faculty, allowing them to have greater freedom and creativity to incorporate fieldwork, internship, and other outside classroom experiences.  In the example given of the flex model, the F2F meetings at the beginning and the end of the semester serve as bookends.  In the hyflex model, the typical online class is supplemented by required F2F meetings.  For example, a student in the hyflex model will be required to see their faculty three times during the term, during which they will get mentoring, coaching, and feedback.  

There is no ideal blended learning model.  The research on blended learning indicates that institutional context is an important consideration in selecting a particular blended learning model; thus, this report makes recommendations specific to APU and its cultural context and resources available for blended learning.

Section 1

Definition

5

6

Figure 1.1 Blended Class Models

Definition Example

Fixed Model

Flex Model

Hyflex Model

Class rotates on a fixed schedule.

Class meets Tuesdays F2F and Thursdays online or in the

field.

Class rotates on a flexible schedule determined by

faculty.

For weeks 1-2 and 14-15, class meets F2F.  The rest of the weeks are online or in the

field.

An online class with required F2F

meetings.

Students are required to meet F2F three times during the

term but complete their coursework online or in the

field.

In recent years blended learning has gained prominence among academic leaders, teachers, and researchers as a strategic response to the need for increased access and reduced cost of higher education without a significant reduction in learning outcomes.  For presidents, provosts, and deans, the interest in blended learning tends to be a response to some of the significant challenges facing higher education.  According to Selingo (2014) in The Innovative University, over 90% of the college presidents surveyed (N=350) are engaging in blended and online learning initiatives at their campus with the aim to increase access, reduce cost, and improve learning.  In Allen and Seaman’s (2014) report based on survey responses from 5,000 colleges and universities, academic leaders believe blended and online learning is critical to their institution’s long-term growth strategy.  Moreover, they are far more positive about the learning outcomes for blended instruction than they are for online learning.  In other words, most academic leaders believe that students perform just as well, if not better, in blended courses, compared to either online or F2F courses (Allen and Seaman, 2014).

The interest in blended learning among university faculty is more often the result of desiring to incorporate technology into their course or create greater learning flexibility. According to Stein and Graham (2014), blended learning has gained popularity among instructors and learners because it offers the best of all worlds: rich interactions in F2F settings and an efficient, convenient, and self-paced learning environment online.  Shifting some of the learning to an online or field environment enables learners in blended courses to have greater flexibility in their schedule, more opportunities for practice and review via online tools, and expanded access to learn beyond the confines of the traditional classroom. For instructors who are skeptical about the validity and value of online learning, blended courses represent an easily implemented first step toward the use of technology in higher education.  Instructors might begin by flipping their class, recording their lectures and posting them online for students to review before class.  During class, the time is focused on problem-solving, interaction, and discussion.  Once faculty are comfortable flipping their classes, they might consider shifting some of the time spent in F2F learning to either an online or fieldwork environment.  This slow and gradual move toward

Section 2

Trends

7

blended learning takes advantage of technology efficiencies without asking instructors to compromise their established pedagogy or to sacrifice the valued F2F interactions.

The emergence of blended learning has led to increased research on best practices, as well as strategies for implementation and adoption.  A meta-analysis found that there were 1,132 studies published between 1996 and 2008 focusing on blended and online learning (Means et al., 2010).   A 2011 literature review found close to 200 dissertations and hundreds of journal articles on the topic (Graham et al., 2013).  Collectively, these trends point to a widespread acceptance and maturation of blended learning as a sound and promising strategy for institutions of higher learning.  

8

Chapter 2

Benefits and Challenges of Blended Learning

Blended courses present both benefits and challenges. This chapter will explore how blended courses impact course flexibility, the student learning experience, actual learning outcomes, course development, and classroom space.

Blended courses have the potential to increase convenience, access, and opportunities for all learners.  Today’s traditional and nontraditional students value flexibility as a factor when registering for their classes.  With work, co-curricular activities, and other responsibilities, increasing numbers of students are attracted to blended and online courses for increased flexibility (Allen & Seaman, 2014). According to a recent report, more than four in five students (85%) took at least one blended course in 2014, up from 79% in 2013 (Dahlstrom & Bichsel, 2014). Convenience and flexibility has been and will likely remain an important benefit leading to the steady growth of blended courses in higher education (Collins and Halverson, 2009).

The challenge of providing such flexibility lies in the planning, coordination, and assignment of classroom space.  Institutions offering courses in various modalities, including blended format, need the cooperation of informed registrars to address this challenge.  Additionally, faculty and students new to the blended format might initially experience confusion.

Section 1

Increased Flexibility

10

In addition to increasing access and offering schedule flexibility, shifting the lecture components of a traditional course to an online context can free up precious time for more in-class interactions, problem-solving, and other active learning strategies that Braxton, Jones, Hirschy, and Hartley (2008) have noted promote greater learning, but also lead to increased student persistence to graduation.  When the Learning Management System (LMS) and accompanying technologies are appropriately supportive of student learning, students feel empowered to take greater responsibility for their own learning and are able to experience learning in a contemporary, relevant, and engaging manner (Wright, Lopes, Montgomerie, Reju, & Schmoller, 2014).

The challenge of blended learning, however, is in supplying the agile technology, including a modern, full-featured LMS, that is necessary to appropriately adapt to the art of teaching, rather than expecting faculty to adapt to a particular technology (Abel, Brown, & Suess, 2013).  Moreover, utilizing reliable and robust technology is crucial to ensure that the online experience of

blended courses remains high-touch, fostering rich engagement, meaningful interactions, and relevant learning.

Section 2

Improved Learning Experience

11

In well-designed research studies that appropriately differentiate blended learning from purely online learning, researchers have found either no significant difference in the learning outcomes of blended courses and other learning modalities or have found that students experienced greater learning outcomes in a blended format.  For example, a meta-analysis conducted by the U.S. Department of Education found that students who took their classes in a blended format performed significantly better, on average, relative to those taking purely face-to-face instruction or purely online instruction (Means et al., 2010).  Other studies have found that, unlike in fully online courses, students in the blended format “pay no price” for this mode of instruction in terms of pass rate, final exam scores, faculty contact, and overall performance (Bowen, Chingos, Lack, & Nygren, 2012).  In a study where blended courses utilized free open online courses already available to students, students in the blended sections fared as well or slightly better than students in traditional sections in terms of pass rates, scores on common assessments, and grades (Griffiths, Chingos, Mulhern, & Spies, 2014).  These findings have been replicated across diverse student populations, such as academically at-risk students,

those from low-income families, underrepresented minorities, first-generation college students, and those with weaker academic preparation.  In short, student learning outcomes in a blended format appear to be equivalent to traditional F2F format.

The challenge remains that to consistently achieve these learning outcomes, blended courses cannot be a simple “digital facelift” of traditional F2F courses (Means et al., 2010; Stein & Graham, 2014).  In other words, simply putting an existing F2F course online will not achieve the same results.  Instead, institutions need to invest in course development.  In particular, a team of instructional designers, media specialists, and faculty need to work together to create engaging and transformative blended courses through an intentional course redesign process, incorporating additional learning approaches that are afforded with the new modality.

Section 3

Same or Better Learning Outcomes

12

Looking for ways to cut course development costs, some universities are using Massively Open Online Courses (MOOCs) and other open online courses to seed their blended courses.  For example, instead of spending money to redesign their courses for blended delivery, the University System of Maryland (USM) employed free Coursera courses, Carnegie Mellon University’s Open Learning Initiative (OLI) courses, and Pearson courses in a variety of subject areas on different campuses (Griffiths, Chingos, Mulhern, & Spies, 2014).  

The challenge of using open resources is that they often incur a hidden cost.  For example, because many MOOCs were not initially designed to be used on campus, it requires additional effort to make the technology systems, intellectual property rights, and the online content integrate coherently in a campus environment.

Section 4

Reduced Costs of Course Development

13

In recent years many institutions of higher education have explored using blended and online courses to improve operational efficiency and effectiveness.  In particular, institutions have used blended courses as a strategy to shift some in-class sessions online, optimizing existing classroom space to accommodate more sections and students.  For example, the University of Central Florida (UCF) leverages blended courses, along with online and video courses, to address its limited classroom space and distributed campus (Dziuban, Moskal, & Hartman, 2005).  Since the inception of its blended learning initiative in 1997, UCF has seen a dramatic increase in the number of students taking blended courses.  In 2010 about half of its 63,000 students enrolled in at least one or more blended, online, or video courses.  UCF’s blended courses consistently rank higher than other modes in student course evaluations and have the highest levels of student success and the lowest withdrawals of any modality -- including face-to-face. Furthermore, faculty surveyed after teaching blended courses expressed a clear willingness to continue teaching in that modality.  Other institutions, such as the University of Wisconsin at Milwaukee, Brigham Young University, and others, have

effectively utilized the blended learning model to increase operational capacity (Aycock, Granham, & Kaleta, 2002; Graham, Woodfield, & Harrison, 2013).  

The challenge of expecting a blended learning format to save classroom space is two-fold.  One is that classroom space allocation and scheduling becomes more complex and requires more careful forethought and planning.  The second challenge is that the changing demographics of higher education project declining enrollments into the foreseeable future, as there are fewer high school graduates (NCES, 2011).  Thus, shifting to a blended learning format to save classroom space is a long-term solution to a temporary problem.  Such a shift would be wise only if there are other benefits to be gained from moving to that learning modality.

Section 5

Classroom Space Saving

14

Chapter 3

Characteristics of Effective Blended Courses

Based on a review of studies of  blended learning in higher education, several design elements that contribute to student success in blended courses were identified.  These elements include: 1) Learner control; 2) Learner reflection; 3) Time on task; 4) Collaboration; 5) Social presence; and 6) Motivation.  Each of these element will be reviewed below.

Courses that give students more power over what they learn and how fast they learn it, what Merrill (1984) calls learner control, are associated with better student outcomes.

Online courses by nature typically provide higher learner control compared to a traditional lecture and even other expository learning experiences, such as video or text. Learners have more control in activities such as online access of content and participation in forums and simulations.

Zhang et al (2006) found student learning with video was improved when learners could control (stop and start) the video and view content in their preferred order.  Other ways students can exert control include viewing a segment a second time or skipping material with which they are already familiar.

Section 1

Learner Control

16

Any form of self study, including the online component of a blended course, gives students the quiet space that is optimal for reflection.  Students no longer surrounded by their peers in the physical classroom can process ideas in a more personal way, making meaning of the material.  While self reflection refers to meditation or serious thought about one's character, actions, and motives, learner reflection includes many forms of cognition about educational material.

For example, Chung, Chung and Severance (1999) found that subjects who were prompted by the computer to engage in self-explanation and self-monitoring in order to integrate lecture concepts into writing assignments performed better on the writing task than did a group without the prompts.  Saito and Miwa’s (2007) research subjects were asked to visualize and compare their web searching processes with those of other learners. These reflective activities led to better learning.

Section 2

Learner Reflection

17

In addition to the potential for greater learner control and opportunity for reflection, adding an online component to a course has been found to increase students’ time on educational task (Means et. al, 2010).  Although Kohn (2007) refers to time on task as a necessary, but not sufficient, condition for learning, Chickering and Gamson (1987) list increasing time on task as one of the seven principles of effective undergraduate instruction.

Students incentivized by points for taking quizzes on weekly readings may be more likely to complete these readings, increasing the time on the task of reading, which allows for more in depth discussion and prepared work on projects during class time.  Lewis (2002) found that students who completed online quizzes were more successful in online forums.

Online forums increase time on task by requiring all students to actively participate, compared to face-to-face discussions in which typically only a portion of the students either choose to or have time to participate.

A caveat on time on task, however, is that a blended course should not be “class plus,” meaning students should not feel they are taking two courses - a face to face and an online one - based on the work required  (Dziuban, Moskal, and Futch, 2007; Woods, Badzinski, and Baker, 2007).

Section 3

Time on Task

18

The notion of online and blended learning may trigger concerns over a loss of student interaction with faculty and other students.  In reality, computers can be leveraged to foster collaboration. Johnson, D.W. & Johnson, R.T. (2004) reported that computer-supported collaborative learning promoted greater mastery of factual information and ability to use this  information  in problem-solving than did the individual condition.  And Means et al’s ( 2010) meta study reported larger effect sizes for studies in which the online instruction was collaborative or instructor-directed than in those studies where online learners worked independently.  Further, Lim & Yoon (2008) found team collaboration to be higher in a blended versus online group.

Collaboration is recommended and readily available in a blended format. The “affordances” or capabilities of online environments provide many pathways to collaboration.  For example, a typical academic LMS allows for forum postings and responses, messaging, chat, and blogging.  Adding other tools such as web conferencing or Tweeting provides additional opportunities for collaboration.

Section 4

Collaboration

19

The benefit of collaboration may support a social construction of knowledge and recommend that faculty create among students a sense of “social presence” which is defined as the sense of being with another (Biocca, Harms & Burgoon, 2003), the degree to which a person is perceived as a real person (Gunawardena, 1995), and a student’s sense of belonging in a course via their ability to interact with other students and the instructor (Picciano, 2002). Learning environments with social presence are sociable, warm, and personal.

So and Brush (2008) looked at the relationship between students’ perceived levels of collaborative learning, social presence, and overall satisfaction in a blended learning environment. The result was that students who perceived high levels of collaborative learning and high levels of social presence tended to be more satisfied with their course. Hostetter & Busch (2006) and Richardson and Swan (2003) found similar results.  Picciano (2002) found a positive correlation between the level of students’ perceptions of social presence in their courses and higher results on learning measures.

Section 5

Social Presence

20

While dropout rates are higher for online courses, compared to  face to face courses (Diaz, 2000), faculty in blended courses can improve student retention by increasing student motivation.  Park, J.H., & Choi, H. J. (2009) found that for adult learners, family support, organizational support, satisfaction, and relevance in addition to individual characteristics, are able to predict learners’ decision to drop out or persist.  The results imply that dropout rates can be improved if online program developers and instructors find ways to enhance the relevance of a course.

King and Arnold (2012) reported that motivation, communication, and course design are three factors that contribute to the overall success of blended learning courses and students’ satisfaction with blended learning courses.  

Atkinson (1957) originally developed the Expectancy-Value Theory of Motivation which proposes that the amount of effort that individuals will put into an activity is a product of the degree to which they expect to successfully perform the activity (expectancy) and the amount of value they place in the rewards

and the opportunity to perform the activity (value).  If either element is missing, the student may not invest effort.

According to the expectancy-value model, instructors can increase motivation by showing the value of the learning and helping students have optimistic expectations about their ability to succeed. In short, faculty who set clear goals, provide immediate, continuous and relevant feedback, and create a reasonable level of challenge, position their students for success (King and Arnold, 2012; Wlodkowski, 2008).

Self-determination theory (Deci & Ryan, 1985, 2002) proposes that activities that promote intrinsic motivation satisfy three innate needs: competence (seeking to control the outcome and experience mastery), autonomy (self-determination in deciding what to do and how to do it), and relatedness (affiliation with other through social relationships).  Competence here is related to Atkinson’s expectancy of success, while autonomy and relatedness bring us full circle from the discussion of learner control and collaboration for social presence.

Section 6

Motivation

21

Chapter 4

The APU Blended Learning Project

Three faculty were selected for this pilot study, based on their ability to teach the same course in face to face and blended formats in consecutive semesters to sufficient numbers of undergraduates and their willingness to comply with the expectations of the project.

In the fall of 2013 the Provost announced the opportunity for faculty who were interested in adopting blended learning to apply for the Blended Learning Project (BLP).  Faculty applications were reviewed by a team of instructional designers and Faculty Fellows from the Center for Teaching, Learning, and Assessment (CTLA).  Three faculty were selected for this pilot study, based on their ability to teach the same course in face to face and blended formats in consecutive semesters to sufficient numbers of undergraduates and their willingness to comply with the expectations of the project.

The three faculty were females who taught in the School of Behavioral and Applied Sciences and the College of Liberal Arts and Sciences.  Although their race, age, and faculty rank varied, all were above-average instructors, as indicated by a historical pattern of IDEA course evaluation T-scores above 50 on “excellent instructor.”  All taught undergraduate sections of courses where the typical enrollment was more than 25 students.

These three faculty provided informed consent to participate in the study, which included an assessment of their course

evaluations, student learning outcomes, student engagement, and satisfaction, as well as a one-on-one interview, participation in a focus group, and keeping a log of the amount of time spent in each of their preparation and teaching activities.  Students in their classes also provided informed consent prior to supplying their feedback, and gave permission for the researchers to access their exam grades and GPAs.

This quasi-experimental study used a recurrent institutional cycle design, with students in the traditional format of the class in Spring 2014 serving as the control group for the students in the blended format of the same class the following semester (Fall 2014).  Each instructor served as her own control; thus, the results described below are disaggregated by instructor as well as aggregated across instructors.

Prior to the traditional class format taught in Spring 2014, all students in the section completed a pretest of their knowledge of the course content.  This pretest served as a covariate in the analyses, so that any differences in learning outcomes between the two formats takes into consideration the pre-existing

Section 1

Procedure and Participants

23

differences in knowledge and ability between the two groups.  Students also completed an assessment of their engaged learning (The Engaged Learning Index; Schreiner & Louis, 2011) the first day of class, so that any differences in learning engagement at the end of the course could be moderated by students’ pre-existing levels of engagement.  Each instructor taught the class the way she had been teaching it in a traditional format, making no changes to the course.  However, during the Spring term these instructors were also attending workshops and gaining one-on-one assistance in preparing to move the class format to a blended approach.  At the end of the term, all students completed the Engaged Learning Index again and also completed an assessment of their satisfaction with the course.  IDEA course evaluations were obtained for each faculty’s section, along with the final exam grades of the students in the section.

During the summer of 2014, additional assistance and resources were provided to the instructors as they prepared to shift their course to a blended learning format.  Instructors also had the opportunity to meet together to discuss the approaches they planned to use.  Each instructor was given the freedom to determine how much class content shifted online, the extent to which fieldwork replaced class time, and the extent to which technology was utilized and in what way throughout the course.  In other words, there was not a uniform approach to implementing the blended learning format; the research team believed that faculty autonomy in course design was a key

element for success.  As a result, there was considerable variation across the three faculty in terms of the extent to which technology was effectively utilized, as well as what types of technology and teaching practices were used.

In Fall 2014 these three faculty taught the same course, but used their self-designed blended learning format to do so.  The same procedure was followed with the students in these sections: on the first day of class, they completed a pretest of course content knowledge and the Engaged Learning Index.  At the end of the course, they again completed the Engaged Learning Index as well as a measure of course satisfaction, and researchers obtained their final exam scores and the instructors’ IDEA course evaluations.

The qualitative portion of this study consisted of the faculty subjects’ participating in a focus group based on one set of questions about their experiences in the project and then individual interviews utilizing a second set of questions.  All participants were asked the same set of questions, but the interviewer also probed in some cases to gain clarification or greater detail.  Each interview took less than an hour. The audio from the sessions was recorded and transcribed. The questionnaires are in the appendices of this report.  Themes were identified to describe the impressions of faculty and students in the blended course environment.

24

Two phases of data analysis comprise this study: (1) a quantitative analysis of the exam scores, Engaged Learning Index scores, course satisfaction, and IDEA course evaluation scores, and (2) a qualitative analysis of the individual faculty interviews, focus groups, and time logs.  Each phase will be described separately.

A multivariate analysis of covariance was conducted to assess the differences in students’ engaged learning and academic performance, as measured by final exam score.  In both cases, the pretests were used as covariates in the analysis so that differences across the two sections took into consideration students’ pre-existing levels of knowledge and engagement in learning.

Students’ academic performance, as measured by scores on the final exams after controlling for their pre-existing level of knowledge in the course, was significantly greater in two of the three blended learning sections (p < .007 across all sections).  This finding seems to indicate that students learned just as much, and in two of three sections they learned more, than in the

traditional class format.  However, when we examined students’ perceptions of their learning gains, as measured by the “progress on relevant objectives” section of the IDEA course evaluations, it was clear that students did not believe they had learned more.  Using the specific “progress on relevant objectives” items from the IDEA course evaluations, an independent samples t-test

indicated that in two of the three sections, students’ ratings of

Section 2

Quantitative Analysis & Findings

25

** p < .01

Table 1 Mean IDEA Non-adjusted T-scores for Progress on Relevant Objectives

Instructor Progress on Relevant Objectives IDEA T-scores (Mean = 50, SD = 10)

Progress on Relevant Objectives IDEA T-scores (Mean = 50, SD = 10)

Traditional Blended

1 52 52

2 65 61

3 59 49**

their perceived progress on the outcomes for the course were significantly lower in the blended sections than in the traditional F2F sections.  However, when the global IDEA non-adjusted T-scores were analyzed, only one of these sections indicated a statistically significant difference, with lower scores in the blended section (see Table 1).

Because IDEA course evaluations are a significant component for consideration in faculty’s opportunities for rank promotion and extended contract, these findings are of concern.  Students appeared to learn as much, if not more, in the blended format, but did not think they had.  One potential source of this incongruency is that learning is measured differently: final exam scores are direct indicators of knowledge acquisition and application.  The Progress on Relevant Objectives scores are indirect indicators of learning, as they consist of student perceptions that they experienced a gain in specific abilities identified as important by the instructor,

such as “learning fundamental principles and theories,” “learning how to find and use resources for answering questions,” and “acquiring an interest in learning more by asking my own questions” -- three of the specific areas where students in the blended learning sections scored lower.  

Students’ IDEA ratings of the instructor were mixed, with one instructor achieving significantly better ratings in the blended

section and the other two with no significant difference in their ratings.The overall rating of the course was significantly lower in the blended learning sections (see Table 2).  

An independent samples t-test analysis indicated that the faith integration scores on the IDEA instrument were significantly lower (p < .05) in the blended section for one of the three faculty on one particular item: “This course helped me better understand the relationship of Christian beliefs and values to the content of the course” (see Table 3). Otherwise, there were no statistically significant differences in students’ perceptions of faith integration in these classes.

26

*p < .05; **p < .01

Table 2 IDEA Course Evaluation Non-adjusted T-Scores on Items “Excellent Teacher” and “Excellent Course”

Instructor Excellent TeacherExcellent Teacher Excellent CourseExcellent Course

Traditional Blended Traditional Blended

1 51 49 53 46*

2 51 56* 50 44*

3 52 53 57 42**

The issue of student perceptions is perhaps one of the most crucial elements for blended learning formats to be successful.  This issue will be discussed in the Recommendations section.  Because of potential faculty concerns regarding IDEA scores, our recommendations will also reflect the importance of allowing IDEA evaluations not to “count” in faculty evaluation the first time a course is taught in the blended format.

A multivariate analysis of covariance conducted on students’ Engaged Learning Index scores indicated that there were no statistically significant differences in global levels of engagement in learning across the two formats, after taking into consideration

students’ pre-existing levels of academic engagement.  However, students in all the blended sections reported significantly lower scores on one item: “I could usually find ways of applying what I was learning in this class to something else in my life.” (p < .05)  Because application is theoretically a strength of the blended learning approach, this particular finding warrants further analysis and exploration.

A multivariate analysis of variance of students’ responses to the course satisfaction measure indicated that global satisfaction scores were not significantly different across the sections, but there were three items where the blended learning sections reported significantly lower levels of satisfaction:

• The amount of preparation that was expected for this class. (p < .05)

• The ease of using the technology that was expected in this class. (p < .01)

• The amount of work that was expected outside of class. (p < .05)

This finding is congruent with much of the research on blended learning, as faculty often report students are dissatisfied with the amount of work and have initial difficulties learning to use the technology (Dziuban, Moskal, and Futch, 2007; Woods, Badzinski, and Baker, 2007).  However, research also indicates that when students are fully aware of what is involved in a blended course, when faculty spend the first day of class

27

*p < .05

Table 3 Mean IDEA Scores on Faith Integration Items (Scored on a 5-point scale)

This course helped me better understand the relationship of

Christian beliefs and values to the

content of the course.

This course helped me better understand the relationship of

Christian beliefs and values to the

content of the course.

The professor modeled a Christian

perspective of truth and life in their

relationship with students in this

course.

The professor modeled a Christian

perspective of truth and life in their

relationship with students in this

course.

This course helped me better

understand the relationship of a

Christian worldview to my life and work

in the world.

This course helped me better

understand the relationship of a

Christian worldview to my life and work

in the world.

Instructor F2F Blended F2F Blended F2F Blended

1 4.2 3.7* 4.3 4.2 4.2 4.2

2 4.1 4.3 4.2 4.4 4.1 4.3

3 4.6 4.5 4.8 4.7 4.8 4.4

carefully explaining the benefits of blended learning and the amount of work expected, and when students are carefully trained on the technology as part of the course expectations, this dissatisfaction is reduced.  The workload should be no greater in a blended course, but there is the possibility that the workload of a F2F course is much less because faculty spend class time lecturing on what students were theoretically expected to have read before class--thus, students are not coming to class prepared because they really do not have to do so to be successful.  In contrast, a blended learning format not only expects this preparation before class, but demands it by virtue of the type of class activities in which the student engages.

A further examination of the course satisfaction items indicated that one instructor had significantly lower student satisfaction with her feedback and accessibility in the blended section, as well as lower levels of student engagement and interaction.  This instructor was one who had most diligently followed all the guidelines and suggestions for making the best possible blended learning environment.  We only were able to collect data for two of the three instructors on engaged learning and student satisfaction; both had students in the blended sections reporting significantly lower satisfaction with the amount of interaction they had with the instructor.  Thus, the issue of how to ensure adequate student-faculty interaction is of paramount concern in making any shift toward blended learning formats.

28

The qualitative analysis of individual faculty interviews, focus groups, and time logs was conducted to understand faculty’s general impressions of the blended modality.  After analyzing the transcripts from the focus group and interviews, a number of themes emerged that shed light on the practice of blended learning. The themes that surfaced from the interviews can be organized around the student experience, the faculty experience, and future possibilities.

Connection with Students

Means et al (2010) cited a sense of social presence as an important element of successful blended courses.  Our faculty had good instinct in this realm and sought to forge a connection with their students.  One professor felt that the affordances of blended learning were suitable for connection, saying: “I didn't really notice a difference in connection between the blended format and the non-blended.  I tried to make myself available, either face-to-face or online – whatever option the groups or the students decided.”

Some faculty used the class meeting time that was released by the blended format to meet one-on-one with students or student project groups, either in person or electronically with tools such as Google Hangouts, Adobe Connect, FaceTime or Gmail chat. One faculty member commented: “There was a lot of one on one. Sometimes they'd come with their research partners, and sit down and chat with me for a while about resources, or directions, or whatever.”  One participant recommended that faculty make themselves available during the time slot when the class is not meeting or even conduct mandatory online meetings with students.

Student Experience

Faculty believed that students would be better able to forge their side of the connection and thrive in the class if they were adequately prepared for the blended learning experience in terms of understanding expectations and  receiving technical training and then held accountable for their learning.

Section 3

Qualitative Analysis & Findings

29

All faculty participants, in their individual interviews and in the focus group, recommended that students be fully apprised of the course format they were enrolling in and how it would work. One comment on this recommendation was : “I think when students know ahead of time this is a blended course, and this is what a blended course is, before they sign up for it, it would be helpful.”

In addition to understanding requirements, students need training on the learning platform. While young adults tend to enjoy social media, they may not be as tech savvy as some think, lacking, for example,skills in business tools such as the Microsoft Office Suite, which many working adults frequently use. College students may or may not have been exposed to a lot of educational technology in younger grades, depending on which school district they came from. One of our participants observed: “I feel like both the students and I were on a learning curve”. One faculty member tried to accommodate this transition by allowing students the option of a paper or video assignment and reported that about half of the students still chose the paper option.

Despite these challenges, all faculty in the study felt that the use of online quizzes to hold students accountable for readings and other content was a great advantage. Some faculty also included weekly forums. One faculty member said of these techniques: “I really felt like that was overall very effective in keeping them up with the reading and the online materials - way more so than my teaching the class in the traditional way.”

Faculty Experience

All faculty acknowledged that achieving the benefits of blended learning required a significant amount of re-work on their courses.. One faculty member devoted some of her non-teaching summer time to course redesign. Another had the help of a teaching assistant. These two participants logged 120 and 144 hours in course redesign and development. The third faculty member worked 180 hours on this phase.

Once the course was  developed and faculty were ready to teach, they reported that it took about the same amount of time to teach the blended course as the face to face course.

All faculty thought that teaching their blended course a second time would require a similar amount of time to teaching a familiar face to face course, but without having to prepare for the lecture.  One participant said: “I think the blended has a learning curve. I think once you pass that hill, I think you can teach it just as easy as the face-to-face.”

Faculty Support

The necessary re-work could not be done without support. All faculty mentioned the great  need for training in the area of blended course design and development (creating the materials) including learning how to use the course development tools. They

30

all participated in some form of training or one-on-one support from ITT.

The faculty member who received the most training said: “The biggest thing I think helped me were the courses offered in the Spring, and also the week-long seminar in the Summer.”  Another faculty member said: “The meetings I had one on one with you were really helpful… I felt very supported.”

Trained faculty can also serve as a point person in their own departments as exemplified by this comment: “I've become the poster child for this. I've been asked a lot of questions about how it's going and what I recommend.”

A Positive Attitude about the Future

Overall, there was a belief in this modality – a sense that with adequate training and time for faculty to build their blended courses and properly set student expectations, offering this type of course is a good direction to go. There was a sense that the faculty will do better the next time around. All participants said they would teach a blended course again.  

Faculty affirmed that there was more time for discussion, projects, and group work in class, saying: “I think the main difference was the opportunity that was opened up for in-class discussions at a deeper level and being able to get into more

advanced concepts… Those are the areas that inspire more scientific inquiry and more passion about the topic… I had some students tell me that this was their favorite class.”

31

Chapter 5

Recommendations

The BLP research team recommends that APU proceed with caution to implement blended learning courses in the undergraduate and graduate curricula.

The BLP research team recommends that APU proceed with caution to implement blended learning courses in the undergraduate and graduate curricula.  We recommend that the approach be introduced over the course of an academic year and offered to a limited number of faculty on a voluntary basis at first, with sufficient training and resources provided for the experience to result in positive learning outcomes for students.

Specifically, we believe the following recommendations will lead to a successful adoption of blended learning at APU:

Recommendation #1

Faculty are required to get sufficient training before they are approved to teach blended courses. Since designing an effective blended course will take at least a semester, it is critical that faculty are provided sufficient training, support, and resources well in advance of course delivery. The Office of the Provost will provide appropriate compensation or release time for faculty engaged in blended courses.

Recommendation #2

Faculty teaching blended courses are expected to employ best practices in both the F2F and online or fieldwork components. Since instructions for blended courses occur in real and virtual/field environments, faculty need to be knowledgeable of the appropriate pedagogy for each environment. CTLA/ITT will provide resources, templates, checklists, APU faculty examples, and assistance on effective pedagogy during training, course design, and delivery of blended course.

Recommendation #3

Faculty teaching a blended course the first time may elect not to submit their IDEA course evaluations into Activity Insight. Faculty trying new pedagogy and taking thoughtful risks in developing blended courses should not be penalized for experimenting. CTLA will support faculty who need to exclude their IDEA course evaluations from Activity Insight.

Section 1

Recommendations

33

Recommendation #4

Faculty teaching blended courses are invited to participate in an informal faculty learning community (FLC). A blended learning community serves as an important space for interested faculty to connect, compare notes, and learn from one another. CTLA/ITT will establish a year-round blended learning community dedicated to blended learning.

Recommendation #5

Students need to understand the unique nature, expectations, and experience of blended learning formats at the beginning of the course. Since a blended course differs from a F2F course or a purely online course, students need to understand how to optimally engage in a blended learning format in order to succeed. Academic units, Registrars, and faculty need to clearly communicate the nature and expectations of blended courses early and often.

Recommendation #6

A formal process will be established for approving blended courses and certifying faculty readiness, similar to current process with online learning. Effectively designed blended courses and well-trained faculty are critical to ensuring a

successful blended learning experience for students. ITT will oversee approval and certification processes for blended courses.

Recommendation #7

The process for scheduling blended courses must be addressed with careful planning and forethought, as well as flexibility when inevitable changes occur. Blended courses present opportunities to improve and optimize classroom usage, with potential to increase enrollment capacity within existing classrooms. Academic units and the Registrars need to work together and coordinate the scheduling of blended courses.

Recommendation #8

Faculty and students should own or have access to appropriate technologies that enhance the blended learning experience (e.g., tablets, modern LMS, etc.). When technology is intuitive, modern, and reliable, faculty feel empowered to take greater risks in their teaching, and students are encouraged to take greater responsibility for their learning. ITT will assess existing gaps in academic technologies and collaborate with IMT on finding appropriate and cost-effective solution.

34

Chapter 6

References & Appendices

Abel, R., Brown, M., and Suess, J.J. (2013). A New Architecture for Learning. EDUCAUSE Review Online.

Atkinson, J. W. (1957). Motivational determinants of risk taking behavior. Psychol. Rev. 64: 359-372.

Aycock, A., Garnham, C., & Kaleta, R. (2002, March). Lessons learned from the hybrid course project. Teaching with Technology Today, 8(6).

Barkley, E. F. (2010) Student Engagement Techniques: A Handbook for College Faculty. San Francisco: Jossey-Bass.

Biocca, F., Harms, C. & Burgoon, J.K. (2003) Toward a more robust theory and measure of social presence: review and suggested criteria. Presence, Teleoperators and Virtual Environments, 12(5), 456-480.

Bowen, W. G., Chingos, M. M., Lack, K. A., & Nygren, T. I. (2012). Interactive Learning Online at Public Universities: Evidence from Randomized Trials: Ithaka S+R. Retrieved from http://

www.sr.ithaka.org/research-publications/interactive-learning-online-public-universities-evidence-randomized-trials

Brophy, J.E. (2004). Motivating students to learn. Mahwah, NJ: Erlbaum.

Campbell, M., W. Gibson, A. Hall, D. Richards, and P. Callery. 2008. Online vs. face-to-face discussion in a Web-based research methods course for postgraduate nursing students: A quasi-experimental study. International Journal of Nursing Studies 45 (5):750–59.

Castaneda, R. 2008. The impact of computer-based simulation within an instructional sequence on learner performance in a Web-based environment. PhD diss., Arizona State University, Tempe.

Chickering, A., and Gamson, Z. (1987). Seven Principles for Good Practice in Undergraduate Education. American Association for Higher Education (AAHE), 39(7), 3- 7.

Section 1

References

36

Chung, S., M.-J. Chung, and C. Severance. 1999, October. Design of support tools and knowledge building in a virtual university course: Effect of reflection and self-explanation prompts. Paper presented at the WebNet 99 World Conference on the WWW and Internet Proceedings, Honolulu, Hawaii. (ERIC Document Reproduction Service No. ED448706).

Collins, A., & Halverson R. (2009). Rethinking education in the age of technology: The digital revolution and schooling in America. New York: Teachers College Press.

Cross, K. T. (2001, February). Motivation: Er… will that be on the test? Mission Viejo, CA: League for Innovation in the community College.

Dahlstrom, E., & Bichsel, J. (2014). ECAR Study of Undergraduate Students and Information Technology, 2014. Louisville, CO: ECAR.  Retrieved from: http://net.educause.edu/ir/library/pdf/ss14/ERS1406.pdf.

Diaz, D. (2002). Online Drop Rates Revisited. The Technology Source Archives at The University of North Carolina. May/June.

Deci, E. & Ryan, R. (1985). Intrinsic motivation and self-determination in human behavior. New York: Plenum.

Deci, E. & Ryan, R. (Eds.).(2002). Handbook of self-determination research. Rochester, NY: University of Rochester Press.

Dziuban, C.D., Moskal, P. D., Hartman, J. (2005). Higher education, blended learning, and the generations: Knowledge is power: No more. In J. Bourne & J.C. Moore (Eds.), Elements of Quality Online Education: Engaging Communities. Needham, MA: Sloan Center for Online Education

Dziuban, C.D., Moskal, P. D., Futch, L. (2007).  Reactive Behavior, Ambivalence, and the Generations: Emerging Patterns in Student Evaluation of Blended Learning. In Picciano, A. G., & Dziuban, C. D. (Eds.). (2007). Blended learning: research perspective. Needham, MA: The Sloan Consortium.

Edgerton, R. (1997) Higher education white paper. Pew Charitable Trusts.

Graham, C., Woodfield, W., & Harrison, J. B. (2013). A framework for institutional adoption and implementation of blended learning in higher education. Internet and Higher Education, 18, 4-14.

Grant, L. K., and M. Courtoreille. 2007. Comparison of fixed-item and response-sensitive versions of an online tutorial. Psychological Record 57 (2):265–72.

37

Griffiths, R., Chingos, M., Mulhern, C., & Spies, R. (2014).  Interactive Online Learning on Campus: Testing MOOCs and Other Platforms in Hybrid Courses in the University System of Maryland. Retrieved from http://www.sr.ithaka.org/research-publications/interactive-online-learning-on-campus#

Gunawardena, C. N. (1995). Social presence theory and implications for interaction and collaborative learning in computer conferences. International Journal of Educational Telecommunications, 1 (2/3), 147-166.

Helfand, D. (2013). Watering the Roots of Knowledge Through Collaborative Learning. Chronicle of Higher Education. July.

Hostetter, C., & Busch, M. (2006). Measuring up online: The relationship between social presence and student learning satisfaction. Journal of Scholarship of Teaching and Learning, 6 (2), 1-12.

Johnson, D.W., & Johnson, R.T. (2004). Cooperation and the use of technology. In D.H. Jonassen (Ed.),Handbook of research on educational communications and technology (pp. 785-811). Mahwah, NJ: Lawrence Erlbaum Associates.

King, S. E. & Arnold , K. C. (2012) Blended Learning Environments in Higher Education: A Case Study of How Professors Make It Happen. Mid-Western Educational Researcher, 25(1-2), 44-59.

Ladd, H., Reynolds, S., & Selingo, J. J. (2013). The differentiated university: Recognizing the diverse needs of today's students. Boston, MA: The Parthenon Group.

Lewis, B. A. 2002. The effectiveness of discussion forums in online learning. Brazilian Review of Open and Distance Learning 1 (1).

Lim, D. & Yoon, S. (2008) Team learning and collaboration between online and blended learner groups. Performance Improvement Quarterly, 21(3), 59-72.

Means, B., Toyama, Y., Murphy, R., Bakia, M., & Jones, K. (2010). Evaluation of evidence-based practices in online learning: A meta-analysis and review of online learning studies. Washington, D.C.: U.S. Department of Education. Retrieved from http://www2.ed.gov/rschstat/eval/tech/evidence-based-practices/finalreport.pdf.

Merrill, M. D. (1984). What is Learner Control? Instructional Development: The State of the Art, p 221-242.

Park, J.-H., & Choi, H. J. (2009). Factors Influencing Adult Learners' Decision to Drop Out or Persist in Online Learning. Educational Technology & Society, 12 (4), 207–217.

38

Picciano, A.G. (2002). “Beyond student perceptions: Issues of interaction, presence and performance in an online course.” Journal of Asynchronous Learning Networks, 6(1).

Richardson, J. and Swan, K. (2003). “Examining social presence in online courses in relation to students’ perceived learning and satisfaction.” Journal of Asynchronous Learning Networks, 7(1). Retrieved from http://www.aln.org/publications/jaln/v7n1/pdf/v7n1)richardson.pdf.

Saito, H., and K. Miwa. 2007. Construction of a learning environment supporting learners’ reflection: A case of information seeking on the Web.  Computers & Education 49 (2):214–29.

Selingo, J. J. (2014). The innovative university: What college presidents think about change in American higher education.

So, H. J., & Brush, T. A. (2007). Student perceptions of collaborative learning, social presence, and satisfaction in a blended learning environment: Relationships and critical factors. Computers & Education. doi:10.1016/j.compedu.2007.05.009

Stein, J. and C. Graham (2014). Essentials for blended learning : a standards-based guide. New York, Routledge.

Wlodkowski, R. J. (2008) Enhancing adult motivation to learn: A comprehensive guide for teaching all adults. San Francisco: Jossey-Bass.

Woods, R., Badzinski, D.M., Baker, J. (2007).  Student Perceptions of Blended Learning in a Traditional Undergraduate Environment.  In Picciano, A. G., & Dziuban, C. D. (Eds.). (2007). Blended learning: research perspective. Needham, MA: The Sloan Consortium.

Wright, C.R., Lopes, V., Montgomerie, C., Reju, S., & Schmoller, S. (2014). Selecting a Learning Management System System: Advice from an Academic Perspective. EDUCAUSE Review Online.

Zhang, D., L. Zhou, R. O. Briggs, and J. F. Nunamaker, Jr. 2006. Instructional video in elearning:Assessing the impact of interactive video on learning effectiveness. Information and Management 43 (1):15–27.

39

Section 2

Appendix 1 - Blended Learning Adoption Framework

40

Focus group Questions

1.    As you think back on this semester, on a scale of 1 – 10 – where 1 is very bad and 10 is the best possible experience - how would you rate the semester and explain why you gave that number.

2.    All things considered – how satisfied do you think students were?

3.    Do you think their learning was enhanced – they learned differently or better? Why?

4.    Tell me about things that did not go well.

5.    What advice would you give a faculty member interested in doing this?

6.    What suggestions do you have for CTLA in terms of the support that faculty are likely to need?

7.    Is there anything else you want to add?

Individual Interviews Questions

1.    You gave us some great insight in the focus group. As you think back over the semester – what do you think were the biggest differences between the face-to-face and blended courses - in the way you taught and experienced this class?

2.    What did you do specifically differently moving from face to face to blended?  Consider your design and delivery approaches and why you made the decisions you did.

3.    How were the students’ experiences different? If I brought in two students – from the two different semesters – what would I hear from them or what would I notice about their differences?

4.    How much time did you spend per week and how did you spend your time on each of the following phases:

• ·         Design (Writing syllabus, defining what happens each week)

Section 3

Appendix 2 - Questions for Qualitative Interviews

41

• Lesson prep (Creating the media: slides, recordings, handouts)

• Delivery (Class time or participating in the discussion boards)

• Feedback (Grading and time w/students outside of class) and assessment.

5.    How do each of these times compare to the work required for the face to face classroom?

6.    Do you think the second time through would require the same amount of effort? What about compared to the second time through for face to face?

7.    Would you do this again? Why?

8.    Do you prefer blended or face to face – and why?

9.    Between the focus group and the interview, we asked a lot of questions. Is there anything you want to add?

42