INTERNATIONAL CONFERENCE OF EDUCATIONAL TECHNOLOGY · Improving the Usability of E-Learning User...

“Toward smart approaches to education: bridging learning theory, technology application, and teaching practice”

INTERNATIONAL CONFERENCE OF EDUCATIONAL TECHNOLOGY

ICET2013ICET2013INTERNATIONAL CONFERENCE OF

EDUCATIONAL TECHNOLOGY

November 23. 2013GwangGaeTo Building 8th / 15th Floor

SEJONG UNIVERSITY

��A4.indd 1 2013-11-19 �� 5:48:55

1

- I -

Table of Content

Welcoming Address - Insook Lee (President of KSET, Sejong University) .......................................................3

Program Table............................................................................................................5

Plenary SessionHistory, Trends, and Issues in Educational Communications and Technology

- Marcus Childress (President of AECT, Emporia State University)..................................7

Keynote SpeechCognitive Load Theory and Educational Technology

- John Sweller (University of New South Wales and Honorary Professorial Fellow University of Wollongong).............................................................................................17

Invited PresentationAn Organization Model for Ubiquitous Learning Resource From Learning Object to Learning Cell

- Shengquan Yu (Beijing Normal University)...................................................................30

Designing Telepresence System for Distance Learning in Hyper-Aged Society

- Atsushi Hiyama (University of Tokyo)...........................................................................59

Teaching to Vitalize, Rather than Neglect, Students’ Motivation

- Johnmarshall Reeve (Korea University).........................................................................76

Concurrent Session I Virtual Worlds in Education: Changes Of Student’s Perspectives And Learning Outcome - Mihwa Kim (Seoul National University).........................................................................88

Using a Virtual Tutee System to Promote Academic Reading Engagement- SeungWon Park (Texas A&M University) / ChanMin Kim (University of Georgia Athens)......93

Factors influencing students’ adoption of MyGuru2 asynchronous online discussion in Malaysia- Mahizer Hamazah (Sultan Idris Education University)..................................................98

user

Stamp

- II -

Relationships among Learners’ Efficacy Beliefs, Perceptions on Scaffolding, Learning Participation and Achievement in Team Project-Based Learning- Youngsoo Kim (Ewha Womans University) / Heeok Heo (Sunchon National University)

/ Youngsun Yang (Kwandong University)...................................................................104

CSCL Scripts for the Collective Working Memory Effect- Jihyun Si (Hanyang University)...................................................................................108

Fostering Students’ Team Shared Mental Model and Team Satisfaction using Debate in the College Classroom- Myongnam Jun (Daegu Haany University) / Heather L Allen (Daegu Hanny University)...112

Multimedia Learning for Speaking Fluency - Joohee Son (Columbia University)...............................................................................118

The Transfer of the Foreign Language Curricular Goals and the Implications for ELP Curricular Development- Luping Zhang (China University)................................................................................123

Concurrent Session IITechnology, Connectedness, and Learning in the Digital Age: A Conceptual Framework for Digital Learning Standards- HyeJeong Kim (Chung-Ang University) / Wonseok Suh (Chung-Ang University) /

Hanho Jeong (Chongshin University) / Youngju Lee (Korea National University of Education) / Hae-Deok Song (Chung-Ang University)................................................134

Transforming Teaching and Learning Innovation: The Center for Teaching and Learning (CTLE) Model- Thapanee Thammetar (Silpakorn University) / Chattiwat Wisa (Silpakorn University) /

Ruangrit Nammon (Silpakorn University) / Bangthamai Eknarin (Silpakorn University)....139

Relationship or Content First? - MiJar Lee (Gwangju National University of Education)................................................143

The Effects of Informal mentoring on Organizational Commitment and Organizational Citizenship Behavior- YoungRan Yoo (Ewha Womans University) / Myunghee Kang (Ewha Womans University)

/ JiHyun Kim (KRIVET) / Jiwon You (Ewha Womans University).................................149

Improving the Usability of E-Learning User Interfaces: Affordance-Based Design- Hae-deok Song (Chung-Ang University).....................................................................155

Developing Design Principles of Emotional Interface for Self-Regulated Learning in an E-Learning Environment- Cheolil Lim (Seoul National University) / Taejung Park (Seoul National University) /

Wonjoon Hong (Seoul National University) / Jungeun Park (Seoul National University)....161

user

Stamp

- III -

Design of Curriculum Management System- Innwoo Park (Korea University) / Wonsuk Shin (Korea University) / SongYi Beak

(Korea University) / HyeYeong Kim (Korea University)...............................................169

Mobile Phone Use, & Lifelong Learning.- Ken Morrison (Hannam University).............................................................................173

Smart Use of LMS in Higher Education: Viewing Students’ Perceptions in a Framework of Activity Theory- Yeonjeong Park (Ewha Womans University) / IlHyun Jo (Ewha Womans University)...178

Design and Practice of Project-Based Collaborative Learning Between Korean and Japanese University Students- Shinichi Sato (Nihon Fukushi University) / Makoto Kageto (Nihon Fukushi University)

/ Jeeheon Ryu (Chonnam National University)............................................................185

Learner-Centered and Collaborative Learning through Designing Digital Textbook for Music Curriculum in South Korea- Dong Yub Lee (KICE) / Sahoon H. Kim (KICE) / Ji Hyun Park (KICE)........................190

Effects of the Types of Communication Media on Collaborative Problem Solving Tasks- Hyewon Kim (Dankook University) / Minjeong Kim (Dankook University) / MiYoung Lee

(Dankook University)...................................................................................................197

Student SessionThe Effects of Creativity and Flow on Learning through the STEAM Education on Elementary School Contexts- Boram Cho (Ewha Womans University) / Jeongmin Lee (Ewha Womans University)..206

The Effects of Desirable Difficulties on Collaboration Load and Learning Outcome in Collaborative Learning Environment- Yoonmee Kim (Hanyang Unversity) / Dongsik Kim (Hanyang University)...................211

The Effects of Academic Emotions on Motivation in e-Learning- Seungho Kim (Visang Education) / Insook Lee (Sejong University)............................215

How Do the Level of Complex Learning Task and the Part-task Sequencing Affect on Mental Model, Cognitive Load, and Learning Time?- Kyungjin Kim (Hanyang University) / Dongsik Kim (Hanyang University)...................220

Predictability of Presence on Learning Persistence and Learning Satisfaction in Facebook -Based Collaborative Learning Environment- Hyunmin Chung (Ewha Womans University) / Sungeun Oh (Ewha Womans University)

Jiyoon Moon (Ewha Womans University) / Jeongmin Lee (Ewha Womans University)......224

user

Stamp

- IV -

Case study of the Need Assessment and Usability Issue in Designing and Developing e-Book- Hyungju Lee (Chonnam National University) / Sinok Kim (Chonnam National

University) Gwansun Hong (Chonnam National University) / Sanha Kang (Chonnam National University) Jeongah Woo (Chonnam National University) / Yoojin Hong (Chonnam National University)....................................................................................229

The Effects of Simulation Game-Based Learning on Academic Emotions and Achievement- Yunha Jung (Ewha Womans University) / KyuYon Lim (Ewha Womans University)...233

The Effects of Part-task Sequencing and the Level of Element Interactivity on Schema Automation and Cognitive Load- Hyejeong Lee (Hanyang University) / Dongsik Kim (Hanyang University)...................238

Effect of Conversational Gesture of Pedagogical Agent and Visual Cueing on Task Comprehension and Eye Fixation- Jewoong Moon (Chonnam National University) / Jeeheon Ryu (Chonnam National

University)...................................................................................................................242

Assessment of Virtual Patients on Realistic Performance- Sun Kim (Chonnam National University) / Jeeheon Ryu (Chonnam National University)....246

Usability Study of Visual Dashboard as Learning Analytics Interventions- Kunhee Ha (Ewha Womans University) / Sohye Lim (Ewha Womans University) /

Il-Hyun Jo (Ewha Womans University).........................................................................249

The Effects of Regulatory Learning Strategies on Collaboration Load and Collaboration Outcomes in Computer-Supported Collaborative Learning- Hyojin Lee (Hanyang University) / Dongsik Kim (Hanyang University).......................256

Improvement of Score Reading Skill By Music Composing Class with SMART Education- Hyerin Lee (Chunchon National University of Education)...........................................261

The Effect of Awareness Information on Affect-based Trust in Collaborative Problem-solving Learning: A Pilot Study- Jongsuk Song (Hanyang University) / Dongsik Kim (Hanyang University)..................266

Student’s Perception on Learning Analytics Dashboard (LAD) Presenting Online Activities in LMS- Stephanie Kang (Ewha Womans University) / Yeonjeong Park (Ewha Womans University)

/ Il-Hyun Jo (Ewha Womans University).....................................................................269

user

Stamp

Student Session

205

user

Stamp

user

Stamp

The Effects of Creativity and Flow on Learning through the STEAM

Education on Elementary School Contexts

Boram Cho

[email protected]

Student

Ewha Womans University

Seoul, Korea

Jeongmin Lee

[email protected]

Professor


Seoul, Korea

ABSTRACT

This study aims to examine the effects of STEAM education on elementary school student‟s

creativity (creative problem solving, creative personality) and flow on learning. STEAM education is

composed of 5 strands: Science, Technology, Engineering, Art, and Mathematics. The STEAM education

provides convergence education to explore diverse thinking and achieve future convergence human

resources. Previous studies on STEAM education have been done on the model development and concept

formulation. There was very little application research. In addition, test subjects were usually middle

school and high school students. Therefore, this study focused on the elementary school students to

investigate the effect of STEAM lessons. This study made STEAM lesson plans that strengthen the

linkages themes among the subjects. It helps students acquire the creative design and emotional experience.

Based on this purpose of study, there are two research hypotheses. First, STEAM education Improves

creativity (creative problem solving, creative personality) on elementary school students. Second, STEAM

education enhances the flow on learning on elementary school students. The subjects in this analysis were

6th graders, two classes from elementary schools. Each class was taught for 45minutes during 8weeks by

the same teacher and performed 3tests as time goes by. After the test, we examine the changes in student‟s

creativity and flow on learning. Creativity was measured by two aspects; cognitive aspect and emotive

personality. Applied statistical methods were two independent samples t-test. As the result, there were

significant differences in creativity (creative problem solving, creative personality) and flow on learning

through the STEAM education. The result indicates that STEAM education was helpful to improve

creativity (creative problem solving, creative personality) and flow on learning.

Keywords: STEAM education, Creativity (creative problem solving, creative personality), Flow on

Learning

INTRODUCTION

The wave of change occurs in fast pace in the information age. Even though the world united as one

with rapid technology development, new complex problems occur by cultural diversity, environmental

diversity, and multiple values. Based on these phase of times, Ministry of Education and Science

Technology(MEST) published Steam education to improve the elementary school student's creativity

improvement in December 17th, 2010. STEAM education stands for convergence education in area of

206

Science, Technology, Engineering, Art, and Mathematics, and it aims for developing talented person with

creativity and personality who can solve various problems in rapidly changing society (MEST, 2010).

STEAM education adopted in Korea to learn association of theoretical principles with reality by

linking science, mathematics, and technology which are difficult subjects for students to arts and

engineering. Students can practice of adopting in reality through engineering and technology, and they

can grow as talented creative person through sensibility of arts (Korean Educational Development

Institute, 2012).

In addition, new rising keyword in our society is creativity and communication (Noh & Ahn, 2012).

There was a limitation on developing talented person with personality of communication with STEM

education that developed in United States, so STEAM education that cultivates creativity and personality

linking imagination and sensibility with technological education was suggested.

Educational outcome of STEAM education can be divided as cognitive side and emotive side.

Improvement of problem solving, creativity, cooperative learning, concentration in subject, and critical

thinking was positively influenced by educational outcome of cognitive aspects (Kim et al., 2011;

Kim&Kim, 2012; Shin et al., 2013). As emotive side, there are positive influences on learner‟s interest,

motivation (Bae, 2011, Kim et al., 2011; Kim & Kim, 2012; Moon, 2009), and attitude (Bae, 2011, Moon,

2009).

Currently, basic research such as concept establishment and model development is in progress

from STEAM education, and it lacks research in program development and application. In addition, test

subjects were usually middle school and high school students. Program development and dissemination

of STEAM education is priority issue for invigorating STEAM education from elementary school

teacher's research (Kum & Bae, 2012). Also, the research from HeSook Han and HwaJung Lee (2012)

stated that STEAM program development and dissemination is the most needed issue for helping

teacher‟s comprehension of STEAM education. In addition, existing STEAM education programs were

lack of links between subjects (Kim, 2012). Looking at the existing problems of subject, technical

education lacks of scientific principle explanation and science education needed to strengthen the linkages

themes among the subjects. Math education focused problem solving oriented class (Baek, 2011). Thus,

according to the passage, this study is needed to make a STEAM lesson plan that based on real life

contents and strengthen the linkages themes among the subjects. After that, we examine the effects of

STEAM education about creativity and flow on learning. Creativity was measured by two aspects;

cognitive aspect and emotive personality. Cognitive aspect measured by creative problem solving and

emotive personality measured by creative personality. The purpose of this study is to analyze about

effects of STEAM education on elementary school contexts.

Research problems

Based on this purpose of study, there are three research questions as follows:

First, does STEAM education improve creative problem solving on elementary school students?

Second, does STEAM education improve creative personality on elementary school students?

Third, does STEAM education enhance the flow on learning on elementary school students?

METHOD

Sample and procedures The subjects in this analysis were 6th graders, two classes from elementary schools. Each STEAM

class was taught for 45minutes, once per week, during 8weeks by the same teacher. The developed

STEAM program was reviewed from 3 elementary specialists. Before and after the instruction, we

examine the changes in student‟s creative problem solving, creative personality, and flow on learning.

Paired t-tests were used for data analysis methods.

207

Measures Creative problem solving

Creative problem solving is developed by Jeong(2008). It is created by Korean Educational Development

Institution in 2001year, and then MI research team of Seoul National University developed „easy creative

problem solving development research(1) in 2004year. It consists of 20 questions. The question contents

are specific areas of knowledge, thinking skills, understanding and mastery of the technology, divergent

thinking, critical reasoning and motivation. The questions were rated on Likert response scale (1=“to a

very slight extent or not at all to 5=”to a very large extent”). A higher score indicates higher level of

creative problem solving. Cronbach α reliability for this scale was .90 in this study.

Creative personality

Creative personality is developed by Ha (2000). It was used to measure creative personality for

Elementary students. It consists of 22 questions. The contents are curiosity, self-confidence, imagination,

patience/obsession and humor. The questions were rated on Likert response scale. Cranach α reliability

for this scale was .88 in this study.

Flow on learning

The flow on learning developed by Seok & Kang (2007) was used to measure flow on learning in

elementary school students. It consists of 35 questions. The sub factors are combination of challenges and

skiils, clear goals, specific feedback, the integration of action and awareness, sense of control,

concentration challenge, loss of consciousness, distorted sense of time, experience and self-purposive.

The questions were rated on Likert response scale. Cronbach α reliability for this scale was .94 in this

study.

Results

<Table1> Change of creative problem solving

n M SD df t p

Pre test 45 67.04 11.91 44 -4.24 0.000

*

Post test 45 73.11 13.40

*P<.05

Based on Table 1, the pre-test average was 67.04 and standard deviation was 11.91. The post-test

average was 73.11 and standard deviation was 13.40. For difference between pre and post test results,

this study verified statistical significance, t statistic value was -4.24, significance probability was .000

(p<.05). The result indicated that STEAM education improve creative problem solving significantly.

<Table2> Change of creative personality

n M SD df t p

Pre test 45 70.90 11.07 44 -2.40 0.021

*

Post test 45 73.80 13.39

*P<.05

208




(p<.05). The result indicated that STEAM education improve creative personality significantly.

<Table 3> Change of flow on learning

n M SD df t p

Pre test 45 69.09 12.92 44 -4.02 0.000

*

Post test 45 75.47 13.79

*P<.05




(p<.05). The result indicated that STEAM education improve flow on learning significantly.

CONCLUSION

The purpose of this study is examining the effects of creativity and flow on learning through

STEAM education on elementary school context. There is a brief summary of results. This study

conducted making a STEAM education lesson plan and then taught STEAM education program 45min/

once a week for 8weeks. After the 8weeks, the results showed that there were statistically significant

(p<.05) differences in creativity (creative problem solving, creative personality) and flow on learning.

Therefore, the results have implications that the STEAM education was helpful to improve creativity and

flow on learning on elementary school students. The results of the study provide useful information for

future researches on STEAM education. The previous research also supports current study‟s results. The

following is suggestions of future research directions. First, future research should be developed

structured model and theoretical framework. Second, STEAM education should reflect teacher, student

and parents‟ needs of the program in the development stage. Third, The STEAM education should create

a long-term integrated curriculum as project 21 in US.

REFERENCES

Bae, S. A. (2011). The Development and Application of Activity-Centered STEM Education

Program of Electricity, Electronics Technology area in Middle School. The Journal of Korean Institute of

Industrial Education, 36(1), 1-22.

Baek, Y. S., Park, H. J., Kim, Y. M., Noh, S. K., Park, J. Y., Lee, J. Y., Jeong, J. S., Choi, Y. H.,

Han, H. S.(2011). STEAM Education in Korea. Journal of Learner-Centered Curriculum and Instruction,

11(4), 149-171.

Ha, J. H. (2000). A Development Creative Personality Scale. Korean Educational Research

Assorciation, 14(2), 187-210. Han, H. S., Lee, H. J. (2012). A Study on the Teachers‟ Perceptions and Needs of STEAM

Education. Journal of Learner-Centered Curriculum and Instruction , 12(3), 573-603.

209

Jeong, U. Y. (2008). (The) Effects of Squeak Etoys based informatics education on elementary

school student's creative problem solving ability. Unpublished master's thesis, Choongbuk: Korea

National University of Education.

Kim, J. A., Kim, B. S., Lee, J. H., Kim, J. H. (2011). A Study of Teaching-Learning Methods for the

IT-Based STEAM Education Model With Regards to Developing People of Interdisciplinary Abilities.

The Korea Society for Fisheries and Marine Sciences Education, 23(3), 445-460.

Kim, T. H., Kim, J. H. (2012). A Development of Android Application for Physics Learning Based

on STEAM . The Korea Society for Fisheries and Marine Sciences Education, 24(1), 25-33.

Korean Educational Development Institute (2012). Understanding of STEAM education through

actual application case. 2012(2). 1-53. from

http://edpolicy.kedi.re.kr/EpnicForum/Epnic/EpnicForum02Viw.php?PageNum=3&S_Key=&S_Menu=

Ac_Code=D0010201&Ac_Num0=13090

Kum, Y. C., Bae, S. A. (2012). The Recognition and Needs of Elementary School Teachers about

STEAM Education. Korean Institute of Industrial Educators, 37(2), 57-75.

Ministry of Education and Science Technology:MEST (2010). Creative talented person and

advanced science technology with future Korea. 2011year working report.

Moon, D. Y. (2009). A Case Study on Elementary Students' Attitudes toward Engineering and

Engineering Problem Solving: Through Applying the Education Program of STEM Integration Approach.

Korean Educational Research Association, 22(4), 51-66.

Noh, S. W., Ahn, D. S. (2012). A Study on Direction of Development in STEAM Education. The

Education Research, 10(3), 75-96.

Seok, I. B. (2007). Construction of Flow on Learning: Scale, Personality, Condition, Participaation.

Unpublished doctoral dissertation, Kyeongbuk: Kyeongbuk University.

Shin, J. H., Nam, K. J. D., Kim, Y., Park, S. S., Jo, J. B., Lee, Y. M., Han, J. Y. (2013).

Development and Adaptation of Art-Oriented STEAM Converged Education Program through Toy-

Drama. Journal of Learner-Centered Curriculum and Instruction, 13(1), 215-240.

Shin, Y. J., Han, S. K. (2011). A Study of the Elementary School Teachers' Perception in

STEAM(Science, Technology, Engineering, Arts, Mathematics) Education. The Elementary Science

Education, 30(4), 514-523.

210

The effects of desirable difficulties on collaboration load

and learning outcome in collaborative learning environment

Younmi Kim

[email protected]

Doctoral Student

Hanyang University

Seoul, Korea

Dongsik Kim

[email protected]

Professor

Hanyang University

Seoul, Korea

ABSTRACT The purpose of this study was to investigate how collaboration load and learning outcome were

affected by the experience of desirable difficulties and the way of designing the experience. We

suggested that problematizing approach might be advantageous for learners who were required to

conduct learning tasks in collaborative environment, although many studies indicated that

scaffolding strategies were useful. 135 students in three classes were participated in this study and

classes were assigned to each group with full script, simple script and delayed full script. The results

revealed that there were significant differences in task load, process satisfaction and flow of

collaboration load categories and no significant differences between individual learning outcomes.

Based on the results, this study discussed the effects of desirable difficulties and implications for

further study.

Keywords: desirable difficulty, collaboration load, collaborative learning environment

INTRODUCTION

Supporting and guiding learners in collaborative learning environment may not always guarantee

better learning outcomes. Some learners can benefit from various supports such as aids, tools, and scripts,

and others not.

There are two approaches for design of collaborative learning (Reiser, 2002): structuring approach

(such as guiding learning process, providing tool that restrict or lead a particular learning activities) and

problematizing approach (to make something in students’ work more problematic). Structuring

approaches provide supports for learners to avoid difficulties that can hinder their learning, whereas

problematizing approaches let them encounter some challenges that may be exposed in real world.

Majority of studies have focused on the structuring approach, but some studies revealed that too much

supports could have negative effects on learning outcomes.

Guiding learners can be advantageous to collaborative learning when unexperienced learners receive

some help from learning supports to perform a collaborative task. Although supporting learners in groups

with various scaffolding strategies and methods can be productive, some studies have shown that guiding

learners do not always lead to positive results of learning process.

Kapur(2008) has introduced the concept of ‘productive failure’ and his researches are based on

problematizing approach. Learners who were in lack of support in collaborative phases and experienced

productive failure showed higher scores with respect to individual achievement, even though they failed

in their collaborative task (Kapur, 2008; Kapur & Kinzer, 2009; Kapur & Bielaczyc, 2012). Also, studies

with script theory addressed that learners’ efforts to perform their tasks themselves without support could

211

mailto:[email protected]

be helpful in learning outcomes (Makitalo, Weinberger, Hakkinen, & Fischer, 2005; Fischer, Kollar,

Stegmann, & Wecker, 2013).

Based on problematizing approach, this study aim to provide that ‘desirable difficulties(this word

means insufficient and problematic situation)’ may have positive effects on learning and can be designed

with to improve collaborative learning.

Research questions

Research questions are as follows:

1. To what extent is the collaboration load affected by experience of desirable difficulties and the

way of designing desirable difficulties in collaborative learning environment?

2. To what extent are the individual and collective outcomes affected by experience of desirable

difficulties and the way of designing desirable difficulties in collaborative learning environment?

METHODS

Participants and Research design

Participants were N=135, freshmen from ‘Career Planning and Self-Development’ classes at

Dankook University in Korea. Three different classes were randomly assigned to group1, 2 and 3. Then,

students of each class were assigned to 3 or 4-person teams and required to solve a collaborative task.

Each group was in one of three conditions including different script types(full script, simple script, and

delayed full script) and required to submit the worksheet at the end of the class. Pretest and posttest were

asked to respond before and after class individually.

<Table 1>. The research design

Phase group1 group2 group3

1 pre-test (individually)

2 full script simple script Nothing full script

3 posttest(individually & collectively)

Instructional Conditions

All groups went through same procedures, during same time, by an instructor. Phrase2 were

conducted during normal classes. The students in a team were got together and presented with one of

three different types of script through smart phone. Then they were asked to complete the form of

worksheets on smart phone through discussion collaboratively. Students were permit to refer to teaching

materials and search on the internet by phone as needed.

Instruments and Data Analysis

The students were measured on collaboration load using a questionnaire with a five-point scale and

on individual and collective outcomes using scored worksheets. We adopted the questionnaire developed

by Jeong & Kim(2012) and graded the worksheets based on criterion developed by the instructor and

subject matter expert

One-way ANOVA tests were used to compare the collaboration load and individual and

collaborative learning outcomes. Then post-hoc Scheffe tests were also used to identify which two groups

had significant differences statistically.

212

RESULTS

Collaboration Load

One-way ANOVA showed that there were significant differences in task load, process satisfaction

and flow of collaboration load categories. And it was confirmed that there were significant difference in

task load between group1 and group3 by post-hoc Scheffe test. Group1 was significantly higher than

group2 and 3 in process satisfaction and flow.

<Table 2>. Collaboration load

Source SS df MS F p

Task load Between Groups 9.032 2 4.516 5.064 .008

Within Groups 117.710 132 .892 Total 126.743 134

Germane load Between Groups .934 2 .467 1.110 .333


Intrinsic load Between Groups 2.101 2 1.050 3.073 .050


Process satisfaction

Between Groups 5.377 2 2.689 5.600 .005 Within Groups 63.368 132 .480

Total 68.745 134

Performance satisfaction

Between Groups 1.358 2 .679 1.607 .204 Within Groups 55.801 132 .423

Total 57.159 134

Extraneous load

Between Groups 3.732 2 1.866 3.048 .051 Within Groups 80.802 132 .612

Total 84.534 134

flow Between Groups 9.787 2 4.893 8.695 .000


Individual and Collaborative Outcomes

The results showed no significant differences between mean scores of individual learning outcomes.

The individuals’ mean scores of group1were higher than others, but the difference was not significant

(p>.05). Otherwise, group 1 and 3 obtained much higher scores than group2 in collaborative outcomes

significantly.

DISCUSSION

In summary, we found that the experience of desirable difficulties affected collaboration load partially

and collaboration outcomes but individual outcomes. From the findings, it may not be sure that the

experience of desirable difficulties was meaningful in collaborative learning environment. Nevertheless,

we need to focus on the result that there was no significant difference between individual learning

outcomes although the experience of desirable difficulties seems to be negative for learning processes.

Despite learners who experienced desirable difficulties felt more pressure by task, were unsatisfied with

learning process, and less concentrated on learning, the individuals achieved similar result compared to

learners who have not experienced desirable difficulties. The findings may imply that desirable

difficulties have possibilities for facilitating learners in some way. Thus, it is necessary to investigate how

the learners get a benefit from desirable difficulties and how the experience of desirable difficulties can

be designed to improve learning in further study.

213

REFERENCES

Dillenbourg, P. (2002). Over-scripting CSCL: The risks of blending collaborative learning with

instructional design. In P. A. Kerschner(Eds.), Three worlds of CSCL: Can we support CSCL? (pp.

61-91). Heerlen: Open University of the Netherlands.

Fischer, F., Kollar. I., Stegmann, K., & Wecker, C.(2013). Toward a script theory of guidance in CSCL.

Educational psychologist, 48(1), 56-66.

Hyojeong, jeong, & Hyewon Kim(2012). An Exploratory Validation for the Constructs of Collaboration

Load. Journal of Educational Technology. 28(3), 619-640.

Kapur, M.(2008). Productive Failure. International Journal of Cognition and Instruction. 26:379-424.

Kapur, M., & Kinzer, C. K.(2009). Productive failure in CSCL groups. International Journal of

Computer-Supported Collaborative Learning, 4, 21-46.

Kapur, M., & Bielaczyc, K.(2012). Designing for productive failure. J. of the learning sciences. 21, 45-83.

Makitalo, K., Weinberger, A., Hakkinen, P., & Fischer, F.(2005). Online collaborative learning: Will

collaborationi scripts reduce uncertainty? Educational technology, 45(5), 25-29.

Reiser, B. J.(2002). Why scaffolding should sometimes make tasks more difficult for learners. In G.

Stahl(Ed.) Computer support for collaborative learning: foundations for a CSCL community,

proceedings of CSCL 2002(pp.255-264), Boulder, CL, Jan. 7-11, 2002.

Weinberger, A., Stegmann, K., Fisher, F., & Mandl, H.(2007). Scripting argumentative knowledge

construction in computer-supported learning environments. In F. Fisher, I. Kollar, & J. M.

Haake(eds.), Scripting computer-supported collaborative learning(192-211). NY: Springer.

Renkl, R. & Atkinson, R. K.(2010). Learning from Worked-Out Examples and Problem Solving. In J. L.

Plass, R. Moreno, & R. Brunken(eds.), COGNITIVE LOAD THEORY(91-108). NY: Cambridge

University Press.

214

The Effects of Academic Emotions on Motivation in e-Learning

Seungho Kim

[email protected]

Visang Education

Seoul, Republic of Korea

Insook Lee

[email protected]

Professor

Sejong University


ABSTRACT

The purpose of this study is to examine the impact of students’ experienced academic emotions on

motivation in e-Learning. In terms of motivation, this study identifies meanings and importance of

students’ academic emotions in e-Learning and draws implications for instructional design. With middle

school students who enrolled an online mathematics course, the researcher measured academic emotions

that study participants experienced during the online lectures and then measured motivational factors after

online lecture by using self-reported instruments. The result indicates that correlations exist among several

academic emotions and motivational factors. Similar to advanced research, furthermore, frustration and

boredom negatively affected on student’s motivation and pride positively influenced on their motivation.

Keywords: Academic emotions, Emotions, Instructional design, Motivation, e-Learning

INTRODUCTION

Student’s academic emotions experienced in learning have not received sufficient attention of

instructional designers or educational researchers. But different types of emotion that student experience

in learning have impacts on concentration, memory, performance and learning process or achievement as

well(Cacioppo & Gardner, 1999; Lee, 2012). Especially, many motivation theories and related researches

include student’s emotional experience as major factor that influences to his/her motivation (Ainely,

2006; Linnenbrink, 2006; Meyer & Turner, 2006; Pekrun, 2006; Schutz et al., 2006). But most of the

advanced researches were conducted in face-to-face learning. e-Learning environment has different

factors that arouse student’s emotions from classroom (Hara & Kling, 2000; Ng, 2001; O’Regan, 2003;

Wegerif, 1998, etc.). Therefore we need to analyze impact of academic emotions that student experience

in e-learning process, and which elements of e-Learning environment have relevance to it.

THEORETICAL BACKGROUND

Emotion and Learning Emotion is individual and psychophysiological experience. It arises in interaction between human

being and surrounding environment (Myers, 2004). Also, human acts in a certain way under the influence

of their emotion (Linnenbrink & Pintrich, 2002). Learning situation is where learner can experience

diverse emotions. Considering the emotions’ role and function, emotions that learner experienced directly

and indirectly can effect to learning process (for example, learner’s concentration, self-regulation,

215

motivation, etc.) and academic achievement (Nummenmaa, 2007; Pekrun et al., 2002; Schutz & DeCuir,

2002).

Regarding cause of arising academic emotions, Elliot(1999), Frijda(1993), Lazarus(1991),

Pintrich(2000), Schutz & Davis(2000), Smith(1991) asserted that learner’s appraisal of learning

environment triggers his/her academic emotions and it related to their academic goal. Agreeing to this

assertion, Pekrun(2000, 2006) integrated various theories related to emotions and suggested Control-

Value Theory. Control-Value Theory assumes that learner conducts two types of cognitive appraisals

related to his/her emotional experience. One is that “can I control my academic activities or

achievement?” The other is that “This academic activities or achievement is valuable to me? (Positive or

Negative)” In short, Learner generally experiences emotion going through the process: 1) Event or

situation stimulates learner. 2) Learner conducts two types of cognitive appraisals. 3) Learner experiences

some kind of academic emotions.

e-Learning and Academic Emotions

e-Learning environment involves factors that cause learner’s emotions. Some of these factors are not

in face-to-face classroom. These relate to computer and web technology. e-Learner experiences various

emotions, because e-Learning environment has properties that arouse student’s academic emotions:

computer media, information network, web cyberspace. And emotions can be quantitatively as well as

qualitatively changed according to that how much e-Learning environment satisfies learner’s need or

expectation (Brave & Nass, 2002). O’Regan(2003) interviews 11 university students who participated in

online courses. He found that students experience frustration because internet connection and instability

of web site, complexity of web site’s structure, and so on. And students can experience anxiety, fear,

concern because time delay, submitting assignment to website, digital literacy, and so on. Also, they feel

excitement because convenience, accessibility to information for learning, connectivity to colleagues,

feedback of instructor. Pride arises during online learning due to that they make public their assignment to

website and feel tutors effectively manage their online course, get positive feedback from professor or

colleagues.

In research of emotional intelligence, there is evidence that appear importance of learner’s emotions.

Lee(2012) measured emotional intelligence of university students who participated in e-learning courses

and verified that their emotional intelligence have effect to academic achievement. In result, empathy of

emotional intelligence relevantly predicted to cognitive and attitude domain of academic achievement.

Also, comparing off-line and on-line learning, on-line cooperative learning, Kang & Goo(2007) found

that, in on-line learning, emotional intelligence significantly predicted learner’s academic achievement.

Academic Emotions and Motivation Weiner(1985) asserted attribution theory that human experiences emotion when they think about the

cause of consequence after behavior. And then, emotions influence their choice in next behavior.

Ford(1992) emphasized that learner’s emotions play a significant role in process of managing his/her

motivation. He found that academic emotions come from interaction between learners, learners and

teacher. Also, emotions play role as indicator that provides important information to motivation and

cognitive management. Pekrun have carried out many researches about emotions in learning. He assumed

that learner’s experienced academic emotions influence his/her motivation. Following the same purpose,

Pekrun et al.(2002) developed Achievement Emotions Questionnaire: AEQ) that measures learner’s

emotions(enjoyment, hope, pride, relief, anxiety, shame, anger, hopelessness, boredom) in classroom,

learning, test and examined middle school and university students’ emotion related to learning and

motivation, learning strategies, cognitive load, self-regulation, learning achievement. Analyzing the

results, they found that positive emotions and inner/external motivation have positive correlation. In

contrast, negative emotions and motivation have negative correlation. Since then, Pekrun(2006)

integrated empirical researches and theories, and suggested conceptual model that include cognitive

appraisal and learner’s emotions, academic achievement. This model appears that learner’s emotions

influence their learning strategies and cognitive resource, self-regulation, motivation (see pekrun, 2006).

These researches show that learner’s experienced emotions have relevance to their motivation, and

provide important information about their learning process.

216

RESEARCH METHODS

In that point of view, in order to investigate the impact of academic emotions (enjoyment, pride,

anxiety, frustration, boredom, and learning environment anxiety from Pekrun et al, 2002) on learning, the

current research analyzes relations between academic emotions and motivational factors

(intrinsic/extrinsic goal orientation, task value from Pintrich et al., 1991) that advanced research (Ainely,

2006; Linnenbrink, 2006; Meyer & Turner, 2006; Pekrun, 2006; Schutz et al., 2006) have suggested.

With middle school students who enrolled an online mathematics course, the researcher measured

academic emotions that study participants experienced during the online lectures and then measured

motivational factors after online lecture by using self-reported instruments. Correlation and regression

have been conducted for analysis of these data.

RESEARCH RESULTS

The result indicates that correlations exist among several academic emotions and motivational factors.

Furthermore, learning environment anxiety and frustration predict intrinsic goal orientation (R2 = 81%, p

< .01). Learning environment anxiety, frustration and pride predict extrinsic goal orientation (R2 = 72%, p

< .01) and learning environment anxiety and boredom predict task value (R2 = 52%, p < .01). Students

who participated in online mathematics course differently experienced both positive and negative

emotions to students who join in face-to-face classroom.

<Table 1>. Effect of Academic Emotions to Motivation

Motivation Academic Emotions R R2 t p

intrinsic goal orientation learning environment anxiety

0.9 0.81 18.276

* 0.00

frustration -8.321* 0.00

extrinsic goal orientation

learning environment anxiety

0.848 0.719

13.565* 0.00

frustration -9.004* 0.00

pride 3.267* 0.002

task value learning environment anxiety

0.721 0.52 9.258

* 0.00

boredom -8.459* 0.00

* p < .01

CONCLUSION

These results come from environmental properties and subjective characteristics that the online

course has. Similar to advanced research, furthermore, frustration and boredom negatively affected on

student’s motivation and pride positively influenced on their motivation. In contradistinction to advanced

research, but, learning environment anxiety negatively predicted to all factors of motivation. In our

research, learning environment anxiety related to guidance of e-learning system, stability of web site,

guidance of online course, etc. And a degree of learning environment anxiety was not serious for learning

(M = 3.91, SD = 0.96). These results present that learner who was motivated sensitively reacts to e-

Learning environment: guidance of e-learning system, stability of web site, guidance of online course, etc.

The present study results implicate that there is a need to considering learner’ emotional experience

when instructor designs e-Learning courses. Course designer checks e-Learning environment (especially,

related to technology) whether there are some problems that cause learner’s negative emotions (for

example, anxiety or frustration, anger, etc.) and resolve it. On the other hand, instructional strategies or

elements that boost learner’s positive emotions were included in e-Learning course or system (for

example, professor or colleagues’ positive feedback).

217

REFERENCES

Ainley, M. (2006). Connecting with Learning: Motivation, Affect and Cognition in Interest Processes.

Educational Psychology Review, 18(4), 391-405.

Brave, S., & Nass, C. (2002). Emotion in human-computer interaction. In J. Jacko, & A. Sears (Eds.),

Handbook of human-computer interaction (pp. 251-271). New York: Lawrence Erlbaum Associates.

Cacioppo, J. T., & Gardner, W. L. (1999). Emotions. Annual Review of Psychology, 50, 191-214.

Elliot, A. J. (1999). Approach and avoidance motivation and achievement goals. Educational Psychologist,

34, 149–169.

Ford, M. E. (1992). Motivating humans: Goals, emotions, and personal agency beliefs. Newbury Park,

California: Sage.

Frijda, N. H. (1993). The place of appraisal in emotion. Cognition and Emotion, 7, 357–387.

Hara, N. & Kling, R. (2000). Students’ distress with a Web-based distance education course: An

ethnographic study of participants’ experiences. Center for Social Informatics. Retrieved October

11, 2010, from http://www.slis.indiana.edu/CSI/wp00-01.html

Kang, Myunghee & Goo, Nahyun (2007). Predictive Validity of Emotional Intelligence on Various

Achievement in Blended Learning Environment. Research Institude of Curriculum Instruction,

11(1), 235-255.

Lazarus, R. S. (1991). Progress on a cognitive–motivational–relational theory of emotion. American

Psychologist, 46, 819–834.

Lee, In-sook. (2012). Research trends on emotional factors in the e-Learning context. International

Journal for Educational Media and Technology, 6(1), 14-22.

Linnenbrink, E. A. (2006). Emotion Research in Education: Theoretical and Methodological Perspectives

on the Integration of Affect, Motivation, and Cognition. Educational Psychology Review, 18(4),

307-314.

Linnenbrink, E. A., & Pintrich, P. R. (2002). Achievement goal theory and affect: An asymmetrical

bidirectional model. Educational Psychologist, 37, 69–78.

Myers, David G. (2004). Theories of Emotion. Psychology: Seventh Edition. New York: Worth

Publishers.

Meyer, D. K., & Turner, J. C. (2006). Re-conceptualizing Emotion and Motivation to Learn in Classroom

Contexts. Educational Psychology Review, 18(4), 377-390.

Ng, K-C. (2001). Using e-mail to foster collaboration in distance education. Open Learning, 16(2), 191–

200.

Nummenmaa, M. (2007). Emotions in a web-based learning environment. Turku: Turun yliopisto.

Doctoral thesis.

O'Regan, K. (2003). Emotion and e-learning. Journal of Asynchronous Learning Networks, 7(3), 78-92.

Pekrun, R. (2000). A social cognitive, control-value theory of achievement emotions. In J. Heckhausen

(Ed.), Motivational psychology of human development (pp. 143-163). Oxford, UK: Elsevier.

Pekrun, R. (2006). The Control-Value Theory of Achievement Emotions: Assumptions, Corollaries, and

Implications for Educational Research and Practice. Educational Psychology Review, 18(4), 315-

341.

Pekrun, R., Goetz, T., Titz, W., & Perry, R. P. (2002). Academic emotions in students' self-regulated

learning and achievement: A program of qualitative and quantitative research. Educational

Psychologist, 37(2), 91-106.

Pintrich, P. R.(2000). Multiple goals, multiple pathways: The role of goal orientation in learning and

achievement. Journal of Educational Psychology, 92, 544-555.

Pintrich, P. R., Smith, D. A., Garcia, T., & McKeachie, W. J. (1991). A manual for the use of the

motivated strategies for learning questionnaire (MSLQ). Ann Arbor, Michigan: National Center for

Research to improve Post Secondary Teaching and Learning (NCRIPTAL). The University of

Michigan, Eric Document Reproduction Service, ED 338122.

Schutz, P. A., & Davis, H. A. (2000). Emotions and self-regulation during test taking. Educational

Psychologist, 35, 243–255.

218

Schutz, P. A., & DeCuir, J. T. (2002). Inquiry on emotions in education. Educational Psychologist, 37(2),

125-134.

Schutz, P. A., Hong, J. I., Cross, D. I., & Osbon, J. N. (2006). Reflection on Investigation Emotion in

Educational Activity Settings. Educational Psychology Review, 18(4), 343-360.

Smith, C. A. (1991). The self, appraisal and coping. In C. R. Snyder & D. R. Forsyth (Eds.), Handbook of

social and clinical psychology: The health perspective (pp. 116–137). Elmsford, NY: Pergamon.

Wegerif, R. (1998). The social dimension of asynchronous learning networks. Journal of Asynchronous

Learning Networks, 2(1), 34-49.

Weiner, B. (1985). An attributional theory of achievement motivation and emotion. Psychological

Review, 92, 548-573.

219

How Do the Level of Complex Learning Task and the Part-task

Sequencing Affect on Mental Model, Cognitive Load,

and Learning Time?

Kyungjin Kim

[email protected]

Doctoral course student

Hanyang University, Korea

Dongsik Kim

[email protected]

Professor at the department of Educational Technology

Hanyang University, Korea

ABSTRACT

This study was conducted to examine effects of the level of task complexity and the method of part-

task sequencing on a learner's mental model and cognitive load in the complex learning task. Task

complexity has been divided into three levels, high, medium, and low. The method of part-task sequencing

has been categorized into two, backward chaining with snowballing and simple backward chaining. In

order to examine the effects, the Computer-based Complex Task Performance Supporting (CTPS)

program was designed and developed. This study has suggested an alternative solution to realize a

complex learning task effectively by presenting instructional design principles with different part-task

sequencing methods depending on a task complexity level in implementing a complex task.

Keywords: Complex task, Part-task sequencing, Cognitive load

INTRODUCTION

A learning task taught in the school may be deemed well-structured in general, well-defined, and

comparatively easy to learn and teach, but is far from authentic problems in a real world. On the contrary,

numerous problems we are facing out of the school are not only not-well defined, but also ill-structured.

One significant goal of learning is to apply what is learned to solve a real problem in an ordinary

routine of life. Learning a simple task may help a learner obtain fact and information which he/she has not

been aware of, but it is not enough to solve a real task which requires complex elements such as

knowledge, skill, and attitude. Therefore, nowadays, researchers emphasize to provide learners in the

learning process with experiences on a complex task (Merrill, 2002; van Merriënboer, Kester, & Pass,

2006).

Recent studies have mainly dealt with instruction materials and instructional design that reduce a

learner's extraneous load in learning a complex task (Merrill, 2002; van Merriënboer, Kester, & Pass,

2006). It simply analyzed which variable influenced more on a cognitive load or not.

In this study, we aim to suggest which part-task sequencing method is most appropriate for effective

performance of a complex task, depending on the complexity level of complex task.

CTPS program development

220

A tool for this study, the CTPS Program (Complex Task Performance Supporting Program) has

been developed. There three main reasons: First, a complex task is composed of the same level of

multiple task types, and shall be designed by using a scaffolding method in which much more

instructional supports are provided in the beginning, but gradually reduced for performing learning in

each task type. Second, a complex task needs detailed learning guidance at an initial stage of learning,

and supports must be consistently provided available anytime during learning, as required. Third, most of

online lecture sites are composed only in order which is presented in the subject, and there are almost no

such sites that have part task sequencing. If there is no standardized program in sequencing part tasks for

a complex task, its result may be distorted due to inconsistency of treatment depending on researcher's

capability. The concrete and standardized program must be developed to ascertain the validity of study.

THEORETICAL BACKGROUND

Task Sequencing The task sequencing means that it determines a task class by categorizing learning tasks from easy

one to hard one, and sequences it. This theory has recently more developed and expanded to have 10 steps

(Van Merriënboer, & Kirschner, 2007). This study will focus on their opinions on task sequencing in

terms of these views.

Whole task sequencing

High

Complexity

Simple

Backward

GABCDEF-FABCDE-EABCD-DABC-CAB-BA-A

Medium

Complexity

Simple

Backward

EABCD-DABC-CAB-BA-A

Low

Complexity Simple

Backward

CAB-BA-A

High

Complexity

Snowballing

Backward

GABCDEF-FGABCDE-EFGABCD-DEFGABC-CDEFGAB-BCDEFGA-ABCDEFG

Medium

Complexity

Snowballing

Backward

EABCD-DEABC-CDEAB-BCDEA-ABCDE

221

Low

Complexity

Snowballing

Backward

CAB-BCA-ABC

Figure 1. Illustration of Sequencing

Figure 1 is part-task sequencing methods which a researcher illustrated according to a complexity

level.

METHODOLOGY

Groups were undergraduate students enrolled in liberal arts classes of community college located at

Seoul city. Each group underwent pre-tests to confirm group homogeneity; then assigned into high-level,

medium-level, and low-level groups in terms of task complexity; and again divided into two sets of

groups depending on which part-task sequencing method was utilized. They were informed with the

functions of main menus on the CTPS program. The CTPS program was given to the groups, they were

expected to experience different learning scenario by the natures of treatment groups. Finally post-tests

were administered to measure students’ mental model development, their cognitive loads, and their

learning time.

CONCLUSION

We found three main results. First, part-tasks sequencing with the backward chaining with

snowballing would best method instructional design in complex learning. Second, when complex learning

tasks are completed through the backward chaining with snowballing, mental model development may be

intensified but vastly more so would cognitive load, with possible results of the learner avoiding a similar

task in the future or even quitting the task during. Hence, instructional design should be formatted with

strong considerations on which part-task sequencing method is adequate for the particular learner. Third,

learners performing complex tasks should be given enough time, and even more so when the complexity

level is higher. Moreover, sequencing part-tasks in the backward chaining with snowballing that requires

extraneous cognitive load would require longer learning time. Instructional design should reflect these

results to offer enough learning time to the learner.

In summary, the underpinning purpose of this study is to overcome the strategic limitations of

previously suggested instructional designs addressing complex learning and to determine which approach

to part-task sequencing would be most effective when completing complex learning tasks with varying

complexity levels. Conclusions are drawn that for elevated performance in completing complex tasks,

instructional methods should be designed to accommodate the level of Complexity part-tasks. And for

effectual mental model development, detailed instructive-learning strategies that motivate learners to

immerse in task completion should be developed and applied in practice. This study reports on the design

principles and strategies of instruction in complex learning environments with regard to learners’ mental

model, cognitive load, and learning time, which provides guideline to building alternative solutions in

instruction.

REFERENCES

Merrill, M. D. (2002). First principles of instruction. Educational Technology Research and Development,

50(3), 43-59.

222

van Merriënboer, J. J. G., Kester, L., & Paas, F. (2006). Teahing complex rather than simple tasks:

Balancing intrinsic and germane load to enhance transfer of learning. Applied Cognitive Psychology,

20, 343-352.

van Merriënboer, J. J. G., & Kirschner, P. A. (2007). Ten steps to complex learning: A systematic

approach to four-component instructional design. Mahwah, NJ: Lawrence Erlbaum Associcates.

223

Predictability of Presence on learning persistence and learning satisfaction

in Facebook-based Collaborative Learning Environment

Hyunmin Chung [email protected]

Graduate student



Sungeun Oh

[email protected]

Graduate student



Jiyoon Moon

[email protected]

Graduate student



Jeongmin Lee

[email protected]

Professor



ABSTRACT

The purpose of this study was to analyze the effect of presence (cognitive, social, and emotional) on the

persistence and satisfaction in SNS-based collaborative learning environment. 95 college students who took a course

“career education” class participated in this research. Data were analyzed by correlation analysis and multiple

regression analysis. The research findings are summarized as follows: first, emotional presence predicted significantly

learning persistence, while cognitive presence and social presence did not predict learning persistence in SNS-based

collaborative learning environment. Second, cognitive presence and emotional presence predicted significantly

learning satisfaction. On the other hand, social presence did not predict learning satisfaction in SNS-based

collaborative learning environment. The implication of this study and future research were discussed in the conference.

Keywords: Social Network Service (SNS), Social learning, Facebook, presence, persistence, satisfaction.

224

PURPOSE OF THE RESEARCH

Social Network Service, an online flatform that enables to generate and strengthen communication, information

share and social relationship between users, has been rapidly spread with a diffusion of smartphone based on its

property of universal accessibility (Pimmer, Linxen & Grohbiel, 2012). Social Network Service like Facebook or

Twitter showed a diverse use in the fields of marketing, media, politics and society (Kang, Hhan & Kim, 2012).

In a domain of educational practice and research, several positive result of study (Ajjan & Hartshorne, 2008;

Mzaman & Usluel, 2010) supported that learners could actively participate, interact, cooperate and share within the

learning environment where all characteristics of SNS, including openness, cooperation and information sharing,

function as a tool for educational purpose.

Among the available service, Facebook users were distinctive in that the majority of users are in the age of 18 to

24 who were in higher education, establishing a sense of familiarity and having an experience by using their online

social activities (Choe & Kwon, 2013). Therefore, Facebook users were expected to share their feelings under the

educational setting of SNS through daily language use. The situation allowed emotional interaction with the increase

of familiarity and friendship. Based on the previous study, the learning effect would be maximized in the process of

achieving shared learning goal in those setting, arousing positive interaction between learner and teacher as well as

learner themselves provided by Facebook interface (Lim, 2011).

The opposite study regarded Facebook as an inappropriate educational tool or defines a limit undermining its

learning effect. Selwyn (2009) revealed that University students tend to use Facebook as a space for casual talk or

social relationship management so that Facebook use was highly seen as „backstage‟ rather than as „front-stage‟.

Moreover, a number of learners preferred face to face setting to Facebook-based one (Baran, 2010) and only a few

shows their active use for learning. Those who did not want to join the learning group explained the reason for

security and privacy issues (Kop, Fournier, & Mak, 2011).

Previous studies have shown conflicting results and few investigate the effectiveness with an experimental

approach, therefore, this study empirically explored the persistence and satisfaction as learning outcome in Facebook-

based learning environment.

Moreover, undemanding condition to produce information freely and actively, to cooperate and to share for new

encourages a sense of presence (Witmer & Singer, 1998) which was defined as the subjective experience of being in a

certain place or an environment. According to Wang and Kang (2006), there were three intersecting domains of

presence in teaching-learning environment: cognitive, social, and emotional and each domain describes learning

content during learning, virtual presence of instructors or peers and self-awareness for emotion. Based on the existing

theory that high cognitive, emotional and social presences increase learning flow connected to successful learning,

this study suggested a model of presence-gaining strategies by analyzing underlying factors of learning presence in

Facebook-based collaborative learning environment.

First, do cognitive presence, social presence and emotional presence predict learning satisfaction in Facebook-

based collaborative learning environment?

Second, do cognitive presence, social presence and emotional presence predict learning persistence in Facebook-

based collaborative learning environment?

METHOD

Subjects & Procedure

225

Ninety-five students participated in this research who took career education course opened at a university in Seoul. The

course was required as academic liberal arts for the first and second year students and all participants had to use Facebook group

for collaborative group task consisted of class interaction such as materials, discussion, feedback and learning-log. Before

collaborative group task by using Facebook group, orientation session was provided.

Learning presence, satisfaction and learning persistence were measured and the results have been reliably analyzed, firstly,

each measuring instrument had item reliability established through Cronbach's α in order to verify internal consistency, secondly,

normality of the distribution was confirmed and then correlation analysis adopted to figure out the relationship between variables,

lastly, hypothetical research model was developed used for multiple regression analysis through SPSS.

Measurement

To measure learning presence, items developed by Wang and Kang (2006) were revised with the use of 5-point Likert scale.

The sense of cognitive presence consisted of 13 test items like „I am able to discuss what I learn from the course‟, 11 items for

social presence like „I feel like learn together with others‟ and emotional presence written as a question: „I easily express my

feeling on Facebook‟. Inter-item consistency had a Cronbach's α of .733 for cognitive presence, of .911 for social presence and

of .837 for emotional presence.

Also, learning satisfaction was measured with the revised version (five items, Cronbach's α = .875) of Shin & Chan (2004).

A sample of the questionnaire item as follows: “I felt a high sense of accomplishment through this course”. Lastly, learning

persistence was measured with the revised version (four items, Cronbach's α = .816) of Kim & Kang (2010) comprised of „I am

willing to take related course‟.

CONCLUSION

The purpose of this study was to analyze the effect of learning presence (cognitive, social, and emotional) whether they predict

learning outcome and investigate each variance‟s relative power of prediction in Facebook-based learning environment. To set

SNS-based collaborative learning environment, the research made use of Facebook group page and performed during 8 weeks.

The result of the study summarized as follows:

Among the sub-factors of learning presence-cognitive, social and emotional, values of two factors was meaningful;

cognitive (ß = .355, p <.05) and emotional (ß = .456, p <.05) aspects. The result showed that the higher competence for self-

cognition in learning, emotional state, the degree of expression and regulation hold, the more positive response for learning

experience followed. It was consistent with previous research that learners with cognitive presence or strategies for acquiring

cognitive presence felt more satisfaction toward their learning (Kang, Kim, Park, 2008), and the same result came from another

research about the effect of emotional presence toward satisfaction for learning (Kang, et, al, 2008).

Second, cognitive (ß = .217, p <.05) and emotional presence (ß = .152, p <.05) influenced on learning persistence. The

findings we obtained were consistent with the result that emotional presence as well as cognitive one meaningfully worked to

make learners persist on their learning (Joo, Kim, & Park, 2009), emotional presence (Kang & Kim, 2006).

Moreover, higher cognitive presence, control of emotional state and self-regulation competency were connected to learning

persistence; willingness to persist their own learning was increased. As a result, we confirmed learning persistence could be

changed depending on the cognitive and emotional presence. That is, the way to increase those factors should be regarded

importantly when designing Facebook based learning for learning persistence.

On the contrary to the fact that cognitive and emotional presence produce meaning value, social presence didn‟t obtain

meaningful result; satisfaction (ß = -.088, p >.05) and learning persistence (ß = .111, p >.05). There was a prior research

explaining that Facebook-based learning could strengthen social presence in that consolidate strong sense of solidarity and

intercommunication and connected to the outcome, comparing to the online or web-based learning environment. (Kang, Hhan, &

Kim, 2012; Omar, Embi, & Yunus, 2012) The result denied the expectancy towards the potential power of network learning.

The rationale behind the failure of prediction was that learners‟ resistance to use their profile information for teaching-

learning environment was unsuccessful to build social presence. In the light of this, Jones, Blackey, Fitzgibbon & Chew (2010)

referred to the fact that using their private account for social networking services with the same as for the learning environment

made leaners feel insecure; learners did not want to intersect the circles they involved. It suggested learners‟ scope of learning

environment needed to be separated from that of social life.

226

Moreover, a lack of understanding the concept of social presence occurred because of the number of the group member;

With 95 members on a group, the characteristic of group collaboration would be shown in a different way comparing to small

group based work. In fact, previous research about the effect of social presence to learning persistence or satisfaction confined

learning situation as small group or major class under 40. According to Jang & Chang (2013), small group discussion with 25

members took advantage of Facebook services in learning like instant responses, convenience, a sense of solidarity and

familiarity and thus, it created higher social presence, learning flow, satisfaction, learning achievement. Also, from the research

concerned with the effect of social presence. Cho, Han (2010) pointed out that social presence could successfully predict learning

achievement, participation and satisfaction. The experiment conducted on 200 learners attending a class opened at a cyber

university but divided into 10 groups to limit the number of participants in each group.

Through the previous and this research, considering social presence could be meaningful when designing SNS-based

teaching. Since a strategy-based approach enhancing social presence and its underlying factors to increase social presence-

coexistence, influences and cohesiveness effectively worked for small size group working collaboratively, it is meaningful to

apply the implication to SNS learning environment.

Finally, there are two limitations of this investigation that are important to be underscored: the data collected to measure

learning presence and missed or hidden factors related to the learner variables and learning environment. With regard to the first,

learners endorsed questionnaire items indicating their level of learning presence. If message analysis by quantitative method

implemented for the experimental design of this investigation, the result could be more reliable. Finally, as learning presence was

regarded as a complex structure, deep understanding of learning presence has to be preceded. While the research was not

designed to make this determination, it is nevertheless important to encourage further research to distinguish underlying factors to

increase learning presence and its interaction between the characteristic of instructors and leaners, the process of learning and

learning environment. Then, meaningful implication could be given for a practical and various course design.

REFERENCES

Ajjan, H., & Hartshorne, R. (2008). Investigating faculty decisions to adopt Web 2.0 technologies: Theory and empirical tests.

The Internet and Higher Education, 11(2), 71-80.

Cho, E., & Han, A. (2010). The Effect of Social Presence on Learning Flow and Learning Effects In Online Learning Community.

The Journal of Educational Information and Media, 16(1), 23-43.

Choe, J., & Kwon, S. (2013). Teaching Strategies in Learning with Facebook to Improve Learner‟s Interaction. Journal of

Lifelong Learning Society, 9(2), 155-180.

Jang, E., & Chang, H. (2013). The differences between Web-based debate and Social Network Service(SNS)-based debate on

social presence, learning flow, satisfaction and self-evaluation. Journal of Educational Technology 29(1), 1-25.

Jones, N., Blackey, H., Fitzgibbon, K., & Chew, E. (2010). Get out of My Space!. Computers & Education, 54(3), 776-782.

Joo, Y., Kim, E., & Park, S.,Y. (2009). The Structural Relationship among Cognitive Presence, Flow and Learning Outcome in

Corporate Cyber Education, The Journal of Educational Information and Media, 15(3) 21-38.

Kang, M., Hhan, J., & Kim, J. (2012). Exploring students‟ learning experiences of Facebook in Service-Learning. The Korean

Journal of Educational Methodology Studies. 24(4), 771-796.

Kang, M., Kim, J., & Park, M. (2008). Investigating Presence as a Predictor of Learning Outcomes in E-learning Environment.

In J. Luca & E. Weippl (Eds.), Proceedings of World Conference on Educational Multimedia, Hypermedia and

Telecommunications 2008 (pp. 4175-4180). Chesapeake, VA: AACE.

227

Kang, M., & Kim, M. (2006). Investigation the Relationship among Perceived Social Presence, Achievement, Satisfaction and

Learning Persistence in e-Learning Environment. Journal of Educational Technology, 22(4), 1-27.

Kang, M., Park, M., Jung, J., & Park, H. (2009). The Effect of Interaction and Learning Presence on Learning Outcome in Web-

Based Project Learning. The Journal of Educational Information and Media, 15(2), 67-85.

Kim, J., & Kang, M. (2010). Structural Relationship among Teaching Presence, Learning Presence, and Effectiveness of e-

Learning in the corporate setting. Asian Journal of Education. 11(2). 29-56.

Kop, R., Fournier, H., & Mak, J. S. F. (2011). A pedagogy of abundance or a pedagogy to support human beings? Participant

support on massive open online courses. International Review of Research in Open and Distance Learning, 12(7), 74-93.

Lim, K. (2010). A Case Study on a Learning with Social Network Services on Smartphones: Communication Contents and

Characteristics Analyses of the Applications. The Korean Journal of Educational Methodology Studies. 22(4), 91-114.

Omar, H., Embi, M. A., & Yunus, M. M. (2012). ESL Learners‟ Interaction in an Online Discussion via Facebook. Asian Social

Science, 8(11), 67.

Pimmer, C., Linxen, S., & Gröhbiel, U. (2012). Facebook as a learning tool? A case study on the appropriation of social network

sites from mobile phones in developing countries. British Journal of Educational Technology, 43(5), 726-738.

Selwyn, N. (2009). Faceworking: exploring students' education‐related use of Facebook Learning. Media and Technology, 34(2),

157-174.

Shin, N. M., & Chan, J. (2004). Direct and indirect effects of online learning on distance education. British Journal of

Educational Technology, 35(3), 275-288.

Wang, M. J., & Kang, M. (2006). Cybergogy for engaged learning: A framework for creating learner engagement through

information and communication technology. In M. S. Khine (Ed.), Engaged Learning with Emerging Technologies (pp.

225-253): New York: Springer Publishing.

Witmer, B. G., & Singer, M. J. (1998). Measuring presence in virtual environments: A presence questionnaire. Presence.

Teleoperators and virtual environments, 7(3), 225-240.

228

A Case Study of The Need Assessment and Usability Issue

in Designing and Developing e-book

Hyungju Lee

[email protected]

Undergraduate Student

Chonnam National University (Gwangju, Korea)

SinOk Kim

[email protected]



Gwansun Hong

[email protected]



SanHa Kang

[email protected]



JeongAh Woo

[email protected]



YooJin Hong

[email protected]



ABSTRACT

The goal of this study is to assess the needs of international students and to evaluate the usability in

designing and developing e-book. The e-book is designed for the first arrival students at Chonnam

National University (CNU). This study will apply survey, interview and pilot-test with small sample size

to meet the purpose of e-book design so the result of study may not be appropriate for generalization.

However, this study will provide context-driven experience of designing and developing e-book for

international students. The value of this study will focus on what needs and usability issues should be

considered for the multicultural readers. Questions were made based on experience with international

students who visited CNU as exchange students. The study is only aimed at international students who

study at CNU in Gwangju, Korea.

The e-book covers the mainly campus life, culture differences, and hot places for lunch and dinner

near the university. That will be used for international students studying at CNU, as it contains a number

of contents about campus life of CNU.

This study will proceed processes of needs assessment and usability evaluation. These are the main

components in the interface of this research. First, needs assessment will be conducted to identify

international students' needs, desired conditions or wants. The assessment is to measure the actual

students' needs specifically and to shed some light on the e-books' contents which are suitable. It can also

improve the quality of e-book by containing international students' needs. Second, usability evaluation is

important to increase the utility of e-book. The evaluation depends on familiarity which the students has

with it. The less familiar the e-book is, the less usable it is. By performing a usability evaluation with

familiarity, the e-book will be optimized for international students..

Keywords: need assessment, usability, e-book

229






INTRODUCTION

An e-book is “an electronic version of a printed book which can be read on a personal computer or

hand held device designed specifically for the purpose” (K.T. Anuradha, 2006). As smart technology

becomes essential in daily life, students are no longer constrained by time and location. They can access

and share information anytime and anywhere. The number of international students at CNU has steadily

increased in recent years (Yang, 2009). Many have to spend weeks on becoming familiar with this

unfamiliar environment. As strangers, a personal and portable location-based service can help them to be

accustomed to their new environment and lead them to interesting sites. Although CNU has provided a

campus guide in PDF format and included many things for the students to feel more comfortable on the

campus, they have difficulties in using the facilities at CNU, treating their peers in class and selecting

restaurants around CNU owing to the lack of information. This study is the try to solve those problems by

applying „contiguity principle‟ and using „e-book‟.

METHOD

The target of this survey was the international students at CNU. The questionnaire was intended to

elicit information for needs assessment. The survey for needs assessment organized two sections. In the

first section, there were multiple questions about CNU’s facilities and functions. It also contained the

question about culture in class such as which one was more difficult when they treat or converse with,

Korean professors or students. In the next section, the participants answered which factors were

preferable or uncomfortable when they chose their food or restaurants. Off-campus life team used „Likert

scale‟ to measure the preference. After needs assessment, each team analyzed their data and proceeded

four steps, brainstorming, storyboarding, paper modeling, and developing e-book. The e-book was

applied „contiguity principle‟ for design and development.

RESULT

Needs assessment In the first section, the participants taking part in the survey were thirty three. In the second

section, the participants partaking in the survey were thirty two. The participants joining in the survey

were almost equal. The needs assessment discovered by on-campus life team was 1) the most visited

areas on campus, 2) the difficulty in treating Korean friends. The participants answered that B, C and D

section were the most visited areas. Each section is an area of CNU. The team tried to assume the reason

why the respondents visited those areas. A section was excluded when on-campus life team analyzed the

data because there are not many available facilities. The respondents probably went to B section because

there was a big lecture hall taking place many lectures. In addition to, the participants frequently visited B

section because there was a student dormitory in B section. If the participants had a free time to do

something, they would often go to D section because there was a playground to exercise. Interestingly,

there was little gap between C section and D section in the responses. The gap of the responses is just 0.3

percentage. In C section, there was the student union which has multiple functions for international

students to use such as medical checkup, post office and restaurants. The second assessment discovered

by the team was the difficulty in classroom culture. The students responded that they faced the difficulty

in treating their Korean friends rather than Korean professors. Comparing that 7 participants usually or

always met difficulty in treating Korean professors, 12 participants usually or always felt difficulty in

treating Korean students.

The needs assessment found by off-campus life team was that 1) taste and service were

important factors when they chose food and restaurants and 2) restaurants didn‟t give enough information

230

such as food ingredients, menu and tip. The respondents regarded taste and service as the important

factors. According to the Table.1, the mean of taste was 4.72 and that of service was 4.59. Interestingly,

the mean of distance was only stayed at 3.59 on Table.1. The respondents didn‟t consider how far the

restaurants are. The second assessment was that the participants thought the restaurants didn‟t provide

enough information. When off-campus life team gave the question, „Do you think the information about

food or restaurants was offered sufficiently?‟, fifteen of participants answered that they thought the

restaurants didn‟t give much information.

Development The development process consisted of 4 steps. In the first step, brainstorming, each team

discussed and decided contents, cognitive load theories, expectation for needs assessment in designing

and developing the e-book. The second step was storyboarding. After each team made non-example and

example, every team compared example applying „contiguity principle‟ with non-example not applying

„contiguity principle‟. The third step was paper modeling. Based on the storyboarding, each team made

hand-made book specifically before developing e-book. The final step was to develop and design e-book.

Although each team separately conducted needs assessment and the development of the contents of e-

book, the contents were finally merged. The contents of the e-book consisted of two parts and types. The

first part was information about Restaurants around CNU. The second part was the structure of the

student union. The teams developed the e-book into two types, but they couldn‟t evaluate the usability of

non-example and example.

<Table 1>. Preference of food & restaurant (n=32)

Options Mean SD

Service 4.59 0.50

Taste 4.72 0.52

Quality 4.53 0.57

Quantity 3.81 0.78

Distance 3.59 0.87

Communication 3.63 0.87

Price 4.25 0.76

Ingredient 4.00 0.88

CONCLUSION

The conclusion through this study is 1) difference from expectation to needs assessment, 2) theory-

driven approach and 3) limitation: usability issues about device compatibility. Before conducting needs

assessment, on-campus life team expected that international students felt more difficulty in treating

Korean professors than Korean students. However, that expectation didn‟t match needs assessment.

Rather, the students thought that they had more difficulty in treating Korean friends. Additionally, off-

campus life team anticipated that distance was a quite important factor when they chose food and

restaurants. Likewise, on-campus life team did, the expectation was not agreed. On the other hand, the

taste and service were more significant than the distance. The teams had a question about which one is

better between „theory-driven one‟ and „the looking good other‟. Referring to „e-learning and the science

of instruction‟, the teams identified theory-driven one is better than the looking good other. As a

limitation, during the steps in developing and designing e-book, the teams faced a problem because the e-

book was not well compatible with some electronic devices and applications. Even though „Galaxy tab‟

didn‟t well operate this e-book, „iPad‟ run well the e-book. At the beginning of this study, the teams were

231

scheduled to evaluate the usability of e-book. Owing to the problem with compatibility, the teams

couldn‟t conduct the usability evaluation.

REFERENCES

Clark, R. C., & Mayer, R. E. (2008). E-learning and the science of instruction: Proven guidelines for consumers and designers of multimedia learning. Wiley

232

The Effects of Simulation Game-Based Learning

on Academic Emotions and Achievement

Yun Ha JUNG

[email protected]

Master’s Student


Seoul, Korea

Kyu Yon LIM

[email protected]

Assistant Professor


Seoul, Korea

ABSTRACT

The purpose of this study is to examine the effects of simulation game-based learning on academic

emotions (positive, negative) and achievement (factual, conceptual, procedural knowledge acquisition). Sixty-

three learners (experimental group: 32, comparison group: 31) from a high school located in Seoul, Korea were

chosen to conduct an experiment. The results of the study demonstrate that there was a significant difference

between the comparison group and the experimental group in both positive and negative academic emotion.

However, there was no significant difference between the comparison group and the experimental group in

factual, conceptual, and procedural knowledge acquisition. These findings show that simulation game-based

learning brings more positive emotion and less negative emotion during learning compared to the instructor-led

lectures, although there were no significant effects on achievement.

Keywords: Simulation game-based learning, Academic emotion, Achievement

INTRODUCTION

There have been many efforts to create more effective, efficient, and engaging learning environment by

adopting educational technology. However there is still a gap between the learning design and the students called

digital native who prefers self-directed, enjoyable, and socially-connected experience (KERIS, 2012; Prensky, 2001).

In order to reduce this gap, researchers paid attention to a simulation game-based learning as an alternative teaching

method. Many early researches have focused on a game-based learning because of its attractive and entertaining

attributes (Anolli, Mantovani, Confalonieri, Ascolese, & Peveri, 2010; Garris, Ahlers, & Driskell, 2002), and situated

learning theory has insisted that simulation can make learning more meaningful (Choi & Hannafin, 1995; Howland,

Jonassen, & Marra, 2012). Simulation game-based learning is defined as a kind of instructional form using the critical

elements of both simulation and game, such as goal-oriented activities and competitive experiences for teaching and

learning (Baek, 2006). Previous research on simulation game has reported that there was a significant effect on

learners’ achievement (Akinsola, 2007; Sowunmi & Aladejana, 2013) and attitude toward learning (Akinsola, 2007).

However, there are some exceptions according to the types of games and instructional context (Sitzmann, 2011),

which requires further investigation on simulation game.

In this particular study, academic emotion has been selected as a major variable or interest, since students in

Korea have been struggling with academic stress due to highly competitive environment. According to Kim (2009),

academic emotion is an emotional state caused by specific subject of stimulation composed of enjoyment, pride,

angry, anxiety, and boredom. Many researchers who studies emotions claimed that academic emotions have

important function to affect student's motivation and learning strategy use to construct knowledge (Do, 2008; Yang &

Kim, 2010). Especially in the domain of game-based learning, researchers theoretically insisted that simulation game

tends to increase positive emotions such as enjoyment and pride, while reducing negative emotions such as angry,

233

anxiety, and boredom (Anolli et al, 2010; Astleitner & Leutner, 2000; Novak & Johnson, 2012). Based on this

theoretical approach, it is necessary to perform empirical study on academic emotion and simulation game-based

learning.

Therefore, this study aimed to examine the effects of a simulation game-based learning on academic emotions

and achievements. Specifically, the use of simulation game-based learning was the independent variable

(experimental group: simulation game-based; comparison group: lecture-oriented), while academic emotion (positive,

negative) and achievement (factual, conceptual, procedural knowledge) were dependent variables. Eventually, the

study results will provide practical implications for designing simulation game-based learning as well as theoretical

justifications for game and academic emotions.

Research questions 1. Are there any differences in academic emotions (positive, negative) between simulation game-based learning

group and traditional lecture-oriented learning group?

2. Are there any differences in achievement (factual, conceptual, procedural knowledge) between simulation game-

based learning group and traditional lecture-oriented learning group?

METHODS

Participants, 12 fonts, Bold) Sixty-three students from a high school located in Seoul, Korea were chosen through convenient sampling.

Among these, 32 were assigned to the experimental group with simulation game-based learning treatment, and 31

were assigned to the comparison group with traditional lecture-oriented treatment. Prior to the intervention,

researchers confirmed the homogeneity of the two groups in terms of prior level of academic emotions as well as

prior knowledge, meaning that there were no significant differences in both dependent variables between

experimental and comparison groups (pre-positive academic emotion: t = .057, p = .955; pre-negative emotion: t = -

1.481, p = .144; prior knowledge: t = -1.735, p = .088).

Treatments Basically, the content and the instructor were identical for both groups. The only difference was the practice

session: Simulation game-based group used simulation game, while lecture-oriented group used paper-pencil-based

exercise. The treatment used for the experimental group was a simulation game called 'Mock stock investment

simulation game' designed by the Bank of Korea. This game allowed students to virtually buy and sell stocks for 40

companies. Every single stock price was reflected by real stock trading market, and the return on investment made by

game players was calculated and announced every midnight. The instructor was able to control how much money the

students can make investment during the game, and the research participants were given 5,000,000 won at the

beginning of the simulation game.

Measurement instruments This study used three measurement instruments. First, in order to measure academic emotions, Academic

Emotion Questionnaire developed by Pekrun et al. (2011) was adopted. Positive academic emotions consisted of

items relevant to enjoyment and pride, and negative academic emotions consisted of items relevant to anger, anxiety,

and boredom. The Cronbach’s alphas calculated using the study data were .95 for positive emotion, and .94 for

negative emotion. Second, prior knowledge test used for checking the homogeneity of the two groups was developed

by two subject matter experts. Third, measurement instruments for the achievement for factual, conceptual, and

procedural knowledge acquisition were also developed by two subject matter experts. Prior knowledge test and

achievement test items were validated by another subject matter expert. Examples of measurement instruments are

illustrated in <Table 1>.

<Table 1> Measurement instruments

Variables Examples Scale # of

items

Academic

emotions

Positive Enjoyment: I enjoy being in class.

Pride: I am proud of the contribution I have

made in class.

Likert

5 16

234

Negative Anger: I feel anger welling up in me.

Anxiety: I worry the others will understand

more than me.

Boredom: Because I get bored, my mind begins

to wander.

24

Achievement Factual

Knowledge What is the appropriate terminology for

‘securities’?

10

points 10

Conceptual

Knowledge Which is incorrect about economic variables

and stock market?

10

points 10

Procedural

Knowledge What is the correct procedure to buy and sell

stocks in the market?

5

points 5

Procedure (Times New Roman, 12 fonts, Bold) Firstly, researchers assigned participants into experimental and comparison groups. Then all the participants

were provided with a 30-min orientation session for the research, and took pre-academic emotion questionnaire and

prior knowledge test. Secondly, each group participated in the intervention for 9 days, respectively. Experimental

group played a mock stock investment simulation game under the guidance of the instructor during their two 80-min

face-to-face sessions, and also played the game as homework by themselves. On the other hand, comparison group

received face-to-face lectures given by the same instructor, same learning contents, and also did paper-pencil-based

homework by themselves. Lastly, after the treatments completed, participants responded to the academic emotion

questionnaire and achievement test. The data were analyzed using independent sample t-test and MANOVA.

RESULTS

Effects on academic emotions Bold MANOVA on academic emotions revealed that there is a significant difference between the experimental group

and the comparison group in both positive and negative academic emotion (Wilks' = .820, P = .003). <Table 2>

summarizes the descriptive statistics and MANOVA results for two experimental conditions.

<Table 2> Descriptive statistics and MANOVA results on academic emotions

(n = 63)

Variable Treatment M SD F P η2

Academic

emotion

Positive Simulation game 4.09 0.639

10.162 .002* .143 Lecture 3.48 0.862

Negative Simulation game 2.27 0.624

9.134 .004* .130 Lecture 2.75 0.648

* p < .05

Effects on an Achievement MANOVA on achievement revealed that there are no significant differences between the experimental group

and the comparison group in factual, conceptual, and procedural knowledge (Wilks' = .977, P = .709). Descriptive

statistics and MANOVA results for two experimental conditions are described in <Table 3>.

<Table 3> Descriptive statistics and MANOVA results on achievement

(n = 63)

Variable Treatment M SD F p η2

Achievement

Factual Simulation game 7.22 1.773

.172 .680 .003 Lecture 7.42 2.062

Conceptual Simulation game 5.28 2.453

.095 .759 .002 Lecture 5.10 2.286

235

Procedural Simulation game 1.84 1.019

.585 .447 .009 Lecture 2.06 1.263

* p < .05

CONCLUSION

This study examined the effects of a simulation game-based learning on academic emotions and achievements.

In conclusion, the findings show that simulation game-based learning brings more positive emotion and less negative

emotion during learning compared to the instructor-led lectures, although there were no significant effects on

achievement.

In terms of academic emotions, similar to the previous studies, the results indicated that the simulation game-

based learning tends to bring more task values and more sense of control for students, which are relevant to the

positive emotions (Pekrun, 2006). In other words, there is a possibility that simulation game increased learners’

perception on task values because of its reality, and also provided a sense of control because of its participant-

oriented procedures (e.g. entering orders of selling and buying stocks). The results implied that these attributes

enabled by the simulation game are likely to create positive emotional environment. In terms of achievement, there is

a possibility that students had a cognitive overload because of its large amount of information to invest stocks (Sun &

Choi, 2013). In order to generate meaningful learning with simulation game, it is required for learners more time for

constructing knowledge. Although there was no significant effect on achievement, it still means that game-based

learning is surely more effective in the emotional aspect.

Limitations and implications for future research are suggested as follows: First, participants for this study were

sixty-three high school students, in Seoul, Korea, which requires caution for generalizing the results. Second, follow-

up studies need to further explore specific emotions such as pleasure, pride, anxiety, and boredom as separated

dependent variables, instead of using positive and negative categories. Third, it is necessary to consider other key

independent variables such as context of instruction and collaboration during simulation game-based learning.

REFERENCES (Times New Roman, 12 font, Bold, All Capital)

Akinsola, M. K. (2007). The effect of simulation-games environment on students achievement in and attitudes to

mathematics in secondary schools. The Turkish Online Journal of Educational Technology, 6(3), 113-119.

Anolli, L., Mantovani, F., Confalonieri, L., Ascolese, A., & Peveri, L. (2010). Emotions in serious games: From

experience to assessment. International Journal of Emerging Technologies in Learning, 5(3), 7-16.

Astleitner, H., & Leutner, D. (2000). Designing Instructional technology from an emotional perspective. Journal of

Research on Computing in Education, 32, 497-510.

Back, Y. K.(2006). Understanding and application of Game based learning. Seoul: Educational science.

Choi, J. I., & Hannafin, M. (1995). Situated cognition and learning environment: Roles, structures, and implications

for design. Educational Technology Research & Development, 43(2), 53-69.

Do, S. L. (2008). Issues and prospects of research on affect in education. The Korean Journal of Educational

Psychology, 22(4), 919-937.

Garris, R., Ahlers, R., & Driskell, J. E. (2002). Games, motivation, and learning: A research and practice model.

Simulation & Gaming, 33(4), 441-467.

Gredler, M. E. (2004). Games and simulations and their relationships to learning. In D. H. Jonassen (Ed), Handbook

of research on educational communications and technology (pp. 571-581). Mahwah, NJ: Lawrence Erlbaum

Associates.

Howland, J. L., Jonassen, D., & Marra, R. M. (2012). Meaningful learning with technology (4th Ed.). Boston, MA:

Pearson Education.

KERIS (2012). Adapting education to the information age. Seoul, Korea: KERIS.

Kim, M. S. (2009). Emotion in Learning Context: Its Origins and Functions. Asian Journal of Education, 10(1), 73-

98.

Lindh, J., Hrastinski, S., Bruhn, C., & Mozgira, L. (2008). Computer-based business simulation games as tools for

learning: A comparative study of student and teacher perceptions. In Proceedings of the 2nd European

conference on games-based learning (ECGBL), 16–17 October 2008, Barcelona, Spain.

Novak, E., Johnson, T. E. (2012). Assessment of student's emotions in game-based learning. In D. Ifenthaler, D.

Eseryel., & X. Ge (Eds.), Assessment in game based learning: Foundations, innovations, and perspectives (pp.

379-399). NY: Springer.

236

Pekrun, R. (2006). The control-value theory of achievement emotions: Assumptions, corollaries, and implications for

educational research and practice. Educational Psychology Review, 18, 315–341.

Pekrun, R., Goetz, T., Frenzel, A. C., Barchfeld, P., & Perry, R. P. (2011). Measuring emotions in students’ learning

and performance: The academic emotions questionnaire (AEQ). Contemporary Educational Psychology 36, 36-

48.

Pekrun, R., Goetz, T., Frenzel, A. C., & Perry, R. P. (2011). Academic Emotions Questionnaire (AEQ). User's

manual (2nd ed.). Munich, Germany: Department of Psychology, University of Munich.

Prensky, M. (2001). Digital Game-Based Learning. NY: McGraw-Hill.

Sitzmann, T. (2011). Meta analytic examination of the instructional effectiveness of computer based simulation

games. Personnel Psychology, 64, 489-528.

Sowunmi, O., & Aladejana, F. (2013). Effect of simulation games and computer assisted instruction on performance

in primary science. Proceedings of the 2013 WEI International Academic Conference, USA, 10-15.

Sun, J, S., & Choi, W. (2013). An inquiry for the utilization strategies of cognitive tools in web-based PBL(Ploblem-

Based Learning) and an analysis of the effects on cognitive load. Journal of Educational Technology, 29(2),

349-382.

Yang, M. H., & Kim, E. J. (2010). The influence of emotional regulation on learning strategy: Mediated by

emotionality. The Korean Journal of Educational Psychology, 24(2), 449-467.

237

The Effects of Part-task Sequencing and the Level of

Element Interactivity on Schema Automation and Cognitive Load

Hyejeong Lee

[email protected]

Graduate Student

Hanyang University

Seoul, Korea

Dongsik Kim

[email protected]

Professor

Hanyang University

Seoul, Korea

ABSTRACT

This study is to reveal the most effective part-task sequencing according to the level of element

interactivity, and to find the appropriate part-task sequencing best suit for human cognitive structure. A

two-way ANOVA with schema automation and cognitive load as dependent variables, and the kind of

part-task sequencing and the level of element interactivity as independent variables was conducted. The

study results show that progressive chaining and snowballing chaining outperformed forward chaining

with regard to schema automation. However, there was no significant difference between progressive

chaining and snowballing chaining. Regarding cognitive load, higher intrinsic cognitive load was occurred

when the element interactivity is high compared to low element interactivity, regardless of the kind of

part-task sequencing. The lowest level of extraneous cognitive load and the highest level of germane

cognitive load were imposed when using progressive chaining.

Keywords: Part-task sequencing, Element interactivity, Cognitive load

INTRODUCTION

Element interactivity Complex learning task is characterized by high element interactivity, which means that learner have

to deal with a large number of interacting elements simultaneously. Element interactivity is described as

one of the sources of intrinsic cognitive load which closely related to the nature of learning task itself. It

was supposed that intrinsic cognitive load is relatively difficult or even impossible to be altered in

comparison with extraneous cognitive load (Sweller & Chandler, 1994; Sweller, van Merrienboer, & Paas,

1998). Thus, earlier studies regarding cognitive load mainly focused on reducing extraneous cognitive

load for effective instructional design (van Merrienboer & Sweller, 2005).

Pollock, Chandler, and Sweller (2002) were known as the first to put an effort to reducing intrinsic

cognitive load by sequencing method. They presented complex task through two phases which are

isolated element phase and interacting element phase: firstly learners studied individual information of the

whole complex task; in the second phase, learners were presented all the information elements at once.

The result of this study indicates that isolated-interacting elements technique is a useful instructional

238


mailto:kimdsik@

sequencing method, especially for novice learners who suffer from high cognitive load due to the

limitation of their working memory.

There are some other researches that have investigated the effect of sequencing strategy on

decreasing intrinsic cognitive load (Clarke, Ayres, & Sweller, 2005; van Merriënboer, Kirschner, &

Kester, 2003). In the line of these kinds of efforts, this study also examined part-task sequencing as an

effective instructional technique for high element interactivity.

Part-task sequencing Part-task sequencing approach was initially proposed because it is hard to start learning with whole

authentic tasks which represent high complexity (Salden, Paas, & van Merrienboer, 2006). Human

working memory has a limited capacity for cognitive processing. If highly complex task is given, the

learner might feel overloaded, which can have a negative impact on learning (Sweller et al., 1998). Part-

task sequencing can decrease working memory load by decomposing a large multicomponent skills into

several separate components and reducing the complexity of the task.

Part-task sequencing is an effective and efficient method when learning complex task(Wickens,

Hutchins, Carolan, & Cumming, 2013). By breaking the complex task into individual elements, learners

have only to learn certain amounts of elements that can be processed in working memory at once, thus

intrinsic cognitive load would be decreased accordingly. Although part-task sequencing can be productive

for reaching isolated, specific objects, it does not always increase transfer when learners deal with whole

integrated objectives (van Merriënboer & Kirschner, 2007).

To increase learner’s understating and transfer over the complex task, learners should practice

whole elements together to learn how all relevant elements interact (Pollock et al., 2002). Snowballing

technique can be complement to such part-task sequencing as snowballing gives learners the opportunity

to practice whole complex task (van Merriënboer & Kirschner, 2007).

Part-task sequencing with snowballing technique look somewhat similar to Pollock et al. (2002)’s

‘isolated-interacting effect’. Even though Pollock et al. (2002)’s study gives us clear suggestion that

sequencing from isolated elements to interacting elements is advantageous for complex learning task, it

does not establish the exact guidelines indicating which ways can be effective when there are more than

two parts of isolated elements.

In this study, two part-task sequencing strategies within isolated-followed-by-interacting-elements

approach were examined to find out how different sequencing methods affect schema automation and

cognitive load when task complexity is converted into three isolated elements. Also this study was

investigated whether the effect of part-task sequencing strategies can be changeable according to the level

of element interactivity.

RESEARCH QUESTIONS

RQ1. To what extent do different part-task sequencing methods (forward, snowballing and

progressive chaining) and element interactivity (high and low) have an effect on participants’ schema

automation?

RQ2. How do different part-task sequencing methods (forward, snowballing and progressive

chaining) and element interactivity (high and low) affect cognitive load?

METHODS

Participants and Research Design To answer these research questions, a 2x3 factorial test design with six experimental conditions was

used. Participants in this study are 111 freshman students taking the course of Practical Training Methods

at Hanyang Women’s University. They were randomly assigned to one of the six experimental conditions

(see Table 1).

239

<Table 1>. Design of the experimental study

Part-task sequencing

Forward Snowballing Progressive

Element

interactivity

Low Group 1 (n=19) Group 2 (n=18) Group 3 (n=18)

High Group 4 (n=19) Group 5 (n=17) Group 6 (n=20)

Task and Procedure The students’ task was to draw various objectives using presentation software. Low element

interactivity groups were asked to draw two-dimensional objectives and high element interactivity groups

were asked to draw three-dimensional objectives. All instructions were presented in the form of softcopy

to standardize the procedure.

The experiment lasted for total four hours. Except the last 30 minutes which is assigned to test

learner’s performance and cognitive load, three hours and a half were designed to give leaners enough

time for practice.

Dependent Variables Dependent variables were learner’s schema automation and cognitive load. Schema automation was

measured by performance accuracy and time taken to complete the task. Performance accuracy and time

are the main key to evaluate to what extent to learners reach to schema automation. To assess the

performance accuracy, checklist was used.

Cognitive load was measured by 5-point self-rating questionnaires. Few studies exist measured

cognitive load by splitting it into extraneous, intrinsic, and germane cognitive load. This study measured

each cognitive load separately.

RESULTS

Schema automation A two-way ANOVA with performance accuracy as a dependent variable, and the kind of part-task

sequencing and the level of element interactivity as independent variables revealed a significant large-size

effect of the kind of part-task sequencing on performance accuracy (F(2,105)=18.94, p=.00). A post hoc

Tukey test showed that the forward group and the snowballing group differed significantly at p=.00 and

the forward group and the progressive group also differed significantly at p=.00. However, there was no

statistically difference between low element interactivity and high element interactivity (p=.82).

A two-way ANOVA showed a significant effect of the kind of part-task sequencing on the time

taken to complete a task by learners (F(2,105)=5.22, p<.01). A post hoc Tukey test showed that the forward

group and progressive group differed significantly at p<.01). There was also significant part-task

sequencing x element interactivity interaction effects (F(2,105)=8.91, p=.00).

Cognitive Load A two-way ANOVA was conducted to explore the effect of the kind of part-task sequencing and the

level of element interactivity on cognitive load.

Firstly, this analysis revealed a significant effect of the level of element interactivity on intrinsic

cognitive load (F(1,105)=12.08, p<.01). Second, a significant effect of the kind of part-task sequencing on

extraneous cognitive load was found (F(2,105)=3.76, p<.05). To find out where the differences line, a

Tuke’s post hoc was run. The forward group and the progressive group differed significantly at p<.05.

Third, with regard to germane cognitive load, there was a significant medium-size effect of the kind of

part-task sequencing (F(2,105)=3.34, p<.05). A post hoc Tukey test showed that the forward group and

progressive group differed significantly at p<.05).

240

DISSCUSION AND CONCLUSION

This study investigated the effect of different part-task sequencing and the level of element

interactivity on schema automation and cognitive load. To summarize the results, progressive chaining

and snowballing chaining outperformed forward chaining with regard to schema automation. However,

there was no significant difference between progressive chaining and snowballing chaining. Regarding

cognitive load, higher intrinsic cognitive load was occurred when the element interactivity is high

compared to low element interactivity, regardless of the kind of part-task sequencing. The lowest level of

extraneous cognitive load and the highest level of germane cognitive load were imposed when using

progressive chaining.

The results from this study provide same evidence with Pollock et al. (2002)’s study that isolated-

followed-by-interacting approach is effective to learners’ schema automation. However, the results cannot

tell us which sequencing strategy is better than others when considering isolated elements are divided into

more than two parts. Snowballing and progressive chaining does not make any significant difference. It

might be because that these two methods are all powerful technique for learning high element

interactivity.

REFERENCES

Clarke, T., Ayres, P., & Sweller, J. (2005). The impact of sequencing and prior knowledge on learning

mathematics through spreadsheet applications. Educational Technology Research and

Development, 53(3), 15-24.

Pollock, E., Chandler, P., & Sweller, J. (2002). Assimilating complex information. Learning and

instruction, 12(1), 61-86.

Salden, R. J., Paas, F., & van Merrienboer, J. J. (2006). A comparison of approaches to learning task

selection in the training of complex cognitive skills. Computers in human behavior, 22(3), 321-

333.

Sweller, J., & Chandler, P. (1994). Why some material is difficult to learn. Cognition and instruction,

12(3), 185-233.

Sweller, J., van Merrienboer, J. J., & Paas, F. G. (1998). Cognitive architecture and instructional design.

Educational psychology review, 10(3), 251-296.

van Merriënboer, J. J., & Kirschner, P. A. (2007). Ten steps to complex learning: A systematic approach

to four-component instructional design: Mahwah, NJ: Erlbaum.

van Merriënboer, J. J., Kirschner, P. A., & Kester, L. (2003). Taking the load off a learner's mind:

Instructional design for complex learning. Educational psychologist, 38(1), 5-13.

van Merrienboer, J. J., & Sweller, J. (2005). Cognitive load theory and complex learning: Recent

developments and future directions. Educational psychology review, 17(2), 147-177.

Wickens, C. D., Hutchins, S., Carolan, T., & Cumming, J. (2013). Effectiveness of Part-Task Training

and Increasing-Difficulty Training Strategies A Meta-Analysis Approach. Human Factors: The

Journal of the Human Factors and Ergonomics Society, 55(2), 461-470.

241

Effect of Conversational Gesture of Pedagogical Agent and Visual

Cueing on Task Comprehension and Eye Fixation in Types of

Information Formats

Jewoong Moon

[email protected]

Master Student

Dept. of Education, College of Education

Gwangju, Republic of Korea

Jeeheon Ryu

[email protected]

Associate Professor

Dept. of Education, College of Education

Gwangju, Republic of Korea

ABSTRACT

This study is to investigate the interaction effect of conversational gesture of pedagogical agent and

visual cueing on learning comprehension and eye fixation time. Pedagogical agent is a virtual character to

help students in multimedia environment. It can motivate learners to engage social interaction. Especially,

conversational gesture can play a crucial role to facilitate the interaction between learners and contents.

Because it provide a realistic human-like movement, learners can get higher social expectancy. Visual

cueing provide deictic information to allocate visual attention for students. It is non-content stimulus how

right direction learners can follow sequential process in learning visually. The participants of study were

sixty-four college students (male=19, female=45). The independent variables were presence of

conversational gesture and visual cueing, the dependent variables were comprehension scores and eye

fixation time. The results revealed that there were main effect (F=7.44, p=.008) and interaction effect

(F=4.57, p=.037) on comprehension score in text formats. The conditions not including gesture

outperformed than those groups including gesture. Furthermore, the condition of gesture and visual cueing

had lowest learning outcome. Rather, the condition only applied visual cueing has highest scores on

comprehension. It mean that visual cueing had a role for attentional guidance for learning. Rather, the

condition including both of gesture and visual cueing stimulated redundancy effect on learning

comprehension. In case of graphic formats, no significant difference by independent variables. Moreover,

it also had significant main effects on eye-fixation by gesture in text format (F=9.95, p=.003) and graphic

format (F=6.27, p=.015). As a result, the conditions except gesture had longer fixation time in

information area. Through these results, gesture of pedagogical agent had negative effect on dependent

variables. Because it might foster split-attention effect on visual attention. It could hinder sequential

learning process. Moreover, there were different results by types of information format. It might be due to

different mental representation with each formats.

Keywords: Pedagogical agent, Gesture effect, Visual cueing, Types of information format.

INTRODCUTION

Pedagogical agent is a cartoon-like to give instructional guidance for learners. Conversational

gesture of agent can foster more social interaction with learners. Because it can show realistic movement

242

like real human. Learners can easily perceive more social presence to animated character. But it could

also trigger distractions since that objectives show fancy stuffs visually beside instructional content. For

reducing split-attention effect by distraction, visual cueing can be applied to agents for allocating right

cognitive process. Visual cueing can guide learners how they should see the content sequentially. In line

with this arguments, types of information formats might also have differential results on using

pedagogical agents. Because text and graphic formats have different cognitive process. For exploring

these sequences, eye-tracking experiment were needed. In eye-mind hypothesis, eye-tracking study can

show human cognitive process through visual movement. Moreover, it can also provide unobtrusive result

for study.

Pedagogical Agent and Conversational Gesture Pedagogical agent is virtual object to provide guidance for learners. For more social interaction

between learners and animated agents, gesture was one of the necessary factors in communication.

Specifically, conversational gesture could provide affective context of communication with learners.

(Frechette & Moreno, 2010). It caused learners to engage learning environment with agent. Gesture is

also used as memory storage facilitating cognitive process, it required less cognitive capacity because it

reduces load automatically (Goldin-Meadow, Nusbaum, Kelly, & Wagner, 2001).

Visual Cueing Visual cueing was the addition of non-content information that captures attention to those objects

that are important in an complex materials (de Koning, Tabbers, Rikers, & Paas, 2007). In cognitive load

theory, if instructional material which are composed of pictorial and text information were allocated

separately, cueing strategy could reduce extraneous load by allocating visual attention. By leveraging

cognitive capacity for germane load, more elaborated mental models would be constructed. For

measuring visual attention, eye-tracking study might be useful tools for unobtrusive result on cognitive

process (Jarodzka, Scheiter, Gerjets, & Van Gog, 2010).

Types of Information Formats In multimedia learning, text and graphic information were normally delivered from two sensory

modality. According to Schnotz and Bannert (2003)’s cognitive model, text and graphical information has

different cognitive process. While text information was extracted from sematic and propositional meaning,

graphic one focused on intuitive perception visually. They classified two types of formats which are

depictive versus descriptive characteristics (Hochpöchler et al., 2012). Text as descriptive features were

heavily related to elicit imagery activity, but graphic as depictive ones could facilitate inference-making

process.

METHOD

Participants and Experimental Design In this study, sixty-four university students were participated in experiment as paid. Independent

variables were presence of gesture of agent and visual cueing. Gesture were used as conversational

movement for social interaction. Visual cueing was applied to red-colored circles, rectangular and arrows.

All participants were randomly assigned in each four conditions (n=16) by independent variables. The

dependent variables were comprehension scores and eye fixation time.

Apparatus All experiment were conducted in eye-tracking study. The SMI iViewX eye-tracker was used for

measuring the eye fixation time in AOI from participants. In experiment room, learners should watch a

video in front of eye-tracking device. The eye-tracker recorded whole experiment sessions. For

instructional materials, the agent was made by software ‘iClone5’. Narration were used to text-to-speech

(TTS) machine voice. For instructional materials, the video clip was used. The instructional materials

243

were about ‘formation of clouds and air mass’ in science lecture. The content was divided in types of

information formats between text-based and graphic-based materials. While text-based information

focused on core concepts and related meaning on theme, graphic-based one had a tendency with spatial

information which were specific locations or procedures.

Data Analysis Before seven days conducting the experiment, pretest were conducted for measurement of

covariance. The comprehension scores were 23 items divided between measurements with types of

information format. There were conducted pencil-to-paper test. For analyzing the eye fixation time, area

of interest (AOI) were designated to content area. The eye fixation time means sums of durations from all

fixation and saccades in designated area of interest (AOI). For measuring dependent variables, two-way

ANCOVA were applied. Covariate was pre-test scores with each types of information formats.

RESULT

Comprehension score In comprehension score, there were two types of score between text and graphic. In condition of text

condition, like figure 1 below, it had a significant main effect on gesture of pedagogical agent

(F(1,63)=7.44, p=.008). No gesture condition were higher comprehension score in text condition. Like

figure below, There had also significant interaction effect between independent variables (F(1,63)=4.57,

p=.037). On the other hand, in graphic condition, no significant result on all conditions.

Figure 1. Graphs of difference between comprehension score in text format

Eye fixation time As a result of eye fixation time in this study, with text formats, there were significant main effect on

gesture in eye fixation time (F(1,63)=9.95, p=.003). The conditions excluding gesture had longer fixation

time rather than those groups including gesture. Furthermore, in graphic condition, there was a main

effect on gesture (F(1,63)=6.27, p=.015) on eye fixation time with graphic formats. In light of previous

result, no gesture condition has higher fixation time rather than other condition.

CONCLUSION

Split-attention Effect by Gesture

244

Like previous results, there were significant differences by gesture of agent on comprehension score.

Especially, no gesture condition were outperformed than those groups including gesture in text format. In

line with this result, there were also significant differences between AOI analyses. In case of no gesture

condition, the longer fixation time were emerged in AOI. It means that learners couldn’t concentrate on

contents and be interfered with learning process. Through these results, gesture of pedagogical agent

might foster split-attention effect by distraction (Craig, Gholson, & Driscoll, 2002).

Attentional or Redundant Visual Cueing Visual cueing is a guidance for cognitive process visually. However, in case of this study, visual cueing

has a contrary results from their role. It was either attentional guidance or redundant information. The

condition with both of gesture and visual cueing had lowest score on comprehension. However, the

condition applied only visual cueing had highest score on comprehension than other conditions. While

only visual cueing could direct intentional cognitive process, the case which is simultaneously shown

from both of visual cueing and conversational gesture of agent could trigger redundancy effect on

multimedia environment.

Cognitive Processing with Information Format While comprehension score has different result in text condition, no significant results were found

in graphic ones. Hochpöchler et al. (2012) revealed that differences between text and graphic formats

were caused by cognitive processing from perception. Textual format was considered for imagery

elaboration, on the other hand, graphical one focused on formation of realistic mental models visually.

Especially, in this study, fixation count in graphic formats had significant differences. It mean that there

were more referential cognitive process in graphical ones because learners tried to comprehend only

visual information including their procedural and hierarchical structures. In contrast, text-based formats

were easily obtained information through sematic representation.

REFERENCES

Craig, S. D., Gholson, B., & Driscoll, D. M. (2002). Animated pedagogical agents in multimedia

educational environments: Effects of agent properties, picture features and redundancy. Journal

of Educational Psychology, 94(2), 428.

de Koning, B. B., Tabbers, H. K., Rikers, R. M., & Paas, F. (2007). Attention cueing as a means to

enhance learning from an animation. Applied Cognitive Psychology, 21(6), 731-746.

Frechette, C., & Moreno, R. (2010). The roles of animated pedagogical agents’ presence and nonverbal

communication in multimedia learning environments. Journal of Media Psychology: Theories,

Methods, and Applications, 22(2), 61.

Goldin-Meadow, S., Nusbaum, H., Kelly, S. D., & Wagner, S. (2001). Explaining math: Gesturing

lightens the load. Psychological Science, 12(6), 516-522.

Hochpöchler, U., Schnotz, W., Rasch, T., Ullrich, M., Horz, H., McElvany, N., & Baumert, J. (2012).

Dynamics of mental model construction from text and graphics. European Journal of

Psychology of Education, 1-22. doi: 10.1007/s10212-012-0156-z

Jarodzka, H., Scheiter, K., Gerjets, P., & Van Gog, T. (2010). In the eyes of the beholder: How experts

and novices interpret dynamic stimuli. Learning and Instruction, 20(2), 146-154.

Schnotz, W., & Bannert, M. (2003). Construction and interference in learning from multiple

representation. Learning and Instruction, 13(2), 141-156.

245

Assessment of Virtual Patients on Realistic Performance

Sun Kim

[email protected]

Master Student

The CaLT Lab, Department of Educational Technology, Chonnam National University

Gwangju, Korea

Jeeheon Ryu

[email protected]

Associate professor

The CaLT Lab, Department of Educational Technology, Chonnam National University

Gwangju, Korea

ABSTRACT

The purpose of this study is to find out a number of necessary items for assessing the virtual patients

on realistic performance. A virtual patient is defined as a computer program that simulates real-life clinical

scenarios for the purpose of healthcare and medical training, education. A virtual patient is being utilized

as a novel way to augment traditional methods of teaching and assessing clinical interviewing skills,

bioethics, basic patient communication, history taking and clinical decision-making skills. Furthermore, a

virtual patient can be used to simulate medical students with various medical conditions associated with

patients. However, there are a few studies that have reported the instrument for assessing virtual patient.

For this reason, this study will investigate the factors assessing virtual patient in terms of realistic

performance.

Keywords: Virtual patient, Assessment, Realistic performance

INTRODUCTION

One of the challenges in medical education concerns the need to teach students how to apply their

knowledge when they are dealing with clinical problems. Research has shown that students develop

clinical reasoning skills by seeing many patients, actively engaging in problem solving and receiving

sufficient feedback (Huwendiek et al., 2009). Especially, virtual patients are developed for training,

assessment and education of medical students. However, there are no suitable frameworks at present for

assessing virtual patients. As a teaching tool or an assessment tool, it is important to find out the

instrument items. For this reason, it needs to develop factors affecting realistic performance of virtual

patients.

THE LITERATURE REVIEW

This study will suggest that the factors having effects on realistic performance assessment of virtual

patients from the literature review. Wind and colleagues (2004) developed a valid, reliable and feasible

instrument to evaluate the performance of Standardized patients (SPs), based on 21questions. Their focus

246



was to access two sub-scales such as the authenticity of role play and the quality of feedback. However,

they focus on performance of SPs not VPs. It might have different perceptions to the SPs compared to the

VPs.

Tait and colleagues (2008) developed and evaluated the critical care e-learning scenario for student

nurses by surveying their attitude to the scenario. The questionnaire consisted of one item in a five-sub

scale that measured participant perceptions of 1) ease of use of the scenario; 2) benefit of its interactive

content; 3) its realism; 4) their increase in confidence in dealing with similar situations in real life; and 5)

overall participants’ attitude for scenario. Scenario-based learning has been implemented in a number of

ways including the use of video clips to develop students’ skills at handling difficult situations, clinical

simulation laboratories and e-learning scenarios. It will be useful to increase student nurses’ knowledge of

critical care and decision-making skills. Zary and colleagues (2006) also evaluated if it was possible to

develop a web-based virtual patient case simulation environment. The questionnaire that presented a Six-

Point Likert-type rating scale focusing on ease of use, experiences learning outcome and realism.

However, they investigated students’ attitude of scenarios, not student’ perception of VPs.

Bearman and Cesnik (2001) compared student attitudes to different models of the same virtual

patients. They used 10questions to measure student attitudes to the simulation. They concluded that the

majority of students were positive, although not wildly enthusiastic, towards the simulations. In addition,

Forsberg and colleagues (2011) investigated students’ opinions about feasibility of using Virtual patients

(VPs) for assessing clinical reasoning in nursing education. The questionnaire was constructed 7

questions to measure two sub-scales: 1) the use of virtual cases for assessment; and 2) the use of web-SP

system. They investigated that overall the students had very positive opinions about the use of virtual

patients as an assessment method. However, such factors are not relevant for performance of VPs.

FACTORS FOR ASSESSING VIRTUAL PATIENT

There is a need for virtual patient realistic performance. Evaluation of VP performance is

important to ensure the educational quality when they use VPs. To find out factors for the instrument, we

collected the instruments used in other studies that investigated VPs. After collating all 76items from

these studies, we selected appropriate factors from initial items and revised them as follows Table 1.

<Table 1>. Factors affecting realistic performance of virtual patients

Study Numbers of

Questions Initial Factors Revised Factors

Wind and

colleagues

(2004)

21

1) Authenticity

2) Feedback

3) Satisfaction

▪ Ease of use

relating to the use of virtual patients

▪ Realism

relating to the level of realism such

as facial expression and voice of the

virtual patients compared to real

patients

▪ Usefulness

relating to the learner’s perceptions

Tait and

colleagues

(2008)

21

1) Ease of use

2) Interactive content

3) Realism of scenario

4) Confidence

5) Satisfaction

Zary and

colleagues

(2006)

17

1) Ease of use

2) Experienced

learning outcome

3) Realism of scenario

247

Bearman and

Cesnik (2001) 10 1) Simulation

of whether they felt more useful

about dealing with similar situations

in real life after using the virtual

patient

▪ Satisfaction

relating to the overall attitude of the

learners to virtual patients

Forsberg and

colleagues

(2011)

7 1) Use of virtual scenario

2) Satisfaction

As shown table 1, there have something in common with factors such as ease of use, realism,

usefulness and satisfaction. So this study selected the factors which are ease of use, realism, usefulness

and satisfaction, and are described as shown table 1.

IMPLICATIONS

There are two limitations which could impact the generalization. The first, only a few studies are

reviewed and the studies are restricted to virtual patients. If other researches such as agents are reviewed,

factors affecting virtual patients’ realistic performance may be different. The second, the results of this

study are identified through literature review, not an experimental research. To make a valid, reliable and

feasible instrument through results of this study, a further study must be investigated.

REFERENCES

Bearman, M., & Cesnik, B. (2001). Comparing student attitudes to different models of the same virtual

patient. Studies in health technology and informatics, (2), 1004-1008.

Dickerson, R., Johnsen, K., Raij, A., Lok, B., Stevens, A., Bernard, T., & Lind, D. S. (2005). Virtual

patients: assessment of synthesized versus recorded speech. Studies in Health Technology and

Informatics, 119, 114.

Forsberg, E., Georg, C., Ziegert, K., & Fors, U. (2011). Virtual patients for assessment of clinical

reasoning in nursing—A pilot study. Nurse Education Today, 31(8), 757-762.

Huwendiek, S., Reichert, F., Bosse, H. M., De Leng, B. A., Van Der Vleuten, C. P., Haag, M., ... &

Tönshoff, B. (2009). Design principles for virtual patients: a focus group study among

students. Medical education, 43(6), 580-588.

Tait, M., Tait, D., Thornton, F., & Edwards, M. (2008). Development and evaluation of a critical care e-

learning scenario. Nurse Education Today, 28(8), 970-980.

Wind, L. A., Van Dalen, J., Muijtjens, A. M., & Rethans, J. J. (2004). Assessing simulated patients in an

educational setting: the MaSP (Maastricht Assessment of Simulated Patients). Medical

Education, 38(1), 39-44.

Zary, N., Johnson, G., Boberg, J., & Fors, U. G. (2006). Development, implementation and pilot

evaluation of a Web-based Virtual Patient Case Simulation environment–Web-SP. BMC medical

education, 6(1), 10.

248

Usability Study of Visual Dashboard as Learning Analytics

Interventions

Kunhee, Ha

[email protected]

Graduate Student


Seoul, Korea

Il-Hyun, Jo

[email protected]

Associate Professor


Seoul, Korea

Sohye, Lim

[email protected]

Assistant Professor


Seoul, Korea

ABSTRACT

Learning analytics has been emerging as a breakthrough research and practice domain for

educational technology (Campbell, DeBlois, & Oblinger, 2007; Elias, 2011). Learning Analytics

Dashboard (LAD), which is a visual presentation of data mining results regarding individual learners‟

online learning behaviors, has been suggested as an effective intervention strategy for their learning and

performance (Jo & Yu, 2013, ; Van Barneveld, Arnold, & Campbell, 2012). However, only a paucity of

empirical studies regarding the design strategy of and usability of online dashboard interventions has been

reported (Koulocheri & Xenos, 2013).

The purpose of this study was to investigate the user experience with visual dashboard designed and

developed based on the learning analytics perspective. Information provided in visual dashboard was

composed of graphs representing the results of statistical analysis including total log-in time, total log-in

frequency, log-in regularity, visits on board, time spent on board and visits on repository. However, it was

assumed that these statistical representations may not be easily understood to students. As a result, it is

possible not to find maximum effects intended by the dashboard designers.

Therefore, as an empirical study, we conducted a usability test of LAD and investigated students‟

real-time responses on LAD. Qualitative research method including stimulated recall protocol was

employed. Users‟ experience on LAD was recorded through the software „Morae‟ and interviews were

conducted with 6 college students. The results of the study provide useful insights on how to design and

develop an instructionally effective and psychologically intuitive dashboard as an intervention of learning

analytics.

Keywords: Learning Analytics, Visual Dashboard, Usability Test

249

INTRODUCTION

Learning analytics is an emerging research field for both practical implications and the theoretical

ground in educational technology (Campbell, DeBlois, and Oblinger, 2007; Elias, 2011). An approach of

learning analytics suggests actionable interventions for students by observing, understanding and

predicting learning behaviors (Brown 2011). Unlike the application of data mining to analyze customers‟

behavior pattern in cooperation setting, the area of learning analytics aims not only to analyze students‟

learning pattern but also to provide effective interventions for learners to develop learning and

performance. Above all, the characteristics of online learning environment where the distance between

instructor and learners is rather extensive (Moore, 1993), and thus the range of direct observation and

interventions are narrow and limited (Scheuer, and Zinn, 2007). Thus, solutions such as Learning

Analytics Dashboard (LAD) can be effective strategies for students to be aware of their learning

behaviors, to reflect themselves, to have new insights, and to change their learning patterns and methods

(Leony, Pardo, de la Fuente Valentın, Quinones, and Kloos; van Barneveld, Arnold, and Campbell, 2012).

Thus, several previous researches have attempted to test and analyze usefulness of dashboards. Katrien,

Erik, Joris, Sten and José (2013) compared twelve dashboards with regard to usefulness, effectiveness.

Govaerts, Verbert, Duval and Pardo (2012) also analyzed that the use and usefulness of different

visualizations and discussed how such dashboards work on learners. These studies suggest that the

usefulness of dashboard be analyzed before verifying whether dashboard can be effective as a learning

intervention. However, the general function of dashboard was limited to presenting accumulated data

without considering effective instructional design to stimulate learners‟ cognition. Therefore

instructionally advanced design techniques have been required in order for learners to improve their

learning performance (Segedy, Sulcer, and Biswas, 2010; Essa, and Ayad, 2012). As an example, Course

Signal (CS) program at Purdue University played a role of early warning that enables instructors to

predict those who are at the risk level for appropriate academic achievement. The high level of risk was

shown with a red signal while the low risk level was indicated in green sign (Arnold, and Pistilli, 2012).

Purdue University‟s CS program has been positively evaluated for it develops students‟ meta-cognitive

strategies as well. In that case, the usefulness of dashboard is very important issue to instructor, but before

examining usefulness of dashboard, its usability should first be investigated. Jo, HEO, Lim, CHOI, and

NOH (2013) suggest usability, the degree of users‟ felt easiness of artifacts, is the necessary condition to

be tested prior to usefulness of them. In line of Jo et al‟s thought, LAD intervention should be tested its

usability before investigating its ultimate goal, usefulness of learning and performance. However, no

study systematically tested LAD‟s usability. The question we address in this study is: Is the learning

analytics dashboard we developed usable to target students? To answer the question, the first step to be

taken is to design and to develop a LAD prototype and then to test its usability because a learning

supporting tool might be effective when it is perceived to be useful tool by its users.

In short, we attempted to focus on students‟ perception and comprehension of the information in LAD.

Usability is the issue of affectiveness, while Usefulness is the issue of effectiveness. It is also the reason

why this study used stimulated recall protocol method which can make subjects to recall past activity

more clearly. In cognitive information processing, people can remember when they are provided with

detailed situation or distinguishing symbols (Jo & Kim, 2006). The usability test requires users‟ precise

memory because it should be designed and modified based on the users‟ experiences and perception.

METHOD

In order to answer the research questions, we adopted a qualitative research method in order to

focus on the usability test to investigate a small number of students‟ reactions and their perception on

LAD.

Participants The participants of this study were 6 college students in E women‟s university in Seoul. They were

sampled out of from an undergraduate course “Knowledge management”. All students were Educational

Technology major.

250

Learning Analytics Dashboard Institute for Teaching & Learning (ITL) of E university collaborated for the design and

development of dashboard. Dashboard is constructed with the log data accumulated for 8 weeks after the

fall semester began. 7 graphs were presented including the first graph that summarized the correlation

between the variables: total log-in time, total log-in frequency, log-in regularity, visits on the board, time

spent on the board and visits on repository. In that plane board, all students in the class were shown and

„I‟ was located as a red dot. It also indicated the average value of the class so that „I‟ could choose the

variables as a X-axis and Y-axis to figure out where „I‟ was and how far „I‟ was apart from average line

of each variable. The other was a bar graph in which trend lines of „my‟ weekly activity was visualized.

Figure 1 is the prototype display that the subjects were exposed to in the usability test.

Figure 1. Learning Analytics Dashboard (LAD)

Setting for Usability Test

To conduct a usability test, 6 participants were placed in a secure classroom equipped with

computers. Since the recording software „Morae‟ was installed on the computers, students‟ mouse

movement, the displayed screen, students‟ face and their voices were automatically recorded as shown in

Figure 2. At the beginning of test, the researcher explained the participants the purpose and the procedure

of the study. {Since their testing duration is related to our study, they could continue testing until they

wanted to quit. As a result, the mean of testing duration was about 10 minutes. After the test, we collected

the data and replayed using the manager program using „Morae‟ for the analysis.

251

Figure 2. The recorded data using „Morae‟ in usability test

Research Procedures

The test consists of four steps. First, as participants used LAD, we recorded their reactions using

software „Morae‟. Second, one day later, we conducted individual interviews with 6 participants. Third,

we analyzed all the data including observation notes, recorded video clips, and the transcribed interview

results.

The interviews were conducted based on the recorded data. The manager program in „Morae‟

provides Mark system, so that we could replay the recorded data and mark on each timeline when the user

reacted to something. The data extracted through the replay process composed the first part of interview.

In addition we proceeded to the second part of interview by asking the degree of understanding on LAD,

perceived usefulness and opinions including suggestions to improve the quality of LAD. The second part

of interview was identical to all 6 students. If a student had already answered the questions of the second

part in the first interview watching recording data, repeated answers were skipped.

We decided that the participants should be free to speak of their opinions so that all the interviews

were asked in the format of open-ended questions. For example, during the first interview, the interviewer

and participant watched the video clip together and paused on the point where the participant exhibited a

unique reaction. By asking the reason for the reaction “why did you do so?” or “what did you think about

it at the moment?” the participant could easily recall their perceptions and answer more precisely based

on their memory. Testing the usability of dashboard including the interviews with six participants were

conducted within two days in order to prevent further memory loss.

RESUSLS

Degree of Understanding

Most participants reported difficulties understanding the first summary graph. We could observe

that they were trying to compare the summary graph with the other graphs. One student mentioned:

“The Summary graph was not comprehensible at first, so I looked down the others…the summary

graph presents too many information.”

In this regard, students pointed out that they needed a description on the meaning of graphs. Student

„E‟ indicated:

“Additional descriptions are necessary for understanding the graph properly. If someone doesn‟t

have sufficient background knowledge in the statistics, these graphs are too hard to understand.

Also, the correlation between the summary graph and the bar graphs should be explained

enough.”

LAD display

The recorded user face

(with web cam)

252

Student „A‟ and „D‟ also answered that they could not understand why and how to see the suggested

data when shown. On the other hand, another student „B‟ mentioned that she could understand LAD

almost perfectly. She also explained how she could analyze the LAD.

“I wondered why the log-in frequency does not match the total log-in time in seventh week, so I

inferred that the data is not enough. I think more data should be updated.”

Through participants‟ diverse reactions on LAD, we discovered that the level of students‟

understanding, especially the literacy of graphic representation, varied and this may impact their

perceived usefulness and the effectiveness of LAD. This underlines the importance of providing enough

instructions and guidelines to help them understand the meaning of graphs on their online learning

patterns.

Perceived Usefulness

Several students commonly commented that the graphs illustrating their relative position in

comparison with other students were useful. Moreover, they added that the function of graph comparing

with other students would boost their motivation. Students „F‟ said:

“I usually wanted to know how other students are studying in the class. For example, I clicked all

the board each time whenever I logged in the cyber campus. I really wondered how and what they

do in there. Especially, in the forum board, students were interacting for their team project, so I

sought for other teams‟ performance as well. If they interacted more frequently than our team, I

was motivated to do harder.”

Students were positive when the information was related with their location in the class and

performance. What students wanted to understand was especially the correlation between their online

activity and actual learning performance. Student „C‟ critiqued:

“Do online activities have a relationship with real learning activities? The log data which is

visualized on the dashboard just shows me that „you are doing what you see‟, but additional

implications are not involved in data, I think. I just understand what I am doing without why the

dashboard presented all this data.”

However, some students negatively responded in regard to the total log in time. Student „B‟

mentioned that like: “Sometimes I log in cyber campus and do other things. I do not study all the time I

log in.” In addition, student „F‟ answered similarly:

“Some classes could leave cyber campus empty. The professor in that class just uploaded a

syllabus or some notices and did nothing with cyber campus. In that case, students did not use

forum or free board at all.”

Opinion and Suggestion

A few students answered that the correlation and interaction between students in the class is needed.

When the class uses forum board, how students interact with other students can be an indicator for their

participation in the group learning. Student „C‟ said:

“Sometimes I would log in but don‟t do anything in particular, so I think the number of reply

could be more helpful information for us. That information was about practical learning.”

Student „D‟ also supported such need for interaction information by mentioning: “Especially online

debates or communication is important. I could compare myself with others.”

In summation, students perceived LAD in a two ways. One is a reminder that they can reflect their

previous behavioral patterns and the other is a comparison tool that they could learn their relative rank in

the class. Lastly, the participants suggested for the number of opinions for modifying LAD. These

include:

253

“I needed the unit of information. For instance, this bar graph indicates the log-in regularity, but I

could not distinguish whether the unit of these numbers was minute or not.” (Student C)

“Is this the number of frequency? Or just time?”(Student D)

CONCLUSION

Usability is a separated concept from usefulness. LAD can be useful but it does not assure that it is

also usable. Why we conducted qualitative research method in this study is to figure out the affectiveness

of students using LAD. Learning interventions with LAD is closely related with a usage pattern. In other

words, how users approach LAD is related with how much LAD is usable and it could be finally

determine the effectiveness of online learning interventions by LAD.

With that regard, several implications on LAD in this study were drawn. First, students‟ response

indicated incompatibility about the perception of instructor. We had concluded in the previous study that

regularity of learning interval is a powerful indicator for learning performance (Kang, Kim, and Park,

2008; Jo, Yoon, and Ha, 2013). However, students did not know that log-in regularity information is

related with learning performance or it could predict learning outcomes. Student „B‟ replied that she did

not know that even log-in regularity could be suggested as a data, but most of students looked like that

they could not find which information relates to the learning performance to a greater extent which means

that the usefulness of information could be varying across the learners and the instructors. Thus,

designing a dashboard should take the learners‟ perception of their learning into account and explain the

reason for the information presented in the dashboard. Thus, dashboard without some descriptions to the

extent the learners can notify why instructor choose that data cannot serve the role of learning

interventions enough. In that regard, designing dashboard should consider the learners‟ perception of their

learning and explain why dashboard presents these kinds of information.

Second, LAD designed in this study is a preliminary step before developing an effective learning

intervention with which learners can observe their learning patterns, predict learning outcomes and adjust

their behaviors accordingly. For that reason, this study results can be the reference for instructional design

to develop dashboard as a learning intervention. As referred to earlier, LAD should be considered in two

perspectives: usefulness and usability. What we intended when design dashboard is that we could observe

how well learners understand the basic visualization of data and how they percept LAD emotively for

their continuous usage. Thus, based on this usability test, instructors could discuss what kind of

instructional design factors should be provided to help learners appropriately.

This study is not without its own limitations. In particular, the dashboard was not subject-specific

and this might have lowered the students‟ motivation to process the information in LAD. Moreover, the

online activity is not the whole learning data about a student, because most of classes are operated in

offline. Thus, the log data are limited to the analysis of the learners themselves, so the instructor needs to

consider this when they use LAD as a learning intervention.

REFERENCES

Arnold, K. E., & Pistilli, M. D. (2012). Course Signals at Purdue: Using learning analytics to increase

student success. In S. Buckingham Shum, D. Gašević, & R. Ferguson (Eds.), Proceedings of the

2nd International Conference on Learning Analytics and Knowledge (LAK ′12) (pp. 267-270). New

York: ACM.

Brown, M. (2011). Learning Analytics: The Coming Third Wave. EDUCAUSE Learning Initiative Brief,

Retrieved from http://www.educause.edu/Resources/LearningAnalyticsTheComingThir/227287.

Campbell, J. P., DeBlois, P. B., & Oblinger, D. G. (2007). Academic analytics: A new tool for a new

era. Educause Review, 42(4), 40.

Elias, T. (2011). Learning analytics: Definitions, processes and potential.Retrieved February, 9, 2012.

Essa, A., & Ayad, H. (2012, April). Student success system: risk analytics and data visualization using

ensembles of predictive models. In Proceedings of the 2nd International Conference on Learning

Analytics and Knowledge (pp. 158-161). ACM.

Gass, S. M., & Mackey, A. (2000). Stimulated Recall Methodology in Second Language Research,

Lawrence Erlbaum Associates; Mahwah, NJ.

254

http://www.educause.edu/Resources/LearningAnalyticsTheComingThir/227287

Govaerts, S., Verbert, K., Duval, E., & Pardo, A. (2012, May). The student activity meter for awareness

and self-reflection. In CHI'12 Extended Abstracts on Human Factors in Computing Systems (pp.

869-884). ACM.

Jo, I., Heeok, HEO., Kyu Yon, Lim., Jeong-Im CHOI., & Jeongmin, NOH. (2013). Usability Study of

Middle School English Digital Textbook: A Stimulated Recall Approach. Educational Technology

International. 14(1). pp. 109-136.

Jo, I. (2012). On the LAPA (Learning Analytics for Prediction & Action) Model suggested. Future

Research Seminar. Korea Society of Knowledge Management. Seoul.

Jo, I., Kang, Y., Yoon, M., & Kang, M. (2012, Fall). Development of cluster-specific learning prediction

models: A learning analytics approach, Paper presented at the HYCU International Conference,

Seoul, Korea.

Jo, I., & Kim, Y. (2006). An Exploratory Study on the Validity of Stimulated Recall Method as a

Usability Test Tool for e-Learning. The Journal of Educational Information and Med. 12(2). pp. 71-

95.

Jo, I. & Kim, Y. (2013). Impact of Learner‟s Time Management Strategies on Achievement in an e-

learning Environment: A Learning Analytics Approach. The Journal of Educational Information and

Media. 19(1). 83-107

Jo, I., Yoon, M., & Ha, K. (2013). Analysis of Relations Between Learner‟s Time Management Strategy,

Regularity of Learning, and Learning Performance: A Learning Analytics Approach. e-Learning

Korea 2013 Conference.

Kang, M., Kim, J., & Park, I. (2008). The Examination of the Variables related to the Students‟ e-

Learning Participartion that Have an Effect on Learning Achievement in e-learnign Environment of

Cyber University. Journal of Korean Internet Information. 10(5). 135-143.

Kim, S. (2003). A correlation between Learning Behavior and Achievement Level of Learners in e-

learning. Master Thesis. Korean University of Technology and Education, Chung Nam.

Kim, Y. (2010). Impact of Regulated of Learning Interval, Total Learning Hours and Number of Access

to e-Learning in a Corporate e-Learning Environment on Academic Achievement. Master thesis,

Seoul, Ewha Womans University.

Kim, Y. (2011). Learning time management variables impact on academic achievement in corporate e-

learning environment. Master Thesis. Ewha Womans University, Seoul.

Leony, D., Pardo, A., de la Fuente Valentın, L., Quinones, I., & Kloos, C. D. Learning analytics in the

LMS: Using browser extensions to embed visualizations into a Learning Management System.

Moore, M. G. (1993). 2 Theory of transactional distance. Theoretical principles of distance education, 22.

Santos, J. L., Verbert, K., Govaerts, S., & Duval, E. (2013, April). Addressing learner issues with

StepUp!: an Evaluation. In Proceedings of the Third International Conference on Learning


Scheuer, O., & Zinn, C. (2007). How did the e-learning session go? The Student Inspector. FRONTIERS

IN ARTIFICIAL INTELLIGENCE AND APPLICATIONS, 158, 487.

Segedy, J., Sulcer, B., & Biswas, G. (2010, January). Are ILEs ready for the classroom? bringing teachers

into the feedback loop. In Intelligent Tutoring Systems (pp. 405-407). Springer Berlin Heidelberg.

van Barneveld, A., Arnold, K. E., & Campbell, J. P. (2012). Analytics in higher education: Establishing a

common language. ELI paper, 1, 2012.

Verbert, K., Duval, E., Klerkx, J., Govaerts, S., & Santos, J. L. (2013). Learning Analytics Dashboard

Applications. American Behavioral Scientist.

255

The effects of regulatory learning strategies on collaboration load

and collaboration outcomes in computer-supported

collaborative learning

Hyojin Lee

[email protected]

Doctoral Student

Hanyang University

Seoul, Korea

Dongsik Kim

[email protected]

Professor

Hanyang University

Seoul, Korea

ABSTRACT

The purpose of this study was to examine the effects of regulatory learning strategies on

collaboration load and collaboration outcome in CSCL. We focuses on collaborative learning effectiveness

in acquiring self-regulation skills and collaborative skills when a regulatory strategies is presented. This

study was driven by a couple of questions. First, What effects do regulatory learning strategies have on

collaboration load in CSCL? Second, What effects do regulatory learning strategies have on collaboration

outcome in CSCL?

Keywords: self-regulation, regulated learning, collaboration load, CSCL

INTRODUCTION

Self-regulated learning has been of interested to many researchers in education, learning

sciences(Shunk & Zimmerman, 2008; Winne & Hadwin, 1998; Volet, Summers, & Thurman, 2009;

Zimmerman, 2008). Similarly, scholars recognize the importance of metacognitive processes in

collaborative learning such as planning, setting goals, monitoring(Gibson, 2001, Rummel & Spada, 2005).

According to Jarvela and Hadwin‟s study(2013, p.28), three dimensions characterize metacognitive

processes in Computer-Supported Collaborative Learning(CSCL). The first dimension is “self-

regulation”, which is guided by environmental conditions that promote individuals to adopt, develop, and

refine strategies; monitor, evaluate, and set goals. This dimension suggests that successful team work

requires each group member to regulate his or her own cognitive process. The second dimension of

regulatory learning that Jarvela and Hadwin(2013, p.28) describes is that “co-regulation” occurs when

individuals‟ regulatory activities are guided by and with others. Co-regulation requires team members to

be aware of one another‟s goals and progress and to consider those in relation to the shared task. The third

constitutive dimension of regulatory learning according to Jarvela and Hadwin(2013, p.28) is the “shared

regulation”. This dimension occurs when groups regulate as a collective such as when they construct

shared goals. In this case, goals and standards are co-constructed, and regulation is distributed and shared

with multiple ideas and being weighed and negotiated until consensus is met.

Many prevailing accounts of self-regulated learning in individual contexts revealed that experts

generally come up with self-regulation strategies than novice(Plass, Kalyuga, & Leutner, 2010). In other

words, the more learners have prior knowledge, the more they acquire self-regulatory skills effectively.

256

The self-regulatory process in collaborative learning is seen as akin to the process in individual learning.

In collaborative learning settings, it is also important to enable individuals to utilize their relevant

disciplinary knowledge, while at the same time making use of self-regulated learning strategies.

However, we intentionally distinguish between the individual and group(and peers) regulatory

process. Contrary to self-regulation skills, when experts in a particular domain did not have collaborative

skills, they were difficult to acquire co-regulation and shared regulation skills because coordination and

communication are interrupted by confusion of interaction. Domain-specific knowledge is one of the

crucial aspects of successful self-regulatory learning, whereas co-regulated learning and shared regulated

learning are strongly related to collaborative skills or previous experience. Therefore, regulatory learning

in CSCL contexts only succeed with respective support for individual and group dimension.

STRATEGIED FOR SUPPORTING REGULATION IN CSCL CONTEXTS

Our approach to supporting self-regulated learning in CSCL is grounded in Cognitive Load

Theory(CLT). As we have described above, the majority of research examining self-regulatory learning

have appeared that the level of learner expertise was a critical factor that influenced the application of

self-regulation strategies(Plass et al., 2010). If a learner has sufficient knowledge to understand the

problem formulation, he or she may successfully apply self-regulatory strategies by reducing intrinsic

load. In contrast, for these less experienced learners, demands of information processing may induce

cognitive overload and interfere with self-regulation learning instead of assisting it.

Self-regulated learning is related to extraneous load as well as intrinsic load. Demands of

monitoring, controlling, reflecting activity might cause cognitive overload and lead to failure of both the

problem solving and learning process. Without adequate support in collaborative learning settings,

learners often fail to complete their joint task or find that it requires too much time and effort. Therefore,

self-regulated learning in CSCL contexts need to be supported to ensure controlling intrinsic load and

extraneous load and promoting germane cognitive process(Plass et al., 2010).

CLT offers principles and methods to design effective and efficient instructional interventions that

support self-regulatory learning in CSCL contexts. Kester, Paas and van Merriënboer(2010) showed the

possibility and promising potential of feedback strategies for the field. The study revealed that germane-

inducing method(i.e., delayed feedback) was more effective in solving complex task than structured

method(i.e., immediate feedback) (Kester et al., 2010). Giving delayed feedback refers to comparing the

current state of group work to a model of desired work after a certain period of time and intervene when

discrepancies between these two states are discovered. On the other hand, immediate feedback occurs

simultaneously with problem solving. Butler and Winne(1995) highlight feedback is generally an inherent

catalyst for all self-regulated activities. While learners monitor their engagement with tasks, internal

feedback is generated by the monitoring process(Butler & Winne,1995). The feedback methods are not

designed for collaboration but may be considered strategies for supporting self-regulation in CSCL.

However, some scholars suggest that germane-inducing methods priority may be limited to novice(Ross

& Kilbane, 1997). For low prior knowledge learners, what they make an effort to evaluate their

engagement with tasks on their own becomes a redundant activity that contributes little or nothing to

further learning and problem solving and causes cognitive overload. Despite the limitation of information

processing, some researchers acknowledge the importance of problematizing task which may make

learners encounter important concepts or processes on their own. Problematizing task have been found to

successfully trigger changes in group dynamics by provoking debate, deliberations, and decisions(Reiser,

2004). Thus, this method may not only cause cognitive load that contributes to learning asennd problem

solving but also lead to enhanced transfer.

Collaboration scripts have been shown to be a promising approach to support co-regulation and

shared regulation in CSCL. A common use of collaboration script in CSCL environments is to decrease

the coordinative effort both on the teacher's and the learner's part. CSCL script enables group members to

engage in the appropriate interaction with peers and the improvement the joint problem-solving(Wecker

& Fischer, 2007). CSCL script can be advantageous to regulation. Awareness of one another's regulatory

process and co-construction a shared task perception are central processes in collaborative learning tasks.

To ensure efficient regulatory activity in the CSCL environment, it is crucial to specify, sequence, and

257

distribute learning activities by using script. Collaboration script not only aims to foster the acquisition of

domain-specific knowledge, but also collaborative skills.

Two different types of scripts may be used to support collaboration: Scripting as a supporting tool

and scripting as a teacher‟s instruction. These approaches differ not only in the timing of the intervention,

but also coercion. The coercion suggests that the degree of freedom participants have in following the

script. Scripting as a supporting tool aims to create favorable conditions for learning by designing and

scripting the situation before the interaction begins(Jermann et al., 2004; Kirschner, & Erkens, 2013). The

method constrains the number of options learners have, thus guiding them along the lines of the

processes(Beers et al., 2005). On the contrary, scripting as a teacher‟s instruction supports co-regulation

and shared regulation by taking actions after the interaction has begun(Jermann et al., 2004; Kirschner, &

Erkens, 2013). The method such as providing students with process worksheets guides processes by

providing just-in-time support for them when a specific problem arises.

Despite the advantages of guiding interaction and problem-solving processes, many prevailing

empirical studies revealed that the coercion of script made learner's motivation dwindling and induced the

short-term effects(Rummel, & Spada, 2005). Enforcement structured interaction might cause redundant

activity and lead to failure of reflection on the „whys‟ of the scripting.

This study aims to examine the effects of regulatory learning strategies on collaboration load and

collaboration outcome in CSCL by designing feedback and script.

1) What effects do regulatory learning strategies have on collaboration load in CSCL?

2) What effects do regulatory learning strategies have on collaboration outcome in CSCL?

RESEARCH DESIGN

The experimental paradigm set up comprised two phases: a self-regulation phase and a group-

regulation phase. The goal of the self-regulation phase was the acquisition of self-regulation skills and

domain-specific knowledge about elements relevant for a good and potentially successful collaboration.

Aspects of the joint work – the collaboration outcome - after the group-regulation phase as well as

collaboration load were investigated as dependent variables. Further, collaboration skills about elements

of a good and potentially successful collaboration was assessed in a posttest.

Independent variables

Feedback

We considered three aspects of feedback:

Immediate feedback by instructor Learners was given feedback from instructors within minutes

whenever there appeared to be „discrepancies between current understanding and performance

and a learning goal‟.

Delayed feedback by instructor Learners were presented with a set of rubric to self-check their

self-regulation strategies and then they worked on some problem collaboratively. A week later,

instructor provided feedback on whether last strategies were effective or not to attain the goal.

Script

We considered two aspects of scripts:

Scripting as a supporting tool: Groups in script condition with collaboration script received a

collection of interaction-related prompts to structure their collaborative problem solving. This

script was targeted at improving collaborative skills. During the group-regulation phase, dyads

in the scripted collaboration condition were provided with a detailed script prescribing specific

phases for their interaction. The collaboration script involved different activities and roles for

each student during the collaborative problem-solving: (a) define objectives of task, (b) scan

case description for potential problems with understanding and formulate questions to the

partner, (c) mutually answer questions and determine course of action(content, time, roles), (d)

individually search related information and come up with idea, (e) exchange information and

258

discuss individual ideas, (f) revise individual ideas and formulate final solution for the problem,

(g) copy individual parts of solution in shared editor and integrate final check of entire joint

solution(Rummel & Spada, 2005). The script was structurally equivalent to the instruction.

Scripting as a teacher’s instruction: After pre-test, students were guided to a 3 hours training

phase in the face to face, which helped them get a first experience on how to handle the learning

environment and how to collaborate. This information was equivalent to the information

presented in the prompts of the collaboration scripts. The students in the instruction condition

received no support beyond training by their teacher to the strategy of collaborative problem

solving.

Dependent variables

Collaboration Outcome

Collaboration outcomes were comprised of (a) joint solution, (b) self-regulation skills, and (c)

collaborative skills.

Joint Solution: To measure collaboration outcomes, each team‟s final report was assessed. To

analyze the quality of the joint solution, a system of quantitative criteria was developed by

experts in the area of A. To analyze the quality of the joint solution, a system of quantitative

criteria was developed by experts.

Self-regulation Skills: To assess self-regulation skills, we adopted microanalytic methodology by

developed Kitsantas and Zimmerman(2002). Specific questions was used to measure well-

established self-regulatory processes and motivational beliefs or feelings at key points before,

during, and after learning. Learners were asked open- or closed-ended questions that produced

both qualitative and quantitative data, respectively. The questions were brief and task specific in

order to minimize disruptions in learning.

Collaboration Skills: As discussed earlier, we assumed that instructional support measures

would improve people‟s collaborative skills. We assessed collaborative skills with individual

posttest. We employed the coding scheme developed by Meier, Spada and Rummel(2007). This

coding sheme distinguished five theoretically and empirically grounded collaborative process:

communication, joint information processing, coordination, interpersonal relationship,

motivation. Table 3 provided detailed dimensions of these aspects.

Collaboration Load

To capture the collaboration load, we adopted subjective rating scale technique developed by Jung

and Kim(2012) for two important reasons. First, subjective rating of mental effort is the best one rather

than objective measures such as time-on-task, eye-tracking analysis because learners can be asked to rate

their perceived cognitive load subjectively. Second, this instrument directly examines three types of

load(e.g. extraneous collaboration load, instrinsic collaboration load, germane collaboration load).

Participants answered 26 questions on a seven-point Likert scale ranging from 1 “strongly disagree” to 7

“strongly agree”. There were 4 questions for physical effort, 4 questions for mental effort, 4 questions for

task difficulty, 4 qeustions for process satisfaction, 4 questions for outcome satisfaction, 2 questions for

environment availability, 4 questions for immersion.

CONCLUSION

In this study, we suggest the strategies to promote self-regulatory learning and collaborative

learning in CSCL contexts. We explored the theoretical framework of regulation‟s types and explained

each strategies to support self-regulation and co-regulation. To sum up, it is expected that the feedback

strategies can promote successful self-regulation activities and effectively induce germane cognitive

process, thereby resulting in learning enhancement. In addition, different types of scripts may cause

dissimilar cognitive load and lead to learning process. We suggest futher studies with empirical findings

that will validate the strategies developed in this study.

259

REFERENCES

Beers, P. J., Kirschner, P. A., Boshuizen, H. P. A., & Gijselaers, W. R. (2005). Coercing knowledge

construction in collaborative learning environments. Proceedings of the international conference on

CSCL 2005 (pp. 8-17), May 30-June 4, 2005, Taipai.

Butler, D. L., & Winne, P. H. (1995). Feedback and self-regulated learning: A theoretical synthesis.

Review of Educational Research, 65(3), 245–274.

Gibson, C. B. (2001). From knowledge accumulation to accommodation: cycles of collective cognition in

work groups, Journal of Organizational Behavior, 22(2), 121-134.

Hattie, J. A. C., & Gan, M. (2011). Instruction based on feedback. In R. Mayer, & P. Alexander (Eds.),

Handbook of research on learning and instruction (pp. 249–271). New York: Routledge.

Järvelä, S. & Hadwin, A. F. (2013) New Frontiers: Regulating Learning in CSCL. Educational

Psychologist. 48(1). 25-39.

Jermann, P., Soller, A., & Lesgold, A. (2004). Computer software support for CSCL. in J. W. Strijbos, P.

A. Kirschner, & R. L. Martens(eds.), What we know about CSCL? (pp. 141-166), NY: Kluwer

Academic Publishers.

Jung. H. J. & Kim. H. Y. (2012). An Exploratory Validation for the Constructs of Collaboration Load.

Journal of Educational Technology, 28(3), 619-640.

Kester, L., Paas, F., & van Merriënboer, J. J. G. (2010). Instructional control of cognitive load in the

design of complex learning environments. In J. Plass, R. Moreno, & R. Brünken (Eds.), Cognitive

load theory (pp. 109-130), New York: Cambridge University Press.

Kirschner, P. A. & Erkens. G. (2013). Towards a framework for CSCL research. Educational

psychologyist, 48(1). 1-8.

Kitsantas, A., & Zimmerman, B. J. (2002). Comparing self-regulatory processes among novice, non-

expert, and expert volleyball players: A microanalytic study. Journal of Applied Sport Psychology,

14, 91–105.

Meier, A., Spada, H., & Rummel, N. (2007). A rating scheme for assessing the quality of computer

supported collaboration processes. International Journal of Computer-Supported Collaborative

Learning, 2, 63–86.

Plass, J. L., Kalyuga, S., & Leutner, D. (2010). Individual differences and cognitive load theory. In J. L.

Plass, R. Moreno, & R. Brunken (Eds.), Cognitive load theory (pp. 65–87). New York: Cambridge

University Press.

Reiser, B. J. (2004). Why scaffolding should sometimes make tasks more difficult for learners. In G.

Stahl(Ed.) Computer support for collaborative learning: foundations for a CSCL community,

proceedings of CSCL 2002 (pp. 255-264), Boulder, CL, Jan. 7-11, 2002.

Ross, B. H., & Kilbane, M. C. (1997). Effects of principle explanation and superficial similarity on

analogical mapping in problem solving. Journal of Experimental Psychology: Learning, Memory,

and Cognition, 23, 427–440.

Rummel, N., & Spada, H. (2005). Learning to collaborate: an instructional approach to promoting

collaborative problem solving in computer-mediated settings. Journal of learning sciences, 14(2),

201-241.

Schunk, D. H., & Zimmerman, B. J. (Eds.). (2008). Motivation and self-regulated learning: Theory,

research, and applications. Mahwah, NJ: Lawrence Erlbaum.

Volet, S. E., Summers, M., & Thurman, J. (2009). High-level co-regulation in collaborative learning: how

does it emerge and how is it sustained? Learning and Instruction, 19(2), 128–143.

Wecker, C., & Fischer, F. (2007). Fading scripts in CSCL: the role of distributed monitoring. Mice, Mins

and Society: CSCL conference 2007(vol. 8) (pp. 763-771), New Brunswick, NJ:SUNJ.

Winne, P., and Hadwin, A. (1998). Studying as self-regulated learning. In Hacker, D., Dunlosky, J., and

Graesser, A. (eds.), Metacognition in Educational Theory and Practice, Erlbaum, Hillsdale, NJ, pp.

279–306.

260

Improvement of Score Reading Skill

By Music Composing Class with SMART Education

Hyerin Lee

[email protected]

Master’s degree student

Chuncehon National University of Education

ABSTRACT

Score reading skill and music composing are related each other deeply. So, to improve score reading

skill, music composing class is effective way. But there are some barriers. Every student needs music

instruments individually. It’s not that easy to give them music instruments each, however, especially in a

big size of class. Therefore, music teacher often made them compose as a group or use simplified music

instruments like a recorder or a melodeon. If we call these ways as a traditional teaching style, we can

overcome some limitations of traditional music composing class by using SMART education. The purpose

of this study is improvement students’ score reading skill in music composing class through SMART

education. For achieving this purpose, 16 of Grade 3 students participated in this study containing a

mental weakness student. They composed their own music individually with a Samik Piano application

and iPad. The period of this study is a week, 6 times of music class.

Students recognized about this study and answered to a paper questionnaire about their interests,

experiences, expectations about music composing class in 1st class. In 2nd~ 3rd class, students composed

own melody through a Samik Piano application by touching every tones of a virtual piano keys and wrote

their melody in a simplified score called ‘grid score’. In 4th class, students composed proper rhythm

according to their melody and wrote the rhythm in the ‘grid score’ In 5th class, students moved their

melody and rhythm from ‘grid score’ to a staff notation. In final 6th class, students played own music in

front of classmates using a Samik Piano application. Students do peer evaluation through 2 standards, flow

of melody and variation of rhythm. Also, students answered to a paper questionnaire the same with

pretest’s one.

In the conclusion, students answered they felt improvements of score reading skill and interests

about music composing. Also, students satisfied about their products as they were able to follow their own

learning speed and had enough time to think and edit. The limitations of this study were wasting time to

instruct students about iPad and worry about novelty effect.

Keywords: Score reading skill, Music composing, SMART education, iPad, Piano application

Needs Analysis

Before starting this study, students answered to a questionnaire about composing class.

<Table 1> Pretest questionnaire result

This questionnaire is designed by 4 questions and students check this. 16 of Gr.3 students

participated in this questionnaire.

1. Have you ever composed music on your

own?

Yes( very much, a little bit) : 9

No( very much, a little bit) : 7

2. Do you like music? Yes (very much, a little bit) : 11


3. Do you have confidence in music class? Yes (very much, a little bit) : 14


261


4. Can you read a staff notation? Yes (very much, a little bit) : 5


5. Do you have expectation about music

composing?

Yes (very much, a little bit) : 5


Low interests about music composing More than half of students had a music composing experience on their own. They said they had

hummed some melody not in normal music class but in break time or a home. Most of students liked

music. However, 5 students who didn’t like music were all boys. 11 students couldn’t read a staff notation.

And Most of students didn’t have expectations or interests about music composing. It was kind of ironic

that they had interests or confidence about music but they didn’t have those things about music

composing. It’s because they couldn’t have many music composing experiences before. If they had

successful experiences, they could think music composing positively and it would help to boost interests

of music and score reading skill.

Limitations of traditional music composing class Through teachers’ experiences, there were two major difficulties in traditional music composing

class. First, students can’t get music instruments individually. They can use some instruments like a

recorder or a melodeon. But students need to spend more time to learn to play a recorder. And a melodeon

has a limitation to realize students’ inspiration because that is simplified shapes of a piano. A real piano is

most easy and effective tools to compose.

Second, composing is entirely personal working. It means, students need enough time and follow

their own speed. But students needed to finish on time in traditional class. It made products’ quality low

and students feel less satisfaction and motivation about composing.

Literature Review

SMART education SMART education can support smart devices boosts collaborated learning, improving recognition

of learners, self-directed learning. (Sunhee Bang, 2012)

SMART education contains 5 compositions which are ‘Self-directed learning, Motivated, Adaptive,

Resource free, Technology embedded’. ‘Self-directed learning’ means, students can get knowledge by

themselves through teacher’s guideline. Also students can control learning process on their own.

‘Motivated’ means learners’ can be motivated by SMART education. As SMART education can help to

provide and share learners’ product immediately. In addition, SMART education can easily figure out

how many students achieve there objectives. So teachers can adaptive worksheets or review contents to

proper students. That is because SMART education is ‘Adaptive’. SMART education can provide

multimedia contents and searching system. Also, a learning group can gather students’ ideas with

technology such as cloud system or collaborative devices. These mean SMART education is ‘Resource

free’ and ‘Technology embedded’.(Jinsook Kim, 2012)

To overcome traditional music composing class’s difficulties, we can use SMART education.

Difficulties of traditional music composing were lack of individual music instrument to compose and

giving chance to control own working speed through need analysis. Among 5 compositions of SMART

education, 4 compositions(S, M, R, T) can help this problems.

Methods

This study was constructed with 6 times of music class. The whole plan of this study is like the

following.

262

Digital device and application Students used iPad and Samik piano application. Samik piano application is free contents in App

store. This application contains 88 keys of piano perfectly. Also, students can choose variety sounds.

Besides, example melody contained this application can help students to practice to play the piano and

recognize syllable names. So, this application makes students feel friendly about a piano.

Introduction and pretest In 1

st class, students recognized plans of this study and participated in pretest questionnaire. This

study composed of 6 music class.

Composing melody In 2

nd~3

rd class, students touched every keys of piano application to ready to compose own melody.

After that, they composed melody with natural flow of their mind. And then, they wrote down the syllable

names of melody in a simplified score called ‘Grid score’. ‘Grid score’ was made of 50 cubes. When they

finished to write ‘grid score’, a teacher tought them locations of tones in a staff notation.

<Picture 1> Composing melody and writing syllable names in ‘Grid score’

Composing rhythm In 4

th class, students reviewed some types of rhythm when they studied in previous music class. And

a teacher tought them how to draw notes in ‘rhythm grid score’. ‘Rhythm grid score’ is a upgraded

version of ‘grid score’ add some lines to put in notes.

<Picture 2> Composing rhythm and

write in ‘Rhythm grid score’.

Writing a staff notation Students wrote their products in a staff notation using ‘Rhythm grid score’. Many students felt

difficulties to principle of a staff notation still, so a teacher needed to teach them locations of tones in a

staff notation first. Also, it was hard to divide a staff notation through the beat, dividing the score was

skipped in this study. After writing tones, they wrote down the title of their product at the top of the score.

Composing melody by touching a

piano application freely.

Writing down own melody in

‘Grid score’.

A mental weakness student was

also participated in this study.

263

<Picture 3> Writing a staff notation in ‘Rhythm grid score’.

Peer evaluation and posttest Students played own product with a piano application or a recorder. And others evaluated by giving

a student from 1 to 5 points with 2 standards. First was ‘Is flowing of melody natural?’ and ‘Did he(or

she) use various types of rhythm?’. After peer evaluation, students were participated in posttest

questionnaire same with pretest one.

<Picture 4> Playing own product and do peer evaluation.

Findings and Interpretations

<Table 2> Posttest questionnaire result

1. Have you ever composed music on your

own?

This question is omitted because they already have

done music composing through this class.

2. Do you like music? Yes (very much, a little bit) : 0


3. Do you have confidence about music? Yes (very much, a little bit) : 0


4. Can you read a staff notation? Yes (very much, a little bit) : 0


5. Do you have expectation about music

composing?

Yes (very much, a little bit) : 15


Improvement of score reading skill Through posttest, every student answered that they could read a staff notation. Also, they could do

better in singing, playing music instruments class.

264

Improvement 4 compositions of SMART education This study could improve 4 compositions of SMART education among ‘Self-directed learning,

Motivated, Adaptive, Resource free, Technology embedded’.

Self-directed learning Students could follow own learning speed using iPad. Music composing needs enough time to think

and touching every tones to find good melody following.

Motivated

Students had rare experiences using a digital device in a class. So, there was novelty effect which

made students motivated. Also, there were a successful music composing result through peer evaluation.

12 of students got more than 8 points in 10 points and 4 students got between 6~7 points. After they got

own points, they could feel confidence , interests and motivation about music composing.

Resource free

Samik piano application was free contents in App store.

Technology embedded

Students could get the application and iPad individually.

CONCLUSION

The purpose of this study was improving students’ score reading skill in music composing class

through SMART education. To achieve this purpose, this study was conducted by SMART education to

overcome traditional composing class’s major difficulties. First, giving iPads with a piano application to

every student could help student to ‘motivate’ and give ‘resource free’, ‘technology embedded’

environment. Second, students could do ‘self-directed learning’. So, they could feel motivation about

music composing and satisfaction about their products.

Through this study, students could feel confidence and interests about music composing. Also, they

could improve score reading skill. Because they felt like that with music composing, and a music teacher

could observe they could singing and playing music instruments with score better than before this study.

In consequence, this study could fulfill the purpose by overcome limitations of traditional composing

class with SMART education. This conclusion means SMART education can enlarge the boundary of

teaching and learning realization.

The limitations of this study are, first, students needed much time to get used to iPad. It was first

time to input iPad to Gr. 3 students, they felt embarrassed when the application’s gone and didn’t know

basic instruction. So, if I would do this type of teaching again, I should teach them how to use digital

devices and applications above all. Second difficulty was novelty effect. Although this study was focused

on improvement of score reading skill, if this study input to classes which already got used to digital

devices like iPad, they might not feel much motivation and not achieve the purpose with less motivation

in the end. So, we need to think more how to motivate students not leaning on novelty effect from digital

devices in SMART education.

REFERENCES

Takako Mato(2009), Intergrating technology in the music classroom.

Sunhee Bang(2012), A study on strategies of self-directed learning to promote smart learning, In journal

of lifelong learning society.

Takako Mato(2009), Integrating technology in the music classroom, St. Mary’s college of Maryland.

Jinsook Kim(2012),SMART education school guidebook, KERIS.

265

The Effect of Awareness Information on Affect-based Trust

in Collaborative Problem-solving Learning: A Pilot Study

Jongsuk Song

[email protected]

Doctoral student

Hanyang University


Dongsik Kim

[email protected]

Professor

Hanyang University


ABSTRACT

This study was conducted as a pilot purpose. The initial purpose of the study was to explore the

effect of awareness information on affect-based trust and the development of competence-based trust.

However, it is beyond the scope of this study to examine the relationship between awareness information

and the development of competence-based trust. Two groups of four participants were involved. They

were asked to find solutions concerning school violence and bullying problems. As a result of the pilot

study, five issues (time limit, facilities, technical problems, selection of participants, and measuring

instruments) were revealed and should be more examined to achieve the purpose of this study.

Keywords: Collaborative learning, awareness information, affect-based trust

INTRODUCTION

Trust is one of the influential factors to determine the success of collaborative learning (Mayer,

1995; Kanawattanachai, 2002; Ge & Hsieh, 2005; Fransen et al., 2013). Of various dimensions of trust,

emphasis is placed on affect-based trust grounded in emotional bonds and friendship in this study.

Though emotional relationship or friendship among collaborators is one of the pivotal factors in

collaborative learning (Hartup, 1992, Kratzer et al., 2005), most of the current studies have shown that

there are several disadvantages. First, it is likely lead to free-riding or dominating effects, that is, group

members' attitude toward collaboration. (Fransen et al., 2013). Molleman (2005), for instance, argued that

affect-based trust may help strengthen mutual trust in a group however, it tends to result in someone’s

dominating behaviors (Fransen et al., 2013). While free-riding tends to occur when learners put less effect

into group learning due to their excessive trust or expectations on others' ability, dominating effects refers

to when one or two members dominate the entire group processes or activities (Dillenbourg, 2005) since

they may be overconfident about their ability in comparison with the others'.

To reduce the disadvantages of the negative aspects related to affect-based trust, one of the

instructional methods, as argued by Aggarwal (2004), is group composition. For free-riding effect, in

addition, the selection of learning tasks can be another alternative as there is high relevance between free-

riding and motivation (Wsezey et al., 1994). Similarly, Weibel and Frederique (2013) argued that high

interest in learning task can result in reduced free-riding behaviors.

Unlike the emphasis on the attitudes of collaborators’ activities toward collaborative learning, this

study was intended to highlight another aspect, that is, their converging activities such as discussion,

negotiation, and final decision making in collaborative problem-solving. Affect-based trust is often

associated with blind or biased trust, which may hinder balanced and accurate judgment or evaluation

(Nooteboom, 2002). Nooteboom (2002) further argues that such trust is likely lead to delusion which

266

keeps them from discerning the possibility of fallacy. In collaborative problem-solving environment, for

example, some contributions can be overestimated despite of possible errors or mistakes since the

contributors are considered the higher achieving ones, and vice versa.

It seems that group composition or choice of task discussed earlier is a possible instructional

solution however, awareness information is suggested as another possible answer in relation to the impact

of affect-based trust on converging processes of collaborative learning. Awareness information is defined

as information regarding collaborators' skills, knowledge, social activities, etc. (Janssen & Bodemer,

2013). In other words, they get information on who contributes more or less, who has more information,

who knows better about the given task, etc. When considering that students are mostly engaged in

collaborative learning under regular courses with any forms of grades and credits, it can be assumed that

they are sensitive to group performance and its results. This study, therefore, expects they use awareness

information when engaging in converging activities and less dependence on their affect-based trust,

probably resulting in biased and incorrect judgment and evaluation during converging processes.

RQ: To what extent does the provision of awareness information affect the collaborating learners’

dependence on their affect-based trust during converging activities?

METHOD

Two groups of four participants were involved in the pilot study. Participants in a control group

were 4-year college (S Univ.) students majoring in Education. Participants in an experimental group were

H middle school students. Each experiment was conducted in different places and at different times (the

control group on S Univ. campus on Oct 1st and the experimental group in a study room in Hanam City on

Oct 12th)

.

Both groups were given the same problem-solving task, “Solutions to school violence and bullying”.

The original task was to seek solutions from five groups concerned in the problem, that is, peer students,

teachers and schools, parents, local community, and government. While the control group was given this

original task, only two parties were included in the learning task of the experimental group for reasons of

(1) the procedures of collaborative learning in control group were simpler than the ones of the

experimental group, and (2) the participants in the experimental group are middle school students, so it

was taken into account that there would be need for the ease of the burden by the learning task.

There were two phases of collaboration, sharing the task-related materials and finding solutions

through converging activities including discussion and negotiation. The control group was only engaged

in the second type of collaboration (individual work in the rest of the task), instead, the experimental

group performed both.

Two online mapping tools were employed, Minddomo (www.minddomo.com) and Groupzap

(www.groupzap.com). To measure participants’ affect-based trust, Affect-based trust (5 items) in

McAllister’s Interpersonal Trust Measure and McGill Friendship Questionnaire – Friendship Functions

(MFQ-FF) was also used. To collect awareness information-related data, this study focused on two parts,

the number of interaction occurred for each contribution and each individual’s score earned (students

evaluate each others’ solutions at the final phase of collaborative learning on a scale of 1 to 5).

LESSIONS LEARNED

To achieve the purpose of this study, five significant aspects should be taken into consideration:

time limit, facilities, technical problems, selection of participants, and measuring instruments. The

following problems, in particular, were identified in the pilot study. First, students wanted to make sure

that they invest limited amount of time (3hrs), as a result, did not have enough time to concentrate on the

task. Also to meet their request, the researcher had to conduct an orientation session and the main learning

session at once. Eventually, data collection was partly failed, particularly for control group (Time limit).

Second, it was hard to find an appropriate place to carry out the experiment since it was not allowed

to use any computer laboratory on the campus. Fortunately, college students in the control group brought

their own laptop, however middle school students did not have their own laptop, so the researcher had to

find any available laptops and provide them (Facilities).

267

Next, during the control group experiment, Minddomo interface suddenly did not provide “invite”

icon even though the researcher made sure that there was not any technical problem using Minddomo in

two time previous test. As a result, the control group failed to proceed with their collaborative learning,

particularly the only sharing phase. To cope with the occurrence of the same problem, the researcher

replaced it with Groupzap for the experimental group, which is much simpler and provides less panels

and icons (Technical problems).

Fourth, both groups were consisted of different age groups since it was also difficult to recruit

participants. Therefore, the study may be not ready enough to answer any questions in terms of the

selection of participants such as “Are there any significant issues to affect the outcomes with respect to

recruitment of participants?”, etc. (Selection of participants).

Finally, there were two main issues were revealed concerning measuring instruments: (1) McGill

questionnaire was used to support or provide any detailed information for McAllister’s affect-based trust

consisted of too small number of items. However, there were no meaningful differences between the

results from the two instruments. In addition, many items in the McGill questionnaire seemed too similar

to one another to participants though deep investigation may show slight difference between them. Even

the research explained the differences, they still looked confused. (2) Another issue is associated with the

validity of their ratings. The researcher provided a descriptive-type questionnaire as an extra data

collection to the experimental group. Of four questions, they were asked to choose one of the group

members in two questions. Three students answered just “All”, which possibly indicates rating inflation

in order to avoid awarding low scores on the others.

CONCLUSION

This pilot study suggests that: (1) participants should be provided with enough time and appropriate

learning environments in order to concentrate on the give task; (2) the researcher should prepare available

alternatives to cope with any possible technical problems concerning an online mapping tool; and (3) for

more valid and accurate data collection, measuring instruments need to be more developed, or additional

instruments can be included.

REFERENCES

Kratzer, J., Leenders, R.T.A.J. and Van Engelen, J.M.L. (2005), 'Informal contacts and performance in

innovation teams', International Journal of Manpower, 26(6), 513–28.

Aggarwal. (2004). Educational Technology. Sarup & Sons: New Delhi.

Antoinette Weibel & Frederique Six. (2013). Trust and Control: the Role of Intrinsic Motivation.

Reinhard Bachmann & Akbar Zaheer (eds.). Handbook of Advances in Trust Research, pp.57-81.

Edward Elgar Publishing, Inc: Massachusetts.

Hartup, W. (1992) “Having Friends, Making Friends, and Keeping Friends: Relationships as Educational

Contexts.” ERIC Digest, ED345854. Urbana, IL: ERIC Clearinghouse on Elementary and Early

Childhood Education. Available at http://bern.library.nenu.edu.cn/upload/soft/0-article/013/14021.doc,

accessed on Oct 22, 2013.

Janssen, J., & Bodemer, D. (2013). Coordinated computer-supported collaborative learning: awareness

and awareness tools. Educational Psychologist, 48(1), 40-55.

Jos Fransen, Armin Weinberger & Paul A. Kirschner. (2013). Team effectiveness and team development

in CSCL. Educational Psychologist, 48(1), 9-24.

McAllister, D. (1995). Affect- and cognition-based trust as foundations for interpersonal cooperation in

organizations. Academy of Management Journal, 38(1), 24-59.

Nooteboom, B. (2002), Trust: Forms, Foundations, Functions, Failures and Figures, Edward Elgar:

Cheltenham.

Pierre Dillenbourg. (2005). Designing Biases That Augment Socio-Cognitive Interactions. Computer-

Supported Collaborative Learning Series, 5, 243-264.

Robert W. Swezey, Andrew L. Meltzer, and Eduardo Salas. (1994). Some Issues Involved in Motivating

Teams. Harold F. O'Neil & Michael Drillings. (eds.). Motivation: Theory and Research. pp.141-170.

Routledge: New Jersey.

268

http://bern.library.nenu.edu.cn/upload/soft/0-article/013/14021.doc

Student's Perception on Learning Analytics Dashboard (LAD)

Presenting Online Activities in LMS

Stephanie Kang [email protected]

Graduate Student


Seoul, Korea

Yeonjeong Park

[email protected]

Research Professor


Seoul, Korea

Il-Hyun Jo

[email protected]

Associate Professor


Seoul, Korea

ABSTRACT

The purpose of this study is to investigate how students react and perceive a dashboard treatment

“Learning Analytics Dashboard (LAD)” designed and developed by the researchers. 37 students in a large

university were invited as subjects. LAD represents individual learners’ activities such as online activity

summary, total log-in time, total log-in frequency, log-in regularity, visits on board, time spent on board,

and visits on repository. These data were analyzed and visualized by statistical algorithms and provided as

graphs on LAD. The survey for students’ perception and understandng were conducted as data collection

method and the repeated measure ANOVA was conducted as data analysis method. We found that

students easily understood the graphs and information in LAD, and felt medium conformity between the

actual online activities shown on the graphs and the online activity they perceived themselves. Also,

participants’ perceived usefulness and degree of understanding on each item were significantly different,

while the difference among items in conformity was not significant. This study is meaningful in terms of

suggesting implication for the development of more refined and effective dashboard treatment, and

providing specific directions for future research.

Keywords: Dashboard, Data mining, LMS

INTRODUCTION

As e-Learning develops, the role of Learning Management Systems (LMS) has expanded from a

provider of contents to a system that helps us managing contents and learning process at the same time by

enabling learners to participate in various learning activities and to manage their own learning time and

place in cyber space (Britton & Tesser, 1991; Zaiane & Luo, 2001). Since an increasingly large number

of educational resources have moved to online, an enormous amount of student’s behavioral data is

accumulated as web-log files. Learning Analytics is an emerging field that extracts valuable information

and knowledge from web-log data to improve students’ learning performance.

In relation to learning performance, learning analytics has addressed a range of issues such as

mining processes, performance prediction, and outlier detection. Since its recent birth, learning analytics

269

research has been flourishing in the area of analysis of learner behaviors and prediction for the outcomes

(Baker & Yacef, 2009; Siemens & Long, 2011). However, the research has yet gone further to the

development of instructionally effective treatment based on the analysis and prediction efforts. Design

and development of effective interventions to control the outcome of the behavior based on the analysis

and prediction efforts should be educational technologists’ ultimate task.

In educational research field, various kinds of educational treatments including tasks, learning

environment, instructional methods have been suggested. Among them, a visualized dashboard utilizing

data mining is widely used because it helps improvement of self-knowledge by providing information

about learner’s own activity (Verbert, Duval, Klerkx, Govaerts, & Santos, 2013), and it promotes self-

evaluation and self-encouragement (Duval, 2013). For example, Essa & Ayad (2012) analyzed three

factors of learner’s success and classified at-risk student, then provided those information with graphs.

Leony and his colleagues (2012) also introduced GLASS, the open-source program which analyzes and

visualizes leaner’s web-log data. However, unfortunately, these statistical representations were not

intuitively understood (Koulocheri & Xenos, 2013). Previous studies suggest thorough investigation on

how the learners perceive and react to the dashboard treatment.

The present study is to investigate the student’s perception on a visual dashboard system designed

and developed by the researchers. 37 students from two different classes in a large university were invited

as subjects. Our online dashboard called “Learning Analytics Dashboard (LAD)” represents individual

learners’ activities such as online activity summary, total log-in time, total log-in frequency, log-in

regularity, visits on board, time spent on board, and visits on repository. These data were analyzed and

visualized by statistical algorithms. The results were provided as dashboard treatment and the survey for

student perception and understanding was conducted.

METHOD

Participants 37 college students in E women’s university in Seoul, Korea were provided LAD via LMS of the

university. Answers of 22 students who answered to the survey were analyzed. Due to the nature of

university, all participants were women, and their year in school were considerably evenly distributed.

Dashboard Development The researchers have collaborated with Institute for Teaching & Learning (ITL) of E university on

LAD development project. The dashboard consists of 7 graphs: each representing the online activity

summary, total log-in time, total log-in frequency, log-in regularity, visits on board, time spent on board,

and visits on repository. As shown in Figure 1, the graph of online activity summary is the scatterplot that

individual learners can choose X-axis and Y-axis to locate their position in class. Other 6 graphs are

provided with trend line of their activity of every week along with average activity information of other

learners in the class. All of graphs in LAD are planned to update every week until the end of semester.

See Figure 1 for the screenshot of LAD. Name of student is covered to protect personal information.

Measurement Survey questionnaire was developed by researchers to measure conformity, perceived usefulness,

and degree of understanding. It consists of 24 questions (21 Likert 5-scale questions and 3 open-ended

questions) to gather the participants’ opinions and suggestions on LAD. Table 1 shows the summary of

survey questionnaire.

Procedure After development of LAD, the researchers provided a brief instruction regarding the dashboard and

asked for the survey verbally and by e-mails. At this point, the participants could access LAD freely (and

repeatedly if they want) via LMS they have been used. They were asked to complete the survey in 5 days

after they were provided LAD and 22 out of 37 participants completed the survey. The researcher

analyzed and compared the means of each questions.

270

Figure 1. Screenshot of “E Learning Analytics Dashboard (LAD)”

<Table 1> Summary of Survey Questionnaire

Part Contents Example N of

Questions Measure

Conformity Degree of conformity

between learner’s

perceived online activity

and real data

How much the total log-in

time graph conforms to your

perceived total log-in time? 7 Likert 5-Scale

Perceived

Usefulness

Degree of learner’s

perceived usefulness of

the information in LAD

How much do you think the

total log-in information will

help your learning process?

7 Likert 5-Scale

Degree of

Understanding

Degree of learner’s

understanding of the

graphs in LAD

How difficult it is to fully

understand the total log-in

time graph?

7 Likert 5-Scale

Opinion and

Suggestion

Participant’s opinion and

suggestion on LAD

Please suggest any opinion

you have on this dashboard. 3 Open-ended

RESULTS

As shown in Figure 2, in comparing the conformity, perceived usefulness, and degree of

understanding for all graphs in LAD, the result indicates that higher degree of understanding(Total Mean

=4.10), relatively lower degree of the perceived usefulness(Total Mean=3.22), and medium level of

271

conformity (Total Mean=3.70). That is, students perceived the information in dashboard reflected their

online activities in LMS. While they could understand the meaning of graphs quite well, it was somehow

weak to perceive this information as useful tools for their learning and performance.

Figure 2. Overview for Comparison (n=22)

<Table 2> Descriptive statistics of conformity, perceived usefulness, and level of understanding (n=22)

Categories

Variables

Conformity Perceived Usefulness Degree of

Understanding

Mean SD Mean SD Mean SD

Online Activity Summary 3.36 .790 3.36 .790 3.73 1.077

Total Log-in Time 3.55 .963 2.68 .839 4.23 .869

Total Log-in Frequency 3.86 .774 3.09 .971 4.32 .839

Log-in Regularity 3.73 .827 2.59 1.008 3.86 1.125

Visits on Board 3.77 .922 3.68 .839 4.14 .990

Time Spent on Board 3.73 .935 3.27 1.032 4.14 .941

Visits on Repository 3.91 .684 3.86 .889 4.32 .839

Total Mean 3.70 .842 3.22 .910 4.10 .954

Conformity The answers of conformity questions were moderately high (Total Mean=3.70) in general. Among

them, the online activity summary (Mean=3.36) and the total log-in time (Mean=3.55) showed relatively

low level of conformity. There were relatively more students who answered the total log-in time and the

log-in frequency information in LAD were different from what they though. In regard to this result, we

suspect that students often stayed online without logging-out even after they achieved their goal of action.

However, this result needs to be carefully observed since the result of repeated measures ANOVA

indicated that no significant mean difference among the 7 variables was detected (F5 = 1.20, p > .05).

272

Perceived Usefulness Students generally have medium perception on the usefulness of LAD (Total Mean=3.22). It means

that they think and expect the information from LAD would help their learning or learning process

somehow but not very much. Follow-up repeated measures ANOVA for Perceived Usefulness detected

significant large mean difference among the 7 variables (F5 = 13.52, p < .001). Participants perceived that

most useful item was the visit on repository, followed by the visit on board, the online activity summary,

the time spent on board, the total log-in frequency, the total log-in time, and the log-in regularity,

respectively. The total log-in time (Mean=2.68) and the log-in regularity (Mean=2.59) showed lowest

level of perceived usefulness. Also, perceived usefulness of the log in regularity is the only question that

received extremely negative answer, “I do not think this information will help my learning at all”.

Degree of Understanding In terms of degree of understanding, students felt it is easy to understand most of the graphs in LAD

(Total Mean=4.10). Another repeated measures ANOVA for Degree of Understanding reports noticeable

significant mean difference among the 7 variables (F5 = 3.17, p < .01). Degree of understanding presented

highest on the total log-in frequency and the visit on board, followed by the total log-in time. The visit on

board and the time spent on board have same mean values, and the lowest degree of understand was

shown in the log-in regularity and the online activity summary. There were many negative answers on the

online activity summary (Mean=3.73) and the log-in regularity (Mean=3.86) suggesting that students

were experiencing difficulties to fully understand those two graphs. Since the online activity summary is

the scatterplot that users should choose X-axis and Y-axis by themselves, it can be assumed that it might

be difficult to use it and understand the information without detailed explanation or manual. In case of the

log-in regularity, the difficulty that students experienced was due to the misunderstanding of the concept

itself. To solve this problem and help students’ understanding, it is necessary to provide the manual of

LAD including the concept of each item and how it is presented as graph.

Discussion & Conclusion

In this study, we found that students easily understood the graphs and information they represent,

and felt medium conformity between the actual online activities shown on the graphs and the online

activity they perceived themselves. Regarding the result of the repeated measure ANOVA, participants

answered consistently on all items on conformity. However, the answers for the open-questions provide

us meaningful implications. In the answers of open-ended questions, students mentioned: “total log-in

time is not the same as my study time”. Such response confirms that even though the time spent on

learning is important for students’ learning achievement (Rau and Durand, 2000), the time they stay

logged-in on LMS does not necessarily mean the time they spent on learning (Cotton and Savard, 1981).

Furthermore, unlike our expectation that participants would perceive LAD useful and feel easy to

understand on all items, the results of the repeated measure ANOVA represent that participants’ different

perceptions on perceived usefulness and degree of understanding. Specifically, participants answered that

the log-in regularity would not help their learning. This result is contradictory with the result of preceding

research that the log-in regularity predicts higher learning achievement (Jo & Kim, 2013). Considering

the answers for open questions (i.e. “It is hard to understand log-in regularity”, “What is the unit or

scale?”, etc.) and low degree of understanding on the log-in regularity, it is possible that participants

experienced difficulty to understand intuitively, so that they did not apprehend the concept of the regular

learning which presented in LAD.

Regarding to the purpose of this study to investigate the students’ perception and understanding on

LAD, in spite of relatively lower levels of perceived usefulness shown in this study, it is remarkable that

the results of the survey showed high perceived usefulness and degree of understanding. Only one

extremely negative answer was show up throughout the survey, even the relatively lower category

showed moderate positiveness. It indicates that students reacted to LAD positively in general, so that we

expect the quality and potential effectiveness of LAD.

The implications of this study raise the needs of empirical research on actual effect of LAD. More

specifically, whether this graphs and information they represent would help students’ learning or not

273

should be clarified. With those future researches, we would be able to refine or change the items and

information we present on LAD to improve its usefulness. For this process, we need to consider that

participants’ perceived usefulness and degree of understanding on each item are significantly different,

while the difference among items in conformity was not significant. It is necessary to carefully examine

the reasons of these results by follow-up empirical researches. In last, another comment from open-ended

question such as “the usefulness is up to how each class utilizes LMS” implies the usefulness and effect of

LAD would be very different between 100% online class and blended learning class. Therefore, as a

further study, it is necessary to examine the effect of LAD empirically in different contexts to prove its

usefulness more precisely.

In conclusion, this study analyzed how end-users react and perceive developed dashboard treatment.

This study is meaningful in terms of suggesting implication for the development of more refined and

effective dashboard treatment, and providing specific directions for future research.

REFERENCES

Baker, R. S. J. D., & Yacef, K. (2009). The state of educational data mining in 2009: A review and future

visions. Journal of Educational Data Mining, 1(1), 3-17.

Britton, B. K., & Tesser, A. (1991). Effects of time-management practices on college grades. Journal of

educational psychology, 83(3), 405.

Cotton, K., & Savard, G. W. (1981). Time factors in learning. research on school effectiveness project:

Topic summary report. Retrieved from

http://search.proquest.com.access.ewha.ac.kr/docview/63544228? accountid=10785

Essa, A., & Ayad, H. (2012). Student success system: risk analytics and data visualization using

ensembles of predictive models. In Proceedings of the 2nd International Conference on Learning

Analytics and Knowledge. 158-161.

Jo, I. H., & Kim, Y. (2013). Impact of Learner’s Time Management Strategies on Achievement in an e-

learning Environment: A Learning Analytics Approach. The Journal of Educational Information

and Media, 19(1), 83-107.

Koulocheri, E., & Xenos, M. (2013). Considering formal assessment in learning analytics within a PLE:

the HOU2LEARN case. In Proceedings of the Third International Conference on Learning


Leony, D., Pardo, A., de la Fuente Valentín, L., de Castro, D. S., & Kloos, C. D. (2012). GLASS: a

learning analytics visualization tool. In Proceedings of the 2nd International Conference on

Learning Analytics and Knowledge. 162-163.

Rau, W., & Durand, A. (2000). The academic ethic and college grades: Does hard work help students to"

make the grade"?. Sociology of Education, 19-38..

Siemens, G., & Long, P. (2011). Penetrating the fog: Analytics in learning and education. Educause

Review, 46(5), 30-32

Verbert, K., Duval, E., Klerkx, J., Govaerts, S., & Santos, J. L. (2013). Learning Analytics Dashboard

Applications. American Behavioral Scientis, 57(10). 1500-1509.

Zaïane, O. R., & Luo, J. (2001). Towards evaluating learners' behaviour in a web-based distance learning

environment. In Advanced Learning Technologies, 2001. Proceedings. IEEE International

Conference on (pp. 357-360). IEEE.

274

275

user

Stamp

user

Stamp

INTERNATIONAL CONFERENCE OF EDUCATIONAL TECHNOLOGY · Improving the Usability of E-Learning User...

Documents

Transcript of INTERNATIONAL CONFERENCE OF EDUCATIONAL TECHNOLOGY · Improving the Usability of E-Learning User...