MOOQ and the Quality of MOOCs: Findings & Toolsmooc-quality.eu/wp-content/uploads/2018/08/... ·...

MOOQ and theQuality of MOOCs:

Findings & ToolsChristian M. Stracke & Esther Tan

Open University of the Netherlands

Global cooperation: ECNU & KNOU

Global initiative ICORE for OR & OE

International WLS / LINQ Conference

eLC European Institute

ICDE Chair in OER

Dr. Christian M. Stracke:Open Learning & Education, Innovations, Policies, Quality & Competences, Impact

Open University of the Netherlands

The Quality of MOOCs

Let’s Learn to Learn

Seamless Learning

Dr. Esther TanTechnology-Enhanced Learning, Innovations in & out Classroom

Presentation Outline1. The Quality of MOOCs: What is the problem?

2. A Conceptual Framework towards MOOC QRF:Theoretical & Methodological Approach

3. Findings from MOOC surveys: Learners, Designers & Facilitators

4. Findings from MOOC OEQ:Learners, Designers & Facilitators

5. Findings from MOOC semi-structured interviews:Providers

Any discussion on the quality of MOOCs should consider the goalsof both, the MOOC learner and the provider. (Hayes, 2015)

Critical questions to be addressed:

Who are the MOOC users, and why?

What makes a good MOOC from the learners’ experienceswith MOOCs?

What are the best design principles and best practicesas indicators of quality?

Are these quality indicators also MOOC domain specific and/orMOOC type specific?

Problem Statement

A Research Framework for MOOQ“When one designs any course, one has to have some learnercohort in mind.” (Macleod et al., 2015, p. 9)

Main research goals:

1. Establish a research framework for the subsequent analysis ofMOOC design patterns and best practices

2. Develop a Quality Reference Framework (QRF) for MOOCs

Theoretical Framework

Global MOOCQuality Survey

Quality Reference Framework withcriteria & checklist for MOOC design

Our main goal is the collaboration with allto improve Open Education & MOOCs

www.MOOC-quality.eu

http://www.mooc-quality.eu/

Subjects of Investigation: Learners, Designers, Facilitators & Providers

Mixed Methods Approach: Quantitative & qualitative data

1. Global MOOC Quality Survey,

2. Open-Ended questions (OEQ) &

3. Semi-structured interviews

Methodological Framework

Instruments of Measure

MOOC Learners

MOOC Designers

MOOC Facilitators

MOOC Providers

TOTAL

Global MOOC Quality Survey

166(69 qns.)

68(89 qns.)

33(58 qns.)

- 267

Open-ended Questions

118(4 qns.)

42(4 qns.)

27(4 qns.)

- 185

Semi-structured interviews

-12

(15 qns.)12

(10 qns.)12

(13 qns.)36

MOOC Survey Constructs

Constructs Learners Designers Facilitators

Experience with MOOC X X X

Learning Objectives X X X

Duration and Structure X X

Learning Resources X X X

Accessibility and Inclusion X X

Learning Progress X

Learning Environment X X

Learning Assessment X X X

Learning Certification X X

Design Process X

Pedagogical Decisions X

Learning Support Feedback & Facilitation Interaction & Collaboration

X X X

Global MOOCQuality Survey

(GMQS)

Global MOOC Quality Survey

Demographic Profile

Age range of all survey participants (learners, designers & facilitators) by gender

Demographic Profile

Educational level of all survey participants (learners, designers & facilitators) by gender

n VB B N G VGLearningexperience

166 4 4 13 75 70

Learning Experience (Learners)

n VB B N G VGDesignexperience

68 1 2 13 33 19

Design Experience (Designers)

Interaction from Learners‘ Perspective

n N/A SD D N A SA

LF 146 20 5 13 48 37 23

LL 146 15 3 17 34 51 26

LR 146 9 2 8 25 61 41

GG 146 37 4 15 50 24 16

Note:LF: Interaction between learners and facilitatorsLL: Interaction among learners LR: Interaction between learners and learning resourcesGG: Interaction among teams and groups

n R2 M2 p

LF by learners

125 .094 9.382 .000***

LL by learners

130 .101 10.818 .000***

LR by learners

136 .112 12.286 .000***

GG by learners

108 .045 4.131 .026*

Bivariate Correlations between LLR4 and LLE4

Interaction from Designers‘ Perspective

n N/A SD D N A SA

LF 52 2 1 5 11 24 9

LL 52 1 1 3 11 19 17

LR 52 3 1 0 4 22 22

GG 52 8 2 10 14 13 5


n R2 M2 p

LF by designers

49 .003 0.109 .703

LL by designers

50 .043 1.595 .143

LR by designers

48 .046 1.537 .138

GG by designers

43 .001 0.038 .821

Bivariate Correlations between DLR4 and DDE4

Interaction from Facilitators‘ Perspective

n N/A SD D N A SA

LF 32 0 1 3 3 15 10

LL 32 1 0 3 2 11 15

LR 32 0 0 0 5 12 15

GG 32 5 4 3 3 9 8


n R2 M2 p

LF by fa-cilitators

30 .181 0.242 .015*

LL by fa-cilitators

29 .021 0.296 .435

LR by fa-cilitators

30 .030 0.287 .342

GG by fa-cilitators

25 .000 0.305 .971

Bivariate Correlations between FLR4 and FDE4

Open-ended Questions(Learners)

No of respondents across the six domains

Domain Learners Designers Facilitators

Social Sciences, Humanities & Law

24 9 2

Education & Lifelong Learning 19 12 18

Computing & Informatics 18 5 2

Science, Math & Engineering 16 5 1

Nature, Environment & Health 21 6 1

Business, Management & Economics

20 5 3

Total 118 42 27

MOOC Open-ended Questions (OEQ)

MOOC Learners

Q1. What were the three main strengths of the MOOC?

Q2. What were the three main weaknesses of the MOOC?

Q3. What was missing in the MOOC?

Q4. Looking ahead into the development of this type oflearning experiences, what could be improved infuture MOOCs?

Age Range of MOOC Learners

18-24 25-34 35-44 45-54 55-64 65-74 >75 Total


0 3 6 9 1 1 0 20

Nature, Environment & Health 1 5 4 1 6 4 0 21

Science, Math & Engineering 0 3 4 6 0 3 0 16

Computing & Informatics 1 4 5 4 4 0 0 18

Education & Lifelong Learning 1 1 7 3 5 2 0 19

Social Sciences, Humanities &Law 1 4 6 6 4 2 1 24

Total 4 20 32 29 20 12 1 118

Demographics of MOOC Learners

Educational Level of Learners

NoSchool-

ing

High school

Bachelor’sdegree

Masters degree

Doctorate degree Total


0 1 4 10 5 20

Nature, Environment & Health 0 2 6 12 1 21

Science, Math & Engineering 0 1 4 7 4 16

Computing & Informatics 0 0 3 9 6 18

Education & Lifelong Learning 1 0 4 5 9 19

Social Sciences, Humanities &Law 0 0 4 12 8 24

Total 1 4 25 55 33 118


Gender of MOOC Learners

Male Female Other Total


12 8 0 20

Nature, Environment & Health

14 7 0 21

Science, Math & Engineering

7 9 0 16

Computing & Informatics

8 10 0 18

Education & Lifelong Learning

11 8 0 19

Social Sciences, Humanities &Law

10 13 1 24

Total 62 55 1 118


Learning Experience

Very Bad Bad Neutral Good Very

Good Total


1 0 2 9 8 20


0 0 0 9 12 21


0 0 1 2 13 16


1 1 1 8 7 18


1 1 0 9 8 19


0 0 1 15 8 24

Total 3 2 5 52 56 118

Open-ended Questions (MOOC Learners)

Open-ended Questions (MOOC Learners)% of Activities Completed

None <25% =50% >75% >100% Total


1 2 2 3 12 20


0 5 2 4 10 21


0 1 3 3 9 16

Computing & Informatics 2 3 1 4 8 18


1 3 2 5 8 19


1 2 3 9 9 24

Total 5 16 13 28 56 118

Open-ended Questions (MOOC Learners)Q1. What were the three main strengths of the MOOC?

0

10

20

30

40

50

60

70

80

90

CurriculumDesign &Delivery

InstructionalDesign &

Technology

Assessment &Evaluation

Facilitation &Feedback

Interaction &Collaboration

Nu

mb

er o

f C

om

men

ts

Social Sciences, Humanitiesand Law

Education and LifelongLearning

Computing and Informatics

Science, Maths andEngineering

Nature, Environment andHealth

Business, Management andEconomics

Open-ended Questions (MOOC Learners)Q2. What were the three main weaknesses of the MOOC?

0

5

10

15

20

25

30

35

Curriculum Design InstructionalDesign &

Technology

Assessment &Evaluation

Faciliation &Feedback

Interaction &Collaboration

Nu

mb

er

of

Co

mm

ents

Social Sciences, Humanities and Law

Education and Lifelong Learning

Computing and Informatics

Science, Maths and Engineering

Nature, Environment and Health

Business, Management and Economics

Open-ended Questions (MOOC Learners)

Three main strengths and weaknesses of the MOOC

Strengths Weaknesses

Curriculum Design & Delivery

• Good choice & quality of content• Good teachers, presenters & tutors• LO aligns with content

• Weak choice & quality of content• Short duration

Instructional Design & Technology

• Good integration of IT & media • Support self-regulation & individual

learning paths

• Poor use of IT technological tools• Resources lack variety & quality

Interaction & Collaboration

• Encourage local group discussion & activities

• Foster interaction with field experts

• Lack interaction: learner-tutor• No support for community building


(Designers)

MOOC Open-ended Questions (OEQ)MOOC DesignersQ1. Which were the main design decisions that you made

during the development of the MOOC that later proved tobe successful?

Q2. Which were the three biggest difficulties that you facedwhen designing the MOOC?

Q3. Which design decisions did not pay off as you expected?

Q4. Looking ahead into the development of this type of learningdesign experiences, what methods and tools could behelpful to improve the design of future MOOCs?

Age Range of MOOC Designers

25-34 35-44 45-54 55-64 65-74 >75 Total


1 1 3 0 0 0 5


1 3 0 1 0 1 6


1 1 1 1 1 0 5


0 1 4 0 0 0 5


1 2 5 3 1 0 12


1 1 1 4 1 1 9

Total 5 9 14 9 3 2 42

Demographics of MOOC Designers

Educational Level of Designers

High school

Bachelor’sdegree

Masters degree



0 0 4 1 5


0 0 3 3 6


0 0 1 4 5


1 0 1 3 5


0 0 3 9 12


0 2 3 4 9

Total 1 2 15 24 42


Gender of MOOC Designers

Male Female Total


3 2 5


2 4 6


2 3 5


2 3 5


5 7 12


6 3 9

Total 20 22 42


Open-ended Questions (MOOC Designers)No. of MOOCs Designed

1 2 - 4 5 - 9 >10 Total


3 2 0 0 5


2 1 2 1 6


1 2 2 0 5

Computing & Informatics 3 2 0 0 5


4 5 1 2 12


3 4 2 0 9

Total 16 16 7 3 42

Design Experience

Very Bad Bad Neutral Good Very

Good Total


0 0 0 5 0 5


1 0 1 3 1 6


0 0 1 2 2 5


0 0 1 3 1 5


0 0 4 5 3 12


0 1 0 3 5 9

Total 1 1 7 21 12 42

Open-ended Questions (MOOC Designers)

Open-ended Questions (MOOC Designers)Q1. Which were the main design decisions that you made during thedevelopment of the MOOC that later proved to be successful?

1

3

3

42

1

1

3

3

6

5

4

0

4

8

12

16

20

24

Assessibility&Inclusion

Assessment&Evaluation

Interaction&Collaboration

Certification&Accreditation

CurriculumDesign&Delivery

Expertise&Manpower

Feedback&Facilitation

InstructionalDesign&

Technology

No.ofCom

ments

SocialSciences,HumanitiesandLaw

EducationandLifelongLearning

ComputerandInformatics

Science,MathsandEngineering

Nature,EnvironmentandHealth

Business,ManagementandEconomics

Open-ended Questions (MOOC Designers)Q2. Which were the three biggest difficulties that you faced when designing theMOOC?

3 3

1

3

32

81

2

2

1

1

1

2

3

5

4

4

2

5

3

0

4

8

12

16

20

24

Assessment&Evaluation

CurriculumDesign&Delivery

Expertise&Manpower

Feedback&Facilitation

InstitutionalSupport&

Funding

InstructionalDesign&

Technology

Interaction&Collaboration

OpenAccess,Copyrights&

Licensing

No.ofComments

SocialSciences,HumanitiesandLaw

EducationandLifelongLearning

ComputerandInformatics

Science,MathsandEngineering

Nature,EnvironmentandHealth

Business,Managementand

Economics

Open-ended Questions (MOOC Designers)

Successful Decisions Biggest Challenges

Curriculum Design & Delivery

• Content delivery format• Content structure & LOs

• Weak choice & quality of content• Short duration

Instructional Design & Technology

• Choice of learning activities• Integration of IT & media

• Platform, software & production decisions

Interaction & Collaboration Expertise & Manpower

• Creating community of learners• Foster interaction between learner

& tutor/ facilitator

• Gap in content & instructional design knowledge

• Coordination & collaboration, e.g., different experts & teaching staff

Three main successful decisions and three biggest challenges


(Facilitators)

MOOC Open-ended Questions (OEQ)

MOOC Facilitators

Q1. Which were the main decisions that you made during thefacilitation of the MOOC that later proved to be successful?

Q2.Which were the three biggest difficulties that you facedwhen facilitating the MOOC?

Q3 Which facilitation decisions did not pay off as you expected?

Q4 Looking ahead into the development of this type of learningexperiences, what methods and tools could be helpful toimprove the facilitation of future MOOCs?

Age Range of MOOC Facilitators

25-34 35-44 45-54 55-64 65-74 >75 Total


0 1 0 0 0 2 3


0 0 0 1 0 0 1


0 1 0 0 0 0 1


1 0 0 1 0 0 2


0 6 6 3 3 0 18


0 0 1 1 0 0 2

Total 1 8 7 6 3 2 27

Demographics of MOOC Facilitators

Educational Level of MOOC Facilitators

High school

Bachelor’sdegree

Masters degree



0 0 1 2 3


0 0 0 1 1


0 0 0 1 1


1 0 1 0 2


0 4 8 6 18


0 0 2 0 2

Total 1 4 12 10 27


Gender of MOOC Facilitators

Male Female Total


0 3 3


1 0 1


1 0 1


0 2 2


10 8 18


0 2 2

Total 12 15 27


Open-ended Questions (MOOC Facilitators)No. of MOOCs Facilitated (OEQ)

1 2 - 4 5 - 9 >10 Total


2 0 0 1 3


0 0 0 1 1


1 0 0 0 1

Computing & Informatics 1 1 0 0 2


6 7 2 3 18


0 2 0 0 2

Total 10 10 2 5 27

Facilitation Experience

Good Very Good Total


3 0 3


0 1 1


1 0 1


1 1 2


12 6 18


1 1 2

Total 18 9 27

Open-ended Questions (MOOC Facilitators)

0

1

2

3

4

5

6

7

8

9

Provide

guidelines on

attendance, quiz

& test

Monitor peer

review process

& forum

discussion

Show presence

& provide

feedback

Leverage

technological

tools to support

e-learning

Promote

interaction &

group dynamic

Provide quality

content &

appropriate

pedagogical

approach

No. of

Com

men

ts

Successful Facilitation Decisions

Social Sciences, Humanities &

Law

Education & Lifelong

Learning




Business, Management &

Economics

0

1

2

3

4

5

Difficult to

engage &

retain learners

Integrate IT

& Media to

support

learning

Lack IT

competencies

(learners)

Mismatch of

content and

target learners

Overload

learners with

quizzes &

assignments

Provide free

certfication

and course

Unable to

foster

interaction &

collaboration

No. o

f C

om

men

ts

Unsucessful Facilitation Decisions

Social Sciences,

Humanities & Law

Education & Lifelong

Learning


Science, Math &

Engineering

Nature, Environment &

Health

Business, Management &

Economics

Semi-Structured Interviews Questions

(Providers)

Categories Descriptor Indicators

Role of MOOCs in HE Providers perspective

on the role of MOOCs

in the current HE

national and

international scenario

Pedagogical innovation

Institutional culture: Toward

hybrid & blended courses;

teaching internally & reaching

out externally

Reasons to provide

MOOCs

Identification of causes

for offering MOOCs

• Institutional mission: to “place

people” as the largest course

provider in FutureLearn; foster

openlearn in the eco system;

educational research on

MOOCs; public engagement to

showcase Uni; capacity

development within the Uni

• Institutional goal: Foster

innovation in digital learning;

stay relevant in the changing

educational ecosystems

Summative Report of Coded Interviews


Partnerships Establishment, organization

and management of

partnerships

Partnership structure: Shared

network with a core group of

universities to share research

project

Great collaboration between

researchers from various

disciplines (excellent model);

good to have external driver to

promote internal change; helped

in best practices.

Institutional

Implementation

Policies

Theoretical design of the

MOOC business model to

implement

More strategic approach

developed: lifelong learning,

apprenticeship; MOOCs as

experimentation spaces -

separate from other learning &

teaching platforms

Business model for delivering

MOOCs: undergraduate and

masters degrees) & chargeable

certification (for some types)



Institutional

organizational

strategies

Description of the MOOC

organization and

development process in

the institution

Support structures: 1. Set up a

content team – work with partners

to design well-designed courses

with quality; 2. Uni put in place a

number pedagogical coordinators

to help teaching staff to prepare

the course and provide ideas and

experiences, technical staff for

recording and post production,

staff for subtitles. 3. Central unit

works with individual academics

to find the best plan/ fit

Re-train/ Assimilate teachers into

the MOOC culture: provide

teachers a checklist on dos &

donts for content in an online

learning environment.



Sustainability Characterization of the

return on investment

Funding source(s):

1. Business model: a number of bis

model. Some funded by external org.

most funded internally thru OU open

learning budget (by media budget fr

media unit)

2. Budget from uni for central unit

plus some money from Edx (but not

sufficient) received from Edx every 3

months some money; the objective is

to improve education; not profit-

making.

3. Four mio raised funding externally

for the fully online masters study



Evaluation Design of the evaluation

process for the MOOC

policy established

Level of success for the

institution:

1. Grant awareness: revenue gains,

paid courses

2. Prestige & research gains.

3. Educational success: moving from

standard to hybrid & blended

learning in accredited courses

4. Community of learners: with other

likeminded people/ other Uni

5. Met objectives, research, Uni

promotion & capacity building

Specialist evaluation apart from

usual questionnaire on

institutional e-learning and

pedagogical innovation

pedagogical model


Every institution should always have a roadmap toimprove and extend the use of current courses, suchas MOOCs:

1. facilitates implicit social learning, and may also helpto attract new students to programmes.2, the sociocultural nature of students is changing inEurope and beyond so it is important for to undertakemulticultural adaptation of courses for the differentstudent types3. ECTS for MOOCs

Globalization strategies of the training spaces:MOOCs raise new research questions/problems thatdeserve further study (e.g., what are theconsequences of these new models to knowledgesharing/building on a global level).

Summative Report of Coded InterviewsFinal words from MOOC Provider

[email protected]@ou.nl

@ChrMStracke@Taneste

www.opening-up.education

Let’s collaborate!

mailto:[email protected]

mailto:[email protected]

http://www.opening-up.education/

Thank you!

Your questions?

To be continued…

Open CC License forsharing & re-using slides

This work is free to share under the creative commons licence:

"Attribution – Noncommercial – Share Alike 3.0"

You can copy, distribute and transmit the work under the following conditions:

1. Attribution –2. Noncommercial –

3. Share Alike

Licence: Attribution-Noncommercial-Share AlikeSome rights reserved, see: http://creativecommons.org/licenses/by-nc-sa/3.0/

First References for GMQS

Stracke, C. M., et al. (2018). Gap between MOOC designers' and MOOC learners' perspectives on interaction and experiences in MOOCs: Findings from the Global MOOC Quality Survey. In M. Chang, N.-S. Chen, R. Huang, Kinshuk, K. Moudgalya, S. Murthy, & D. G. Sampson (Eds.), Proceedings 18th IEEE International Conference on Advanced Learning Technologies (ICALT) (pp. 1-5). IEEE: Computer Society. DOI 10.1109/ICALT.2018.0000

Stracke, C. M., & Tan, E. (2018). The Quality of Open Online Learning and Education: Towards a Quality Reference Framework for MOOCs. In J. Kay, & R. Luckin (Eds.), Rethinking learning in the digital age. Making the Learning Sciences Count: The International Conference of the Learning Sciences (ICLS) 2018 (pp. 1029-1032). London: ISLS.

Stracke, C. M. et al. (2017). The Quality of Open Online Education: Towards a Reference Framework for MOOCs. In Proceedings of 2017 IEEE Global Engineering Education Conference (EDUCON) (pp. 1712-1715). IEEE Xplore. DOI: 10.1109/EDUCON.2017.7943080

To be continued …

The Quality Reference Framework

Dimension 1: Phases Analysis, Design, Implementation, Realization, Evaluation

Dimension 2: Perspectives Pedagogical, Technological, and Strategic

Dimension 3: Roles Designer, Facilitator, and Provider

MOOQ project:www.MOOC-quality.eu

Online community for theQuality Reference Framework:

www.MOOC-quality.net

http://www.mooc-quality.eu/

http://www.mooc-quality.net/

Session Title:

Minds-on session on Design

Processes for MOOCs

Facilitators: Christian M. StrackeCleo SgouropoulouNikos Palavitsinis

Session Title:

Dealing with practical quality problems when

running a MOOC

Facilitators:Bill Vassiliadis

Antonia StefaniEsther Tan

Achilles Kameas

Parallel Sessions

Session 1: Minds-on session on Design Processes for MOOCs

Insights on:• The Quality Reference

Framework in practice,• Design decisions,• Roles of designers,

facilitators and providers

Room: 3rd floor

Session 2: Dealing with practical quality problems when running a MOOC

Insights on:• Drop-out rates, • Failure to collaborate, • Fairness in automatic

assessment

Room: here (1st floor)

Parallel Sessions

MOOQ and the Quality of MOOCs: Findings & Toolsmooc-quality.eu/wp-content/uploads/2018/08/... ·...

Documents

Transcript of MOOQ and the Quality of MOOCs: Findings & Toolsmooc-quality.eu/wp-content/uploads/2018/08/... ·...