Using the Community of Inquiry Framework Survey for Multi-Level Institutional Evaluation

Phil Ice, Ed.D.26th Annual Conference on Distance Teaching & Learning

CoI Framework

A process model of learning in online and blended educational environments

Grounded in a collaborative constructivist view of higher education

Assumes effective online learning requires the development of a community of learners that supports meaningful inquiry and deep learning

Three Presences

Social Presence

The ability of participants in a community of inquiry to project themselves socially and emotionally -- as ‘real’ people

The degree to which participants in computer mediated communication feel socially and emotionally connected

Social Presence - Elements

Affective expression (expressing emotion, self-projection)

Open communication (learning climate, risk free expression)

Group cohesion (group identity, collaboration)

Cognitive Presence

The extent to which learners are able to construct and confirm meaning through sustained reflection and discourse in a critical community of inquiry

Cognitive Presence - Elements

Triggering event (sense of puzzlement)

Exploration (sharing information & ideas)

Integration (connecting ideas) Resolution (synthesizing & applying

new ideas)

Cognitive Presence - Elements

triggering event (sense of puzzlement)

exploration (sharing information & ideas)

integration (connecting ideas) resolution (synthesizing & applying

new ideas)

Teaching Presence

The design, facilitation and direction of cognitive and social processes for the purpose of realizing personally meaningful and educationally worthwhile learning outcomes

Teaching Presence - Elements

Design and organization (setting curriculum & activities)

Facilitation (shaping constructive discourse)

Direct instruction (focusing & resolving issues)

CoI Survey

9 social presence items (3 affective expression, 3 open communication, 3 group cohesion)

12 cognitive presence items (3 triggering, 3 exploration, 3 integration, 3 resolution)

13 teaching presence items (4 design & facilitation, 6 facilitation of discourse, 3 direct instruction)

CoI Survey - Validation

Tested in graduate courses at four institutions in the US and Canada

Principal component factor analysis Three factor model predicted by CoI

framework confirmed Arbaugh, Cleveland-Innes, Diaz,

Garrison, Ice, Richardson, Shea & Swan – 2008

Subsequent validation and cumulative n over 500,000

Community of Inquiry Survey Instrument (draft v15)Developed by Ben Arbaugh, Marti Cleveland-Innes, Sebastian Diaz, Randy

Garrison, Phil Ice, Jennifer Richardson, Peter Shea & Karen Swan 

Teaching PresenceDesign & Organization1. The instructor clearly communicated important course topi2. The instructor clearly communicated important course goals.3. The instructor provided clear instructions on how to participate in course learning activities.4. The instructor clearly communicated important due dates/time frames for learning activities.

Facilitation of Discourse5. The instructor was helpful in identifying areas of agreement and disagreement on course topics that

helped me to learn.6. The instructor was helpful in guiding the class towards understanding course topics in a way that

helped me clarify my thinking. 7. The instructor helped to keep course participants engaged and participating in productive dialogue.8. The instructor helped keep the course participants on task in a way that helped me to learn.9. The instructor encouraged course participants to explore new concepts in this course.10. Instructor actions reinforced the development of a sense of community among course participants.

Direct Instruction11. The instructor helped to focus discussion on relevant issues in a way that helped me to learn.12. The instructor provided feedback that helped me understand my strengths and weaknesses. 13. The instructor provided feedback in a timely fashion.

 

Social PresenceAffective Expression 

14. Getting to know other course participants gave me a sense of belonging in the course.15. I was able to form distinct impressions of some course participants.16. Online or web-based communication is an excellent medium for social interaction.

Open communication17. I felt comfortable conversing through the online medium. 18. I felt comfortable participating in the course discussions. 19. I felt comfortable interacting with other course participants.

Group cohesion20. I felt comfortable disagreeing with other course participants while still maintaining a sense of trust.21. I felt that my point of view was acknowledged by other course participants. 22. Online discussions help me to develop a sense of collaboration.

 

Cognitive PresenceTriggering Even23. Problems posed increased my interest in course issues.

24. Course activities piqued my curiosity.

25. I felt motivated to explore content related questions. 

Exploration26. I utilized a variety of information sources to explore problems posed in this course.

27. Brainstorming and finding relevant information helped me resolve content related questions. 

28. Discussing course content with my classmates was valuable in helping me appreciate different perspectives.  

Integration29. Combining new information helped me answer questions raised in course activities.

30. Learning activities helped me construct explanations/solutions.

31. Reflection on course content and discussions helped me understand fundamental concepts in this class.

Resolution32. I can describe ways to test and apply the knowledge created in this course. 

33. I have developed solutions to course problems that can be applied in practice.

34. I can apply the knowledge created in this course to my work or other non-class related activities.

TEACHING PRESENCE

1 2 3

1. The instructor clearly communicated important course topics. 0.826 0.088 0.067

2. The instructor clearly communicated important course goals. 0.877 -0.021 0.046

3. The instructor provided clear instructions on how to participate in course learning activities. 0.592 0.246 -0.035

4. The instructor clearly communicated important due dates/time frames for learning activities. 0.611 0.078 0.040

5. The instructor was helpful in identifying areas of agreement and disagreement on course topics that helped me to learn. 0.579 0.162 -0.138

6. The instructor was helpful in guiding the class towards understanding course topics in a way that helped me clarify my thinking. 0.575 0.091 -0.281

7. The instructor helped to keep course participants engaged and participating in productive dialogue. 0.633 0.149 -0.160

8. The instructor helped keep the course participants on task in a way that helped me to learn. 0.579 0.042 -0.285

9. The instructor encouraged course participants to explore new concepts in this course. 0.523 0.099 -0.233

10. Instructor actions reinforced the development of a sense of community among course participants. 0.569 0.174 -0.176

11. The instructor helped to focus discussion on relevant issues in a way that helped me to learn. 0.425 0.146 -0.374

12. The instructor provided feedback that helped me understand my strengths and weaknesses relative to the course’s goals and objectives. 0.649 -0.123 -0.201

13. The instructor provided feedback in a timely fashion. 0.513 -0.025 -0.103

SOCIAL PRESENCE

1 2 3

14. Getting to know other course participants gave me a sense of belonging in the course. 0.050 0.619 -0.233

15. I was able to form distinct impressions of some course participants. 0.172 0.473 0.013

16. Online or web-based communication is an excellent medium for social interaction. -0.181 0.674 -0.226

17. I felt comfortable conversing through the online medium. -0.039 0.814 0.015

18. I felt comfortable participating in the course discussions. 0.109 0.788 0.005

19. I felt comfortable interacting with other course participants. 0.286 0.701 0.038

20. I felt comfortable disagreeing with other course participants while still maintaining a sense of trust. 0.103 0.620 -0.034

21. I felt that my point of view was acknowledged by other course participants. 0.319 0.556 0.025

22. Online discussions help me to develop a sense of collaboration. 0.047 0.561 -0.340

COGNITIVE PRESENCE

1 2 3

23. Problems posed increased my interest in course issues. -0.099 0.172 -0.785

24. Course activities piqued my curiosity. 0.064 0.070 -0.712

25. I felt motivated to explore content related questions. 0.082 -0.031 -0.770

26. I utilized a variety of information sources to explore problems posed in this course. 0.078 -0.158 -0.759

27. Brainstorming and finding relevant information helped me resolve content related questions. -0.106 0.130 -0.794

28. Online discussions were valuable in helping me appreciate different perspectives. -0.096 0.286 -0.699

29. Combining new information helped me answer questions raised in course activities. 0.101 0.043 -0.716

30. Learning activities helped me construct explanations/solutions. 0.128 0.030 -0.732

31. Reflection on course content and discussions helped me understand fundamental concepts in this class. 0.008 0.237 -0.640

32. I can describe ways to test and apply the knowledge created in this course. 0.239 -0.097 -0.619

33. I have developed solutions to course problems that can be applied in practice. 0.147 0.026 -0.653

34. I can apply the knowledge created in this course to my work or other non-class related activities. 0.171 -0.041 -0.687

The CoI Survey at APUS

Used as the end of course survey since January 2009

45% return rate To date n = approximately 130,000 Data analyzed using comparative

descriptive statistics, regression analysis and factor analysis at the University, School, Program, Course and Instructor levels

Recognized as Sloan-C’s effective practice of the year 2009

Recommendations…Part II5. As part of a process of Continuous Quality

Improvement (CQI), work to strengthen data to improve decision-making.

6. When expanding existing systems for assessment and evaluation of student achievement, identify new measures that may be collected to more fully explain student retention. For example, the Community of Inquiry Framework survey has been used by other online institutions to help inform student retention, and items from this survey have been found to be significant predictors (both statistically and pragmatically) of student retention.

TeachingPresence

q12.38

q11.28

q10.24

q9.27

q8.18

q7.23

q6.19

q5.24

q4.53

q3.43

q2.40

q1.37

.89

1

.85

1

.87

1

.85

1

.90

1

.881

.901

.87

1.69

1

1

.78

1

.80

1

q13.46

.74

1

CognitivePresence

q34 .44

q33 .39

q32 .40

q31 .27

q30 .27

q29 .29

q28 .49

q27 .44

q26 .47

q25 .34

q24 ..36

q23 .48

.75

1

.78

1

.781

.851

.86

1.84

1

.721

.75

1

.72

1

.81

1

.80

1

.72

1

SocialPresence

q22

.40

q21

.38

q20

.41

q19

.18

q18

.29

q17

.27

q16

.62

q15

.67

q14

.65

.77

1

.78

1

.77

1

.91

1

.86

1

.85

1

.62

1

.58

1

.59

1

.52(.49)**

Gender Age AcademicLevel

.06(.0

4)*

.02(.08)**

.00(.01)

.06*

.00

.22**

.75

.52(

.52)

**

.49(

.47)

**

Implementation

Quarterly audit form revised to mirror the CoI

Annual review form revised to mirror the CoI with Focus on: Course Organization Content Knowledge/Learning for

Cognitive Presence Direct Instruction and Feedback Social Interaction and Discussion Work Agreement and Achievements Professional Development

Technology Integration

Difference between means in split testing

Difference in variance accounted for Discrete differences in CHAID

analysis SEM pattern alternation Qualitative instrumentation

derivative of the COI utilized where appropriate

Focus on efficacy and ROI

Testing

Feedback – audio and video implementations have proven very effective

Split testing used to assess differences

Analysis of differences in means Strengthen factor loadings on Social

Presence Impact on eight CoI items

Examples

Online or web-based communication is an excellent medium for social interaction. Text group / mean = 3.90 Audio group / mean = 4.27

The instructor was helpful in identifying areas of agreement and disagreement on course topics that helped me to learn. Text group / mean = 3.90 Audio group / mean = 4.27

Instructional Design 1

CoI analysis informs optimal design strategy

Bifurcation of Teaching Presence is an indicator of need for review of “instructor voice”

Social Presence bifurcation is an indicator of the need for more collaboration

Strength of factor loadings can indicate areas where content review is needed

Instructional Design 2

Socio-epistemological orientation – objectivist vs. constructivist

Two factor loading pattern indicative of an objectivist orientation

Constructivist paradigm important for those most impacted – 38-47 years of age

Thank You!

Phil Ice, Ed.D.Director of Course Design, Research andDevelopmentAmerican Public University Systempice@apus.edu

Vice President Sage Road Analyticsphil@sageroadanalytics.com