Summary of efficacy studies May 2015 - OpenCon Community Webcasts
-
Upload
right-to-research -
Category
Government & Nonprofit
-
view
278 -
download
0
Transcript of Summary of efficacy studies May 2015 - OpenCon Community Webcasts
A REVIEW OF RESEARCH ON THE PERCEPTIONS AND
EFFICACY OF OER (AND A CALL FOR MORE!)
John Hilton III http://johnhiltoniii.org
Open Education Grouphttp://openedgroup.org
PROBLEM
A recent nationally representative survey of 2,144 faculty members in the United States found that “most faculty remain unaware of OER.”
Source: Babson 2014 Survey, “Opening the Curriculum.”
POSSIBLE SOLUTIONS
Increasing efforts to “market” OER. Increasing number of outstanding OER material.
Increasing the number of academic, peer-reviewed studies regarding the efficacy and teacher and student perceptions of OER materials.
INCREASING THE NUMBER OF ACADEMIC, PEER-REVIEWED STUDIES REGARDING THE EFFICACY AND TEACHER AND STUDENT PERCEPTIONS OF OER MATERIALS.
The Babson 2014 survey found that college professors rate “proven efficacy” and “trusted quality” as the number 1 and number 2 most important criteria for selecting teaching resources.
PUBLISHED EFFICACY AND PERCEPTION STUDIES
1. Article focused on efficacy or perception in actual practice (not simply theory).
2. The resource(s) examined in the study needed to be OER that were the primary learning resource(s) used in the class.
3. In order to be selected for inclusion in this study, the research needed to have been published by a peer-reviewed journal, or be an institutional research report. Blog posts and conference proceedings were excluded from this data set.
THE BIG PICTURE
Across Eight Different Academic Studies 3594 Students and Professors Were Surveyed…
THE BIG PICTURE
*Photo of students courtesy of http://acreelman.blogspot.com/2014_09_01_archive.html
REFERENCES - ONLINE SUMMARY:HTTP://OPENEDGROUP.ORG/REVIEW
1. Allen, G., Guzman-Alvarez, A., Molinaro, M., Larsen, D. (2015). Assessing the Impact and Efficacy of the Open-Access ChemWiki Textbook Project. Educause Learning Initiative Brief, January 2015. See also this newsletter.
2. Allen, I., Seaman, J. (2014). Opening the Curriculum: Open Educational Resources in U.S. Higher Education, 2014.
3. Bliss, T., Robinson, T. J., Hilton, J., & Wiley, D. (2013). An OER COUP: College teacher and student perceptions of Open Educational Resources. Journal of Interactive Media in Education, 1–25.
4. Bliss, T., Hilton, J., Wiley, D., Thanos, K. (2013). The cost and quality of open textbooks: Perceptions of community college faculty and students. First Monday, 18:1.
5. Bowen, W. G., Chingos, M. M., Lack, K. A., & Nygren, T. I. (2012). Interactive Learning Online at Public Universities: Evidence from Randomized Trials. Ithaka S+R.
6. Bowen, W. G., Chingos, M. M., Lack, K. A., & Nygren, T. I. (2014). Interactive Learning Online at Public Universities: Evidence from a Six‐Campus Randomized Trial. Journal of Policy Analysis and Management, 33(1), 94-111.
7. Feldstein, A., Martin, M., Hudson, A., Warren, K., Hilton, J., & Wiley, D. (2012). Open textbooks and increased student access and outcomes. European Journal of Open, Distance and E-Learning..
8. Gil, P., Candelas, F., Jara, C., Garcia, G., Torres, F (2013). Web-based OERs in Computer Networks. International Journal of Engineering Education, 29(6), 1537-1550. (OA preprint)
9. Hilton, J., Gaudet, D., Clark, P., Robinson, J., & Wiley, D. (2013). The adoption of open educational resources by one community college math department. The International Review of Research in Open and Distance Learning, 14(4), 37–50.
10. Hilton, J., & Laman, C. (2012). One college’s use of an open psychology textbook. Open Learning: The Journal of Open and Distance Learning, 27(3), 201–217. (Open Repository Preprint).
11. Lindshield, B., & Adhikari, K. (2013). Online and campus college students like using an open educational resource instead of a traditional textbook. Journal of Online Learning & Teaching,9(1), 1–7.
12. Lovett, M., Meyer, O., & Thille, C. (2008). The open learning initiative: Measuring the effectiveness of the OLI statistics course in accelerating student learning. Journal of Interactive Media in Education, 2008 (1).
13. Pawlyshyn, Braddlee, Casper and Miller (2013). Adopting OER: A Case Study of Cross-Institutional Collaboration and Innovation. Educause Review.
14. Petrides, L., Jimes, C., Middleton‐Detzner, C., Walling, J., & Weiss, S. (2011). Open textbook adoption and use: Implications for teachers and learners. Open learning, 26(1), 39-49.
15. Robinson T. J., Fischer, L., Wiley, D. A., & Hilton, J. (2014). The impact of open textbooks on secondary science learning outcomes. Educational Researcher, 43(7): 341-351.
16. Wiley, D., Hilton, J. Ellington, S., and Hall, T. (2012). “A preliminary examination of the cost savings and learning impacts of using open textbooks in middle and high school science classes.” International Review of Research in Open and Distance Learning. 13 (3), pp. 261-276.
LET’S LOOK AT FOUR OF THE SIXTEEN PUBLISHED PERCEPTIONS/EFFICACY STUDIES
Bliss et al., – Perceptions across eight colleges
Hilton et al., – Scottsdale Community College Math
Allen et al., – UC Davis ChemistryRobinson et al., – Secondary science
KALEIDOSCOPE OPEN COURSE INITIATIVE
The study context is an open education initiative called the Kaleidoscope Open Course Initiative (KOCI) (http://www.project-kaleidoscope.org/). This initiative was originally comprised of eight community colleges serving predominantly at-risk students (this number has now expanded). These colleges work together to create courses that replace traditional, expensive textbooks with OER. During the 2011-2012 academic year, KOCI impacted over 4,000 students across 80 teachers in a controlled pilot, with another 5,000 students using KOCI course designs and materials outside the controlled pilot.
KOCI – TEACHER PERCEPTIONS
58 teachers from 8 KOCI institutions completed some items on the questionnaire.
Student preparedness. Most teachers (63%) believed that their students were equally prepared in the course with PK texts compared to students in the same course before implementation of KOCI texts (Figure 3). At the same time, more than a quarter of teachers (26%) felt that students were more prepared than students in the same course in the past. Only 11 percent perceived students as being less prepared.
KOCI TEACHER PERCEPTIONS
58 TOTAL TEACHERS
More Prepared Equally Prepared Less Prepared0%
10%
20%
30%
40%
50%
60%
70%
26%
63%
11%
Teachers’ Perceptions of student preparedness after implementation of KOCI Texts
KOCI– TEACHER PERCEPTIONS
Teacher perceptions of quality. Twenty out of 57 (35%) teachers indicated that they thought the OER textbook was better than texts they had used previously to teach the course. Only 6 (11%) felt the OER texts were worse than traditional texts. The remaining 31(54%) teachers indicated that their OER texts were of the nearly the same quality as other texts they had used in the past.
KOCI– TEACHER PERCEPTIONS
57 TOTAL TEACHERS
Better Quality Equal Quality Worse Quality0%
10%
20%
30%
40%
50%
60%
35%
54%
11%
Teachers’ perceptions of OER text quality compared to traditional texts
KOCI – STUDENT PERCEPTIONS
490 students from all eight KOCI institutions completed the questionnaire. 60% were female.
39% of students indicated that they thought the OER textbook was better than texts they had used in other courses. 6% felt the OER texts were worse and 55% indicated that their OER texts were of the nearly the same quality as other texts they had used in the past.
KOCI – STUDENT PERCEPTIONS
490 TOTAL STUDENTS
Better Quality Equal Quality Worse Quality0%
10%
20%
30%
40%
50%
60%
39%
55%
6%
Student Perceptions of KOCI Texts compared to traditional texts
KOCI– STUDENT PERCEPTIONS
160 students provided a description of what they thought made the OER texts better. Their responses clustered in six major categories: technical advantages (8%), learning aides (9%), customization (10%), cost (20%), access (26%), and quality/presentation (27%).
Conversely, 20 students provided a description of what they thought made the OER texts worse than other texts they had used in the past. Their responses clustered in 2 major categories; six students described technology issues and 15 students took issue with text quality.
KOCI– STUDENT PERCEPTIONS
SAMPLE SIZES - 160 STUDENTS – BETTER 20 STUDENTS - WORSE
0%
5%
10%
15%
20%
25%
30%
8% 9% 10%
20%
26%
17%
Student descriptions of OER Benefits
Technology Issues
Text Quality0
2
4
6
8
10
12
14
16
6
15
Student descriptions of OER Downsides
LET’S LOOK AT FOUR OF THE SIXTEEN PUBLISHED PERCEPTIONS/EFFICACY STUDIES
Bliss et al., – Perceptions across eight colleges
Hilton et al., – Scottsdale Community College Math
Allen et al., – UC Davis ChemistryRobinson et al., – Secondary science
Text Book and Interactive Learning
Environment Package provided by Wiley and
Pearson Education
Intermediate Algebra1352 Students
Publisher Learning Resources
Introductory Algebra
1250 Students
$110.26 per Student$182,086.76 per Semester
College Algebra
701 Students
SCC’S TRANSITION TO OER
Text Book and Interactive Learning Environment provided through OER
Introductory Algebra
1250 Students
College Algebra
700 Students
Traditional Copyrighted
Material
Transition
Open Educational Resources
Maximum of$15 per Student
Text Book and Interactive Learning
Environment provided through OER
86% decrease in costs
RESULTSPercentage of Students Earning a C Grade or Better
Fall 2010
Fall 2011
Fall 2012
AT 12x 60% (n=748)
63% (n=721)
62% (n=764)
MAT 15x 65% (n=448)
64% (n=388)
65% (n=461)
MAT 182 56% (n=106)
61% (n=109)
58% (n=95)
MAT 187 53% (n=72)
48% (n=82)
55% (n=80)
RESULTS
Completion Rates Fall 2010 Fall 2011 Fall 2012
MAT 12x 79% (n=748)
76% (n=721)
75% (n=764)
MAT 15x 73% (n=448)
72% (n=388)
76% (n=461)
MAT 182 58% (n=106)
72% (n=109)
72% (n=95)
MAT 187 68% (n=72) 62% (n=82)
64% (n=80)
No Re-sponse
Strongly agree
Slightly agree
Neutral Slightly disagree
Strongly disagree
0
50
100
150
200
250
300
350
400
450
500
550
12
491
263
105
2811
Overall, the materials ade-quately supported the work I
did in class
No Re-sponse
Strongly agree
Slightly agree
Neutral Slightly disagree
Strongly disagree
050
100150200250300350400450500550
9
426
285
139
3417
Overall, the materials ade-quately supported the work I
did outside of class
No Re-sponse
Strongly agree
Slightly agree
Neutral Slightly disagree
Strongly disagree
050
100150200250300350400450500550
26
483
210
133
33 25
I would recommend the use of these materials to my class-
mates.
“WHAT ADDITIONAL COMMENTS DO YOU HAVE REGARDING THE QUALITY OF THE OPEN MATERIALS USED IN YOUR CLASS?” (210/255 (82%) POSITIVE)
“They were good. Definitely worth not having a massed produced book for.”
“I never had an open materials class before. It made work less stressful and learning more enjoyable – didn’t constantly feel frustrated and was able to look through notes when I got stuck.”
“The quality was excellent. It really helped my understanding.”
Positive Comments
Negative Comments
0
50
100
150
200
250
210
45
Types of Comments
“WHAT ADDITIONAL COMMENTS DO YOU HAVE REGARDING THE QUALITY OF THE OPEN MATERIALS USED IN YOUR CLASS?” (210/255 (82%) POSITIVE)
“Buying textbooks is out of date and I think materials should be inexpensive.”
“I love saving money, I am poor.” “I like the open materials, textbooks are so expensive that it makes me not want to buy them.”
FACULTY RESPONSES
Of the fifteen instructors who responded to the question, “Do you feel that the OER materials adequately supported the work that was completed INSIDE the classroom? Why or why not?” thirteen answered “yes,”
Yes No0
2
4
6
8
10
12
14
13
2
Did OER materials adequately support work INSIDE the
Classroom?
FACULTY RESPONSES
Similarly, of the fifteen instructors who responded to the question, “Do you feel that the OER materials adequately supported the work that was completed OUTSIDE the classroom? Why or why not?” thirteen answered “yes.” Sample responses are “Yes. These materials provided the students useful resources” and “Yes. It focused in the lesson well.” The remaining two responses were “Mostly - often students had online questions, which were discussed through MathAS or in class” and “In Math 12x, the online HW contained questions that were not written in the style of the workbook questions. Some students had difficulty in transferring their skills to the online questions.”
FACULTY RESPONSE
Yes No02468
101214
13
2
Did OER materials adequately support work OUTSIDE of the Classroom?
LET’S LOOK AT FOUR OF THE SIXTEEN PUBLISHED PERCEPTIONS/EFFICACY STUDIESBliss et al., – Perceptions across eight colleges
Hilton et al., – Scottsdale Community College Math
Allen et al., – UC Davis ChemistryRobinson et al., – Secondary science
STUDY DESCRIPTION
Researcher at UC-Davis designed an experiment in which an experimental class of 478 students used the OER ChemWiki as its primary textbook, while the control class of 448 utilized a commercial textbook. The two sections were taught the same semester at back-to-back times using the same faculty member and teaching assistants. Identical lectures were used in each section. Students in both sections were given the same midterm and final exams.
STUDY RESULTS
Researchers found no significant differences between the two groups both with overall exam results and item-specific questions. Beginning of the semester pre-tests, combined with final exams showed no significant differences in individual learning gains between the two groups. Student surveys regarding time spent on the class found that students in both groups spent approximately the same amount of time preparing for class.
ChemWiki Group Control Group
Test Scores
Learning Gains
Pre-Class Preparation Time
Test Scores
Learning Gains
Pre-Class Preparation Time
LET’S LOOK AT FOUR OF THE SIXTEEN PUBLISHED PERCEPTIONS/EFFICACY STUDIESBliss et al., – Perceptions across eight colleges
Hilton et al., – Scottsdale Community College Math
Allen et al., – UC Davis ChemistryRobinson et al., – Secondary science
DESIGN
Quasi-experimental ex post facto comparison of group differences in terms of year-end standardized science CRT scores.
POTENTIAL CONFOUNDS
SPEDELPPrevious Science Achievement
General Academic Achievement
SES
GenderAgeCourse (Chemistry, Earth Science, Biology)
Teacher EffectSelection Bias
AND WHAT ABOUT TEACHER EFFECT?
Isn’t it possible that the teacher that are competent enough to develop their own materials are actually more competent as teachers in general?
Textbooks and teachers are completely confounded.
Textbook may be a complete surrogate for teacher effect.
CONTROLLING FOR TEACHER EFFECT
Consider previous achievement of students as an indication of general teacher effectiveness.
Create a standardized value of teacher effectiveness.
Use standardized value as a covariate control.
PROPENSITY SCORE MATCHING
Using available covariates, match individual treatment subjects to subjects in a larger pool of possible controls.
Unmatched Treatment Pool: n = 1612Unmatched Control Pool: n = 2168Matched Treatment: n = 1274Matched Control: n = 1274 Matched Total: n = 2548
MATCHED SAMPLES
Matched Control
Matched Treatment
Matched Total
Number of students
1274 1274 2548 Mean age
15.99 15.98 15.99
Percentage of Female Students
50.08% 49.69% 49.88% Percentage of Male Students
49.92% 50.31% 50.12%
Percentage of ESL Students
0.94% 1.02% 0.98% Percentage of Asian Students
0.31% 0.23% 0.27%
Percentage of Black Students
0.47% 0.31% 0.39% Percentage of White Students
88.30% 87.52% 87.91%
Percentage of Hispanic Students
8.30% 9.50% 8.90% Percentage of Native American Students
0.55% 0.63% 0.59%
Percentage of Pacific Islander Students
0.78% 0.86% 0.82% Percentage of Multi-race Students
1.20% 0.94% 1.10%
Percentage of 9th Graders
39.08% 39.80% 39.44% Percentage of 10th Graders
44.82% 44.98% 44.90%
Percentage of 11th Graders
15.54% 14.68% 15.11% Percentage of 12th Graders
0.55% 0.55% 0.55%
Special Education
6.99% 7.22% 7.10% Mean GPA
3.44 3.40 3.42
Percentage of Students in Biology
38.14% 39.17% 38.66% Percentage of Students in Chemistry
34.30% 32.65% 33.48%
Percentage of Students in Earth Systems 27.55 28.18% 27.86%
PROPENSITY SCORE MATCHED SAMPLES
FINAL RESULTS
Simultaneous Regression of 2012 Scaled Scores Predictor Estimate S.E. t Intercept 57.01 5.12 11.13 *** Treatment 0.65 0.24 2.67 ** 2011 GPA -1.64 0.23 7.25 *** 2012 CRT1 Chemistry -6.59 1.01 -6.53 *** Earth Science 3.94 0.53 7.44 *** 2011 CRT1 Chemistry 3.91 2.51 1.56 Earth Science -2.88 1.02 -2.83 ** Physics 5.1 0.89 5.75 *** 8th Grade Science -5.96 1.17 -5.08 *** 2011 Scaled Score 0.68 0.15 44.42 *** Gender2 1.16 0.24 4.84 *** Age -0.49 0.26 -1.91 English Language Proficiency -3.86 1.18 -3.27 ** Special Education3 -1.73 0.48 -3.62 *** Free and Reduced Lunch4 -0.31 0.26 -1.19 Teacher Effect 2.26 0.19 11.45 ***
* p< .05 **p<.01 ***p<.001 1The dummy coded reference group for both the 2012 and 2011 CRTs is those who took the biology test. 2The dummy coded reference group for Gender is Female. 3The dummy coded reference group for Special Education is Not Special Education. 4The dummy coded reference group for Free-and-Reduced Lunch is Not Free-and-Reduced Lunch.
CONCLUSION
There was a small but statistically significant difference between students using traditional classroom sets of textbooks and students using teacher developed individual textbooks in terms of achievement as measured by standardized year-end science CRT scores.
Even after controlling for multiple confounds, students using teacher-developed open textbooks performed better than control group students.
SYNTHESIZING
While some may be disappointed that OER materials have not been found to significantly increase student learning outcomes, this “non-finding” is nevertheless very important.
Given that (1) students and teachers generally find OER to be as good or better as traditional textbooks, (2) students do not perform worse when utilizing OER, students, parents and taxpayers stand to save literally billions of dollars without any negative impact on learning through the adoption of OER.
A REQUEST
Will you initiate research studies focused on perceptions and efficacy of OER? Scholarly articles in this arena will increase awareness and adoption of OER. If you would like help in designing or implementing such studies, my colleagues at the Open Education Group are happy to assist.
A REVIEW OF RESEARCH ON THE PERCEPTIONS AND
EFFICACY OF OER (AND A CALL FOR MORE!)
John Hilton III http://johnhiltoniii.org
Open Education Grouphttp://openedgroup.org