Scientifically Informed Web-Based InstructionFinancial and Intellectual Support:The William and Flora Hewlett FoundationCarnegie Mellon University through the Office of Technology for Education and the Eberly Center for Teaching Excellence

Joel M. Smith, Vice Provost , CIOCandace Thille, Director of OLI

Improvement in post-secondary education will require converting teaching from a solo sport to a community-based research activity

Herb Simon, Last Lecture Series, Carnegie Mellon, 1998

What Is a Community Based Research Activity in Teaching?Theory-basedFeedback LoopsDiversity of Perspective, Roles & Contexts

Open Learning Initiative

OLI Project GoalsDevelop exemplars of scientifically based online courses and course materials that enact instruction and support instructors Provide open access to these courses and materials Develop a community of use that contributes to the evaluation, iterative improvement, and ongoing growth of the courses and materials.

OLI is Currently a Small Scale Community Based Research Activity in Teaching

Theory Based: Course designs based on current theories in the learning sciences Feedback Loops: Courses record student activity for robust feedback mechanismsDiversity of Perspectives, Roles and Contexts: Courses developed and deployed by teams that include faculty content experts, learning scientists, software engineers

Theory Based: Build on Prior/Informal Knowledge

What is a Cognitive Tutor?A computerized learning environment whose design is based on cognitive principles and whose interaction with students is based on that of a (human) tutori.e., making comments when the student errs, answering questions about what to do next, and maintaining a low profile when the student is performing well.

Theory Based: Provide Immediate Feedback in the Problem Solving Context

Theory Based: Promote Coherence

Theory Based: Multiple Representations With Explicit Connections

Theory Based: Promote Authenticity, Flexibility & Applicability

Learning environments with ambiguous problems that require flexible application of procedural knowledge

Feedback Loops in Learning

EvaluationChemistry: Post-test scores by treatment group show significant positive correlation for the OLI treatment. Most significant indicator was time spent in Virtual Lab Activities made all other variables drop out.

Biology: End of the 3rd week showed an advantage for the OLI section. There was a positive and significant association between students time spent working on particular activities and performance on quiz questions testing the corresponding topics even after total time with OLI has been regressed out

EvaluationStatistics 1st Study:

EvaluationIncrease: 7.9% [t(487) = 13.8, p

Evaluation

Measured learning Outcome% correct CAOS% correct CMUPrePostPrePostBox plots provide accurate estimates of % data above & below only for quartiles22.2 22.2 22.2 50.0Correctly estimate and compare SDs for different histograms. 31.5 46.4 66.7 83.3 41.8 46.4 59.3 75.0Correlation does not imply causation 51.9 49.4 48.1 70.8Calculating appropriate conditional probabilities given table of data 49.6 47.4 70.4 70.8

Student SatisfactionEnd of course survey for online section:All students reported at an increase in their interest in statistics.75% Definitely Recommend 25% Probably Recommend 0% Probably not Recommend 0% Definitely not Recommend

Student Quotes

I have found it to be one of the fastest/most efficient ways to learn new material. Compared to reading the text, the interactivity makes it harder to skim/gloss over the material so I retain more at the end of the session.

Student QuotesI really like the way you can learn individually and at your own pace. If I understand something, I can move through it quickly and take more time on challenging things.

"This is so much better than reading a textbook or listening to a lecture! My mind didnt wander, and I was not bored while doing the lessons. I actually learned something."

Feedback Loop Current ResearchInstructors can use such data to adjust their teaching to students needs.

The VisionInstructor assigns students to work through online instructionSystem collects data as students work System automatically analyzes and organizes the data to present instructor with the students current learning stateInstructor reviews this data summary and adapts instruction accordingly

The Anticipated BenefitsInstructors get a window onto students progressThey can adapt their teaching accordinglyStudents get better feedback to monitor and adjust their learningStrengthens the student-instructor connection

Core OLI CommunityFaculty Content ExpertsLearning ScientistsHuman Computer InteractionSoftware EngineersEvaluation/Assessment SpecialistsLearnersA community of scholars from diverse disciplines who are committed to improving quality and access to instruction. The collaborative nature of the OLI course design process inspired participating faculty to rethink their approach to classroom teaching.

Extended Community: Ecology of Use and ReuseOLI Workshops in USA, Qatar, Mexico OLI Tools and support for adapting, localizing contentFormal and informal evaluationCarnegie Foundation KEEP Toolkit - document scholarship of teaching

Feedback from Evaluation The success of the implementation of the OLI course materials, in terms of learning gains and a renewed student enthusiasm for statistics, led the Department of Psychology at Universidad de los Andes to completely change the instructional sequence in research methodology .

Extended Community: Ecology of Use and Reuse

OLI Appliance in locations with poor internet accessCollaborative Development with institutions in US, Chile, India, QatarDevelopers workshop in US, Taiwan

The Challenge of ScaleTheory Based : Learning Theories are like toothbrushes: Everyone has one and no one wants to use anyone else's.Feedback Loops: The nature of valuable feedback is contextualEcology of Use and Reuse: Mix and mash-up of content in isolation.

Improvement in post-secondary education will require converting teaching from a solo sport to a community-based research activity

Herbert Simon

www.cmu.edu/oli

cthille@cmu.edujoelms@cmu.edu

This work is protected under the Creative Commons Attribution-Noncommercial-Share Alike 2.5 License.

You are free:

* to Share -- to copy, distribute, display, and perform the work * to Remix -- to make derivative works

Under the following conditions:

* Attribution. You must attribute the work in the manner specified by the author or licensor. * Noncommercial. You may not use this work for commercial purposes. * Share Alike. If you alter, transform, or build upon this work, you may distribute the resulting work only under a license identical to this one. * * For any reuse or distribution, you must make clear to others the license terms of this work. * Any of these conditions can be waived if you get permission from the copyright holder. Scientifically based online education requires that the design of the online courses are based on principles from current cognitive and learning theory. Both during the design process and during use, the courses are evaluated through studies of actual student use. The results of the studies are used to inform both the next iteration of the course and the underlying learning theory.

The Open Learning Initiative (OLI) is a project devoted to developing scientifically based, openly available online courses and course materials. OLI started in the fall of 2002, funded by a grant from The William and Flora Hewlett Foundation. The project has the potential to have a positive impact on higher education by increasing access, enhancing quality and providing new exemplars for online courses and course materials.

At the bottom of the slide along with the logo for OLI, you will see the LearnLab Logo for the Pittsburgh Sciences of Learning Center (PSLC). OLI and PSLC share the vision of improving education by transferring knowledge developed in learning science research into teaching and learning practices. The PSLC is a large learning science research center co-led by Carnegie Mellon and the University of Pittsburgh and funded by the National Science Foundation.

The phrase Online courses is used to refer to many different types of electronically supported instruction. While the Open Learning Initiative provides a wide variety of online materials to support instructors, the most challenging goal of the project is to develop online courses that are the complete enactment of instruction. OLI online courses are designed to support an individual learner who does not have the benefit of an instructor to learn a subject at the introductory college level.

A fundamental goal of the project is to provide open access to these courses and course materials to learners world wide. The goals of providing greater access to education and improving the quality of education are, for us, inextricably tied together. In addition to our mission to improve access and quality, we also have a practical and research driven motivation for making these courses openly and freely available. As you will see in this presentation, our use-driven design process depends on the courses being used by a large number of students with varied background knowledge, relevant skills and future goals.

The presentation will go into greater detail explaining each of these components. We will start with several examples of how current findings in learning sciences are used to inform the OLI course designs.Learning Research tells us that building on a students prior or informal knowledge promotes deep learning.

In the economics course we use the affordances of the web to start with students' informal experiential understanding and bridge to more powerful formal forms of understanding.

Students begin each economics unit by participating in a carefully structured online experiment with other online students. In each online experiment the student is an active participant attempting to make deals with other traders in a market. Each student is randomly assigned a role and cost price in each experiment. To make a profit in the experiment, the students must either sell higher than their cost price or buy lower then their cost price. No one knows each others role or cost price. When we use this material at CMU, part of the students grade is dependent on how much profit each student makes in each of the experiments. The students experience first hand the issues that all economic agents confront.

Once the experiment is complete, the data generated by the students participation in the experiment is transferred to a custom workbook. The students can now see the details of the experiment, e.g. the number of suppliers and the number of demanders and everyones cost price. They use this data to complete the tables in the workbook and see how well the economic theories predict their behavior. This approach of bridging from experiential knowledge to formal theory allows students to learn very sophisticated concepts in economics such as the theory of Asymmetric Information and Adverse Selection and how to apply these ideas to real-world markets and understand various policy options.

Mini tutors are used throughout OLI courses in conjunction with other learning activities. In our economics course, for example, we use the mini tutors to support students in learning basic skills of drawing and interpreting supply and demand graphs.

The mini tutors used throughout OLI courses such as the ones we just saw in the economics course, are built on the 20 years of work that has been done at Carnegie Mellon on cognitive tutors. The mini-tutors in OLI courses are not full cognitive tutors in that they do not have full production rule sets or student models but their behavior is similar to a cognitive tutor for the given problem they are intended to tutor.

Many learning studies have shown that students learning improves and their understanding deepens when they are given timely and targeted feedback on their work (Butler & Winne, 1995; Corbett & Anderson, 2001; NRC, 2000, 2001). By feedback we refer to corrections, suggestions, cues, and explanations that are tailored to the individuals current performance and that encourage revision and refinement.

This tutor is in a section of the Mechanical Engineering Statics Course on Summing Force Vectors, and helps students learn how to determine the sum of concurrent forces by resolving them into components. It is intended to be an opportunity for students to do a "self-check" to make sure they understand the concept.

The student is presented with a graphical representation of the problem and asked for the answer. If the student is unsure of the procedure for solving the problem, the first hint provides a link which, when clicked, expands the tutor into the various steps needed to solve the problem. The tutor provides scaffolding to support the student to learn the steps of the procedure when needed. The hints and feedback given by the tutor change depending on which part of the exercise the student is attempting. The tutor recognizes when a student has used the scaffolding and hints and when the student gives the correct answer after having used the scaffolding and hints, the tutor suggests the student to try another problem. The graph, the problem statement, hints, feedback and answers are dynamically-generated. The student can work through the tutor multiple times, receiving a different problem each time, until the student is confident that he or she understands the concept and has developed fluency with the proceedure. This provides the student with virtually unlimited opportunities for supported practice.Promoting coherence means teaching students how the discrete skills they are learning fit together in a meaningful big picture. Although the conceptual structure of knowledge in a domain is clear to experts, it is not to novices. The array of new ideas and unfamiliar terminology in introductory courses tends to overwhelm students into memorizing a set of isolated facts without understanding the underlying common principles (Chi, 2005; diSessa, 1993).

Studies have shown that students learn new material better when it is presented in a structured manner that provides the conceptual framework or big picture. For students to benefit from a conceptual framework that organizes the material they are learning, instruction needs to make that framework salient, and students need to practice making connections between related ideas (Eylon & Reif, 1984).

StatTutor is a computerized learning tool that presents students with data-analysis problems and guides students through solutions using cognitive scaffolding and a Cognitive Tutor. StatTutor highlights common structure across problems, covers all the elements of the Big Picture, provides scaffolding in choosing appropriate analysis and offers hints and feedback as the student works. Studies have shown the benefits of "multiple representations," that is, of providing learners with more than one representation of a complex concept and providing explicit connections among the representations.

This simulation uses text, animation, graphics and interactivity to teach the concepts of equilibrium and weak electrolytes. The animation illustrates the dynamic character of a process and the involvement of populations, the graphics translate the visual animation into concrete determinants and the interactivity allows the student to observe the effect of altering variables on both the visual and graphic representations. A challenge in chemistry education is that students can be quite proficient at solving the mathematical problems in chemistry textbooks without being able to flexibly apply those tools to novel chemistry phenomenon in which their application would be useful. Our goal in the OLI chemistry course is to bridge from mathematical procedures to chemical phenomena through use of the Virtual laboratory and to bridge from chemical phenomena to real world through scenario based learning.

Prior to designing our course, we observed that students typically solve traditional chemistry text book problems via a shallow ends-means analysis, by matching the information given in the problem statement with the equations they can pull from the chapter text.

To address this and other issues in chemistry education, rather than the traditional approach of teaching the abstract mathematical skills of chemistry out of context, the OLI chemistry course situates the learning in an authentic investigation that addresses real world applications and asks students to approach chemistry problems as a chemist would approach them. The OLI unit on stoichiometry is situated in a real world problem of arsenic contamination of the water supply in Bangladesh.

We address the challenge of connecting the mathematical procedure to use in chemistry by replacing traditional textbook problems with problems to be constructed and solved in the virtual chemistry lab. We use the virtual chemistry lab to create learning environments with ill-structured, ambiguous problems that require flexible application of procedural knowledge. The virtual chemistry lab provides opportunities for students to interact with the environment by exploring and manipulating objects, wrestling with questions and designing experiments. This approach promotes deeper learning and lets different students solve problems in different ways.

At each step of exploring a solution to the arsenic contamination problem, the student is introduced to and practices one of the target stoichiometric concepts or skills. In the very first step, determining the level of arsenic contamination in a sample of well water, the student uses the Chemistry Virtual Lab to analyze the sample and compare the level of arsenic to the acceptable levels set by the World Health Organization. In order to evaluate the safety of the water, the student must either understand the concept of the mole and apply dimensional analysis, composition stoichiometry and solution stoichiometry. If the student does already understand these concepts or cant demonstrate mastery of these procedures in the context of solving the problem, the student is directed into an instructional sequence that includes demonstrations, worked examples and minitutors.

In the last several slides, we have shown how the feedback loops are used to improve student performance.

The other feedback loops shown on this slide are part of the research activity of the project which is enacted from the initial design of the courses through implementation.

During the design process and during use, the courses are continually evaluated through studies of student use and learning. All student learning activities in OLI courses and labs are, with the students permission, digitally recorded in considerable detail to monitor student activity and capture the data required by such studies. The data gathered is analyzed to address questions such as how students interacted with the OLI course materials, what were their patterns of use, and to what degree were their use patterns correlated with various learning outcome measures. The results of this built-in research inform the next iteration of the course. The research results may also contribute back to the underlying design principle or learning theory. The original goal of OLI was to provide access to high quality post secondary courses similar to those taught at Carnegie Mellon to those who did not have the privilege of attending an institution like ours. Our evaluation efforts to date have demonstrated the degree to which we have achieved that goal by developing online courses that enact instruction as effectively as existing instructor lead courses. In the next few slides, we will show some results from those studies.

We believe such studies are a necessary condition for the transformation in access to learning through online instruction we seek. In an article in the New York Times on July 23rd, 2006, Henry Jenkins, an M.I.T. professor who studies games and learning, says of that domain that a lack of studies that show that gaming accomplishes educational goals will create a condition where oxygens going to be sucked out of this. We believe this is true of all open learning efforts. Sustaining an open learning movement requires demonstration of effectiveness. More studies of impact of OERs are needed. Much of the data remains anecdotal. The careful studies that have been done have hopeful results, but in the end, data about the value of the efforts is part of maintaining the current high level of enthusiasm.

More about the design of the Chemistry Study:The participants were incoming freshman whose course of study places them in the introductory general chemistry course during their first two years at CMU. Most of these students were enrolled in Introductory Modern Chemistry I (CHEM105) during the fall 05 or spring 06 terms. Participants in our study were randomly assigned to one of two groups. The experimental group studied stoichiometry using the our online stoichiometry course. The other group studied stoichiometry using the same explanatory and practice problems in text form. Members of both groups took three tests on the study materials. One test was administered online immediately following a participants completion of the study materials. A second parallel test was taken in a classroom setting during Freshman Orientation Week. A third test was administered 6 months later.

More about the design of the Biology Study:The participants were mostly incoming freshman who were taking introductory Modern Biology in their first semester at CMU.

Participants in our study were assigned to one of two groups by the course section in which they had enrolled all students in modern biology participated in the course configuration but we are analyzing data only for those students who signed consent forms. Because of the study design both groups served as control and treatment groups at different points during the study. In weeks 2&3 Group A was the treatment group and group B was the control. In weeks 4&5 group B was the treatment group and Group A was the control. During weeks 2&3 only group A had access to the online course in weeks 4&5 only group B had access to the online course. Each group had access only to those parts of the course taught during their treatment condition even for studying for the midterm. Our original plan was to have the treatment group divided into 3 discussion groups of 50 students each but that proved to be logistically too difficult so the treatment group met once per week with the instructor but in the 150 person large group in retrospect this turned out to be a good thing because it automatically controlled for class size as a variable in measuring class participation. An initial observation is that students in the treatment group asked 5 times as many questions and the participation was more evenly distributed across participants than the students in the lecture course.

More about the Statistics Course Study DesignThe participants were freshmen and sophmores whose course of study places them in the introductory statistics course during their first two years at CMU. The traditional course is 3 lecture hours and 1 lab hour per week. Study participants were recruited on the first day of lecture. We announced that we were looking for volunteers to take an online version of the course. Students in the online version would not attend lecture or lab or use the statistics textbook but rather would work in the online course and meet with a statistics faculty member once a week to ask questions and give feedback. The statistics faculty member did not prepare or deliver additional instruction to the students during the face to face meetings. Students in the study took 3 paper midterms and a final the same as the traditional lecture students.

We instructed students who would be interested in participating in the study to complete an online survey within the next 24 hours. We had 40 volunteers complete the survey within the 24 hour period and we selected 20 volunteers for the study. We grouped the volunteer surveys by gender and race and randomly selected from each group in accordance with the demographics of the course.

Our second study was conducted the following semester and used the same study design as the first study but we added a nationally recognized test of statistical literacy. We used the CAOS Test - Comprehensive Assessment of Outcomes in a first Statistics course (Delmas, Garfield, Chance, Ooms)

The CAOS test includes 40 multiple choice items that measure statistical literacy & conceptual understanding.The Focus is on reasoning about variability. 18 expert raters agreed with the statement: CAOS measures outcomes for which I would be disappointed if they were not achieved by students who succeed in my statistics courses.

Additionally, we measured outcomes (as defined in DelMas et al. AERA, 2006) of items on which less than 50% of students in the national sample were correct on posttest. As you can see in these results, the students taking the OLI course did significantly better than the national group on those particularly challenging items.

This feedback loop is always cycling in traditional face to face classes and it works well there when you have an expert teacher and a not unreasonably heterogenous class: the instructor sees students faces look confused or sees poor results on a test and can adjust instruction to accordingly.

Although originally designed to support individual learners, OLI courses are increasingly used by instructors inside and outside of Carnegie Mellon as a complement to their instructor led course to address the challenges they confront as a result of the increasing variability in their students background knowledge, relevant skills and future goals. The richness of the data we are collecting about student use and learning provides an unprecedented opportunity for keeping instructors in tune with the many aspects of students learning.

Creating an effective feedback loop to instructors using the OLI courses is our current area of research. Reviewing a classs answers to multiple-choice questions is of limited value. Reviewing text responses from a large lecture class is more informative, but also so labor intensive that few faculty have the time to do it. How do we take adaptive instruction to the next level by providing significant easy-to-use, and easy-to-interpret feedback to faculty? A vibrant interactive online environment should provide not only students but also faculty with many opportunities to interact with feedback-rich instructional materials. Our ultimate goal is to create a Digital Dashboard designed for Learning (DDL) as a tool both for students and faculty. A simple definition of a DDL is a tool that provides visibility into key indicators of student learning through simple visual graphics such as gauges, charts and tables within a web browser.

A vision of how the DDL would be used to support teaching.In OLI we have created a community of instructors, learning scientists, instructional support specialists and others who engage in an ongoing movement to make learning as effective as it can be by leveraging scientific work on how people learn and by engaging in use-driven design processes. OLI has been an example of creating such a community on a small scale. OLI faculty remain engaged in the project because they have the opportunity to redefine both what to teach and how to teach their domain in light of the affordances of the technology and the information from the learning sciences. In regular meetings and outside those meetings, the OLI faculty and staff feed off each others ideas, often challenging one another about the research basis and assessment designs for their educational designs. In short, the effort has produced a community of scholars from diverse disciplines who are also committed to scientifically-based, online teaching as a path to improving quality and access to instruction. From the beginning of the project we have conducted workshops for faculty who wish to use OLI courses to support their teaching. For many faculty at smaller institutions the OLI faculty workshop is a rare chance to participate in a community of research and practice on teaching and learning. Over the 3 days of the workshop, the faculty from other institutions work with Carnegie Mellon faculty and cognitive and learning scientists to explore the course material and discuss with each other how they might use it in their teaching and how we might collaboratively extend the courses. The faculty also build relationships that extend beyond the workshop

We have developed a number of tools to support faculty with no technical expertise to adapt the courses to better support their teaching. We invite all faculty to participate in our ongoing formal and informal evaluation studies.

The Knowledge Media Lab (KML) at the Carnegie Foundation has been investigating how to support educators and students in documenting, sharing, and building knowledge of effective practices and successful educational resources to collectively advance teaching and learning for some time. OLI is using the KEEP toolkit developed by KML to document and communicate both the original design rational and the many variations on contextual use of the OLI courses. Course development teams complete the KEEP OLI author template to document and communicate the instructional goals and learning theory that guided course development. Faculty at a variety of institutions who are using OLI courses complete the KEEP OLI user template to document and communicate a description of the context in which the online courses are delivered and the impact of using the OLI course on teaching and learning. The combination of the KEEP OLI author document and the collection of the KEEP OLI user documents for each course provide potential users with an understanding of the logic and goals behind each of the courses and with rich information about the institutional, sociocultural, and curricular contexts of teaching and learning. The processes of a community based research activity are intensely reflexive. Our challenge now is how to extend this enthusiasm and process to an even larger community without breaking the feedback loops that are critical to its success. We have had successes with our partners at other institutions both in the U.S. and abroad but the processes are time and labor intensive and do not scale as easily as the more common processes of individual learners or faculty producing or selecting and re-using, re-mixing or mashing-up resources on their own.