Lessons Learned
Findings from Ten Formative Assessments
of Educational Initiatives at MIT
(2000-2003)
Lori Breslow, Ph.D.
and
The Staff of the
Teaching and Learning Laboratory
The Context
From 1999 to the present, MIT has undertaken approximately 40 experiments in educational innovation. TLL has assessed:
Subjects
1.00,* 6.001*, 6.002x,* 8.02 (TEAL),* 8.224, HST582J, 18.03, Mission 200X*
Programs
The Undergraduate Exchange with University of Cambridge,* Residence-Based Advising*
Technologies
PIVoT, Cross Media Annotation System (XMAS),* MetaMedia,* PRS
Space
The d’Arbeloff/TEAL classroom (26-152)
*Denotes multi-year or multi-semester assessment
Agenda
• Four lessons about the use of educational technology
• Observations on best practices in design, implementation, and assessment
• Where do we go from here?
Four Lessons about Educational Technology
• Educational technologies have contributed to gains in
learning
• Educational technologies that have met a specific educational need that has been unmet or poorly met by traditional media have been more successful
• Too much new technology, too many technologies, technologies that are poorly integrated into the curriculum have been less successful
• Studying the relationship between educational technologies and the contexts in which they are embedded has yielded important understandings
Technology-Enabled Active Learning Increased Learning Gains
Sou
r ce:
Dor
i, Y
. & B
elch
er, J
. (20
04)
“How
Doe
s T
echn
olog
y-E
nabl
ed
Act
ive
Lea
rni n
g A
ffec
t Und
ergr
adua
te S
tude
nts’
Und
ers t
andi
ng o
f E
l ect
r om
agne
tic
Con
cept
s?”
unde
r re
view
at T
he J
ourn
al o
f the
Lea
rnin
g Sc
ienc
es
Lesson #1
Learning gains <g> = %Correct post-test - % Correct pre-test
100% - %Correct pre-test
0
0.1
0.2
0.3
0.4
0.5
0.6
Low Intermediate High
Le
arn
ing
ga
ins
<g
>.
Lecture Students 2002 TEAL Students 2003
A Problem-Based Module Increased Learning Gains
Source: Greenberg, J. Smith, N. & Newman, J. (2003) “Instructional Module in Fourier Spectral Analysis, Based on Principles of How People Learn,” Journal of Engineering Education
Lesson #1
What Has Worked
• Online lectures to teach students basic concepts• Visualizations to help students see what cannot
be seen• Archives of visual images to help students
strengthen different kinds of literacies• Wireless laptops to learn programming• Remote online laboratories• Technologies that provide instantaneous, more
individualized feedback
Lesson #2
Online Lectures Were More Effective in Intro Computer Science
Lesson #2
Source: Newman, J. (2002) “6.001 Report on the Outcomes of Online Learning,” unpublished assessment report
Visualizations Help Students See What Cannot Be Seen
An animation used in Physics II, a course in electromagnetism” (8.02T). Click to see how field lines move. For more information on the Technology-Enabled Active Learning project, go to http://web.mit.edu/8.02t/www.
MetaMedia Creates Visual Archives that Expand Literacy
MetaMedia homepage. For more information on MetaMedia, go to http://metaphor.mit.edu/.
iLab Makes Laboratory Facilities Available Remotely
The Flagpole Project allows students to take readings remotely. For more information on iLab, go to http://icampus.mit.edu/projects/iLab.shtml.
Two Other Effective Applications of Educational Technology
• Using wireless laptops to learn programming in “Introduction to Computers and Engineering Problem Solving” (1.00)
• Improving feedback– To the students through hint and check buttons
used to answer problems embedded in online lectures (6.001)
– To the faculty through a personal response system that records student answers to conceptual questions asked during class (8.02T)
Lesson #2
What Hasn’t Worked• Technologies designed for interaction (e.g.,
discussion boards) have been less successful at MIT
• When students need to learn the technology before they can learn the material, they have two things to master
• Too much technology can be detrimental
• When technology is poorly integrated into the curriculum, it is either not used at all or not used effectively by the students
Lesson #3
Applications That Needed Some Work
• Functionality of – MetaMedia – XMAS– Simulation in “Biomedical Signal Image and Processing”
• Discussion boards in “Exploring Black Holes”• Unchat in Mission 200X• Technology in 8.02T
– PRS– PowerPoint
• Laptops when used in lectures in 1.00
Lesson #3
Edtech and the Learning Environment
• Edtech has been most successful when there are strong connections between it and– Learning objectives– Pedagogies– Assessment
• The same technology will have different effects in different environments
• Edtech exerts its impact by changing the nature of information in the system
Lesson #4
There Are Strong Connections in TEAL
• Objective: to increase students’ conceptual and analytical understanding about the nature of electromagnetic fields
• Technology’s contribution– Visualizations– Desktop experiments– PRS
Lesson #4
But the Same Technology May Have Different Effects . . .
• In different learning environments– PIVoT as used at MIT, RPI, and Wellesley
• In different situations– Laptops in 1.00 during active learning and during lecture
• For different students– For example, less prepared MIT students were significantly
more likely than better prepared MIT students to think PIVoT helped their conceptual understanding
– But at RPI, students with better high school preparation derived greater benefit than those with weaker preparation
Lesson #4
Edtech Changes the Nature of Information in the System
Information Has Five Properties
Form Magnitude Velocity Direction Access
Source: Nystrom, C. (1973) “Towards a Science of Media Ecology: The Formulation of Integrated ConceptualParadigms for the Study of Human Communication Systems,” unpublished doctoral dissertation
Lesson #4
How Edtech Changes Information in the System and the Impact
• If form changes student literacies expand
• If magnitude changes students can be overwhelmed by information
• If accessibility and direction changes relationship between students and instructor shifts
• If velocity changes feedback can be more instantaneous
Lesson #4
Best Practices in Design, Implementation, and . . .
• Design– Begin with learning objectives– Research what is already known or has been
done related to the innovation
• Implementation– Estimate the amount of resources (time,
funding, space, etc.) needed; then increase it– Assume mid-course corrections will be needed
... Assessment
Assessment at MIT works best when:• It is formative• It is collaborative, which means
– Demands on faculty time must be honored– Differences in research in the “hard” and
“soft” sciences must be made explicit• Quantitative and qualitative methodologies
are used
Next Steps• The longer term impact of the earliest projects are
being studied– TEAL– Mission 200X– 6.002x
• The “lessons learned” will be disseminated through a variety of venues at the Institute and in the wider academic community
• The next set of research questions of interest to the faculty and the TLL staff will be prioritized
Top Related