Technology: Access to the Future

Regional Meetings

Ellen B. Mandinach

Naomi Hupert

EDC Center for Children and Technology

WWW.edc.org/CCT

November 21, 2003

January 23, 2004

February 20, 2004

March 19, 2004

“The problem we face in inserting new technologies into education is that they partly represent new ways of teaching

and partly represent new content that ought to be taught now.”

Lesgold (2000)

“A paradox gradually became evident: The more a technology, and its usages,

fits the prevailing educational philosophy and its pedagogical

application, the more it is welcome and embraced, but the less of an effect it has. When some technology can be smoothly assimilated into existing

educational practices without challenging them, its chances of

stimulating a worthwhile change are very small.”

Salomon and Almog, 1998

According to Papert (1987), if an instructional technology is

harmless that it is easily integrated into existing

pedagogical practices without many changes, then it will be equally harmless in making an

instructional difference.

Question:

What is the difference between elephants mating and the implementation of educational technology?

Answer:

There is a lot of dust and noise, and nothing happens for a very long period of time!

Think of a pointillist painting by Seurat or an impressionist

work by Monet

Step up to the painting.

Step all the way back.

Compare what you see.

That is precisely what evaluation and assessment require - taking multiple perspectives of the same phenomena and getting different feedback.

Question:

How can the accountability issue be addressed?

Answer:With great difficulty

Use a number of measurement strategies, asking different, but related questions, all concerned with various aspects of the learning process and outcomes.

Mandinach & Cline (1994)

Assessment, Evaluation, and Research

Different purposes, questions, and objectives:

Assessment - the measurement of learner performance. Can be part of an evaluation, but not synonymous.

Evaluation - the examination of a system, product, or program - formative, summative, or both - to determine how it is functioning, being implemented, and how it can be improved.

Research - encompasses assessment, evaluation, and much more.

Evaluation and Research:Some Tradeoffs

Is likely to be most effective when it is both formative and summative.

Should be used for planning as well as a systematic research tool.

Must consider the information desired by the stakeholders.

Must use measures that will maximize the potential for detecting impact.

Must use standardized tests or targeted assessments.

Must use experimental versus other designs. Must use comparison/control or matched

groups.

Some Caveats

Need for continuous adjustments in: pedagogical philosophy.

assessment techniques.

strategies, roles, priorities, and schedules.

All parties involved need to change their conceptions of proof of successful implementation of technology and its impact on teaching and learning.

What does it mean to say, “it works”?

Is there enough implementation of the technology to enable

measurement?

There must be enough technology implementation to produce the desired outcomes.

If you are only using technology for a small amount of instructional time, there will be limited exposure for each student each day. Therefore the targeted outcomes are not likely to show substantial effects size.

Evaluation/Research Issues Experimental paradigm

– eliminate possible explanations– verify hypothesized causal relationship– control for contaminating influences– random assignment of students to experimental

and control groups– controlled application of the stimuli

Barriers to and issues in the use of an experimental design.

– selection, acquisition, and installation of hardware and software

– training and ongoing support of teachers– enlisting support, encouragement, and

participation of students, teachers, administrators, school board members, and parents

– actual classroom implementation (for as long as it takes)

Quasi-Experiment Versus Formative Experiment

Not sufficiently powerful

Not sufficiently sensitive to changes

Inadequate evaluative question -

Does it work?

“Rarely does one study produce an unequivocal and

durable result: multiple methods, applied over time

and tied to evidentiary standards, are essential to

establishing a base of scientific knowledge.”

Shavelson & Towne, 2002

Methodological Implications for Technology-Based Educational

Reform Efforts

Longitudinal Design

Multiple Methods

Hierarchical Analyses

System Dynamics

Longitudinal Design

Reasons that quasi-experiments are difficult to implement in classroom settings:

– pre and post measures

– experimental and control groups

– random assignments of students

– control of how and when the “stimulus” is administered

Sacrifice the quasi-experimental design in favor of ongoing longitudinal data collection and analyses that are carried out continuously and indefinitely.

Multiple Methods

Think aloud protocols Classroom observations Peer observations In-depth interviews with teachers, students, and

administrators Content analyses of essays Paper and pencil assessments Performance assessments Notebooks and portfolios Performance on traditional tests, assignments, and projects Many data items routinely generated in the

operation of any school (e.g., GPA, tardiness, attendance, drop-out rates, course taking patterns)

Gather data at multiple time points Case studies Bottom line: Identify consistent patterns across

data sources

Educational innovation efforts often generate multiple outcomes that require the triangulation of traditional and nontraditional data collection and assessment methods. For example:

HIERARCHICAL ANALYSES

Student Learning - Processes and Outcomes

Classroom Dynamics - the Changing Patterns of Interactions between Students and Teachers

The School as a Social Organization - its Structure and Functions

Examine impact at different levels

of analysis:

SYSTEM DYNAMICS

A school is an interdependent, multilayered system. One has to understand what is happening across layers and levels.

Reform efforts take place in real world contexts that are composed of many interrelated, dynamic factors.

Any attempt at educational reform will have multiple components and multiple impacts, and they will interact across levels of organization over time.

Asking a single question about impact or outcome is naive.

Thus, we need to approach educational reform systemically.

“The character of education not only affects the research

enterprise, but also necessitates careful consideration of how the understanding or use of results

can be impeded or facilitated by conditions at different levels of

the system. Organizational, structural, and leadership qualities

all influence how the complex education system works in

practice.”

Shavelson & Towne, 2002

Learning Organization

Administration

Students Teachers

Community

Interactive Learner Directed

Technology

Role ChangesTransformation

Accountability

Curriculum

ResourcesSupport

Assessment

Student Motivation

Procedural Traditional Instruction

Systems InstructionNoname 1

Enhancement

ORGANIZATIONAL CHANGE LEVEL

CLASSROOM PROCESS LEVEL

STUDENT LEARNING LEVEL

How do you know if the technology is “working”?

Questions - better, more, what ifs, what and how they ask

Deeper understanding More engaged and on task How they interact Challenged Argumentation Application Independence and self-directedness Planfulness and organizational skills Increased problem solving, logical analysis Reponses in discussion Decision making See the light bulb go on - the aha experience See the obvious Students in charge of their own learning Success is being able to handle failure and learn

from it

Mandinach & Cline (1994)

QuickTime™ and aTIFF (Uncompressed) decompressorare needed to see this picture.

Source: Trenton Tim es, September, 2003

WHAT Does NCLB Want?

To determine with scientific rigor:

WHAT WORKS.

Translation: The impact of the intervention must be to:

INCREASE TEST SCORES!

What is Required by the What Works Clearinghouse and NCLB

Randomized, controlled, experimental studies, using the medical model of research.

Not matched comparisons. Not quasi-experimental designs. Must establish causality, ruling out plausible

explanations. Small, focused “interventions.” Limited teacher professional development

components. Short-term. School patterns are not changed. Students are the unit of assignment, not classrooms

or schools. No contextualization.

Foremost, there must be valid and reliable evidence that the intervention improves student achievement through scientific evidence.

The Medical Model as the Gold Standard

The Institute for Educational Sciences (IES) in the US Department of Education invokes the medical model of research as the standard toward which all research should strive.

Yet is this gold standard achievable? Is it the right gold standard or a silver

bullet? For example, can an instructional

“intervention” be examined in the same way as a course of pharmaceutical treatment?

Research and Evaluation Methodology Required by NCLB:Randomized Field Trials (RFT’s)

The rationale for RFT’s is the quest for unambiguous information in education.

“To be scientific, the design must allow direct, empirical

investigation of an important question, account for the context in which the study is carried out,

align with a conceptual framework, reflect careful and

thorough reasoning, and disclose results to encourage debate in the

scientific community.”

Shavelson & Towne, 2002

The Six Guiding Principles of Scientific Inquiry

(Not the Seven Deadly Sins)

1. Pose significant questions that can be investigated empirically (ruling out counter interpretations and bringing evidence to bear on alternative explanations)

2. Link research to relevant theory3. Use methods that permit direct

investigation of the question4. Provide a coherent and explicit

chain of reasoning5. Replicate and generalize across

studies6. Disclose research to encourage

professional scrutiny and critique

Which Really is the Driving Factor -

Research Questions or Methods?

The question should drive the research methodology, not the research methodology driving the questions.

Unfortunately, all too often the reverse has been happening because of political pressures.

Mandated questions, methods, and potentially answers as well.

The Question Should Drive the Research Design

What is happening (e.g., descriptions of population characteristics)?

Is there a systematic effect (i.e., systematic means causal)?

How or why does it happen?

Need to account for contextual factors.

Replicability of patterns across groups and time.

Evaluation

Should be meaningful and constructive. The results and information should benefit the students, teachers, school, and district.

Should not be punitive.

Should be informative, providing information on what is going on, how to improve, or other important questions.

Designing Evaluations

Use targeted evaluations.

Match your goals to data collection activities - that is, let the questions drive the methods.

Use measurable components.

Consider the design before it is implemented.

Be flexible. Things change and the evaluation design must change accordingly.

Numerous Caveats to RFT’s

Fidelity of implementation

Variability of treatment

Overlap between treatment and control groups

Adequacy of outcome measures

Multiple treatment interference

Relevance of control condition to policy issues

External validity

NCLB Goals:Impact on Students

Primary Goal - To improve student academic achievement through the use of technology in elementary schools and secondary schools.

Additional Goals – To assist every student in crossing the digital

divide by ensuring that every student is technologically literate by the time the student finishes the eighth grade, regardless of the student’s race, ethnicity, gender, family income, geographic location, or disability.

– To encourage the effective integration of technology resources and systems with teacher training and curriculum development to establish research-based instructional methods that can be widely implemented as best practices by State educational agencies and local educational agencies.

NCLB Questions:Impact on Students

Is academic achievement improving with effective technology use?

Are students acquiring 21st century skills through effective technology use?

Are students more engaged in learning through effective technology use?

Necessary Conditions

Effective Practice - Is classroom practice characterized by powerful, research-based strategies that effectively and appropriately use technology?

Educator Proficiency - Are educators proficient in implementing, assessing, and supporting a variety of technology-based teaching and learning practices?

Robust Access Anywhere Anytime - Do students and staff have robust access to technology anywhere, any time, to support effective designs for teaching and learning?

Digital Age Equity - Is the digital divide being monitored and addressed through resources and strategies aligned to 21st century vision?

Vision and Leadership - Is there a 21st century vision? Is the education system transforming into a high-performance learning organization?

QuickTime™ and aPhoto - JPEG decompressor

are needed to see this picture.

Information and Communication Technology Literacy

ICT literacy is more than just the mastery of technical skills. It also includes:

Cognitive skills.

The application of cognitive skills and knowledge.

ICT literacy is seen as a continuum of skills and abilities from simple, everyday tasks to complex applications.

ICT literacy is using digital technology, communications tools, and/or networks to access, manage,

integrate, evaluate, and create information in order to function in a

knowledge society.

Source: ICT Literacy Panel, 2002.

A Working Definition

ICT Proficiency Skills

ACCESS - knowing about and knowing how to collect and/or retrieve information.

MANAGE - applying an existing organizational or classification scheme.

INTEGRATE - interpreting and representing information. It involves summarizing, comparing, and contrasting.

EVALUATE - making judgments about the quality, relevance, usefulness, or efficiency of information.

CREATE - generating information by adapting, applying, designing, inventing, or authoring information.

Three Proficiencies

Cognitive Proficiency - the desired foundational skills of everyday life at school, at home, and at work. Literacy, numeracy, problem solving, and spatial/visual literacy demonstrate these proficiencies.

Technical Proficiency - the basic components of digital literacy. It includes a foundational knowledge of hardware, software applications, networks, and elements of digital technology.

ICT Proficiency - the integration and application of cognitive and technical skill. Seen as enablers, they allow individuals to maximize the capabilities of technology. At the highest level, ICT proficiencies result in innovation, individual transformation, and societal change.

The SETDA Technology Literacy Working Definition

Technology literacy is the ability to responsibly use appropriate

technology to communicate, solve problems, and access, manage, integrate, evaluate, and create

information to improve learning in all subject areas and to acquire

lifelong knowledge and skills in the 21st century.

Sources of 21st Century Skill Definitions

www.ncrel.org/engauge/skills/sources.htm

The enGauge 21st-Century Skills National Education Technology Standards SCANS (Secretary’s Commission on Achieving

Necessary Skills) A Nation of Opportunity: Building America’s 21st

Century Workforce Preparing Students for the 21st Century Standards for Technological Literacy, Content for

the Study of Technology Being Fluent with Information Technology Information Literacy Standards for Student

Learning Growing Up Digital

Interactive Assessment and Evaluation

Assessment can be used both as a teaching tool and an evaluation mechanism.

Assessment and instruction should be iterative - a feedback loop to provide information to both student and instructor - continuous and interactive.

Assessments should assist students to evaluate their learning processes and outcomes. They should help to facilitate learning.

Techniques should capitalize on the affordances of the technology.

Online assessment should not be restricted by time constraints or resources.

Interactive Assessment and Evaluation

(Continued)

Techniques should be creative (e.g., games, puzzles, competitions).

Group assessments can be effective tools. Consider collaborative learning groups and team projects.

Information from assessments should provide the teacher with information about a student’s strengths and weaknesses.

Design thought provoking questions that stimulate interesting debate and threaded discussions.

Interactive Assessment and Evaluation

(Continued)

One form of assessment effective online is project-based learning - intensive, long-term, and focused on specific, real-world topics and authentic problems - accomplished individually or collaboratively.

Another form is a short, iterative instruction - assessment feedback cycle.

Use multiple methods.

Focus on the formative, not just summative - processes, not just outcomes, with chances for reassessment, acknowledging that every student has a different learning curve.

Interactive Assessment, Evaluation, and Instruction:

Issues

Collaborative learning necessitates new paradigms for learning, instruction, assessment, and honesty.

Rules for assessment need to be made clear to the students. If assessments are conducted properly, cheating may become a moot issue.

Expectations should be made clear in terms of participation online, including threaded discussions.

Teacher training is a key issue in terms of the technology, paradigm shifts, and role changes.

Interactive Assessment, Evaluation, and Instruction:

More Issues

Encourage interaction among student and teacher.

Provide opportunities for peer review, not just teacher feedback.

Encourage active participation and learning.

Provide prompt feedback.

Recognize diversity in learning - different paths, different paces, different learning styles.

Tailored to individual needs.

Provides a learning trail.

Some Existing K-12 Products and Companies

Smarterkids.com

LeapFrog

K12 - William Bennett’s diagnostic tests

Classroom Connect NOW

ETS K-12 Works

Ignite! - Neil Bush’s company

Many others

A mega-market now at $105 billion (NY Times, 1/21/01)

A Major Caution

Many of the companies are jumping onto the bandwagon and the testing craze.

Tests should be professionally developed and psychometrically sound. Many companies are cutting corners in the rush to market.

Consider in the selection process the pedagogical theory that underlies the tests and the psychometric models. Are the tests even grounded in theory????

BE CAREFUL!

Interactive Assessment, Evaluation, and Instruction: A

Paradigm Shift

Any time, any where, any pace. Immediate knowledge and infinite resources. Global learning. Multimedia. Interactive rather than lecturing or didactic. Learner-directed, not teacher-directed. Active versus passive. Student engagement. Role changes and shifts in responsibility for

learning. Collaboration versus individual learning. “Guide on the side” versus “sage of the stage”.

The teacher as a facilitator and coach, not as the primary transmitter of information.

Intellectual exploration.

Recommendations

Implementation

Don’t depend on the technology itself to improve teaching.

Teachers change their practices more readily when they can work in teams and have the support of administrators.

Teachers are more likely to use technology effectively when they have a computer at home on which to practice.

Technical support for teachers should be available at the building level, not just at the district level.

Recommendations (continued)

Use the technology to support existing activities instead of designing the activities around technology. Develop activities for the sake of learning, not for the chance to use the technology.

Don’t rely on traditional tests to determine what students have learned. Create activities that allow students to demonstrate what they’ve discovered through their projects to enable a more accurate assessment of their knowledge and skills.

Be patient. Teachers need time to learn how the technology can be used in the classroom. Students then need time to discover the knowledge by themselves and in collaboration with classmates.

Recommendations (continued)

Teacher Professional Development

Training should focus on how technology can enhance learning – and be embedded in real projects – rather than simply on how to work the machines.

Teachers need time to rework lessons to use technology effectively and experiment with new teaching styles.

If teachers are encouraged to share with other teachers what they are trying in the classroom, momentum for change can grow.

Teachers need professional development opportunities and continuing support from a facilitator or coach to learn how to use the technology.

Agenda for Research and Action

Conduct targeted case studies that address effective implementations in diverse educational settings.

Conduct research that addresses the “ramping up” issue.

Provide collaborative outreach and research opportunities to schools to begin to bridge the digital divide.

Shift the focus of teacher professional development to include technology as an important component of the emerging educational paradigms and pedagogical philosophies.

QuickTime™ and aPhoto - JPEG decompressor

are needed to see this picture.

The Center for Children and TechnologyA division of the Education Development Center

WWW.edc.org/CCT

Resource:Identifying and Implementing Educational

Practices Supported By Rigorous Evidence: A User Friendly Guide

www.excelgov.org/evidence