OCRI OCRI SymposiumSymposium

OCRI is a joint effort between the Oregon Association of Colleges of Teacher OCRI is a joint effort between the Oregon Association of Colleges of Teacher Education (OACTE) and the Oregon Association of Teacher Educators (ORATE).Education (OACTE) and the Oregon Association of Teacher Educators (ORATE).

“Investigating the efficacy of teacher work sample methodology in teacher

preparation”

AACTE February 25th, 2007

This Symposium is Dedicated to the This Symposium is Dedicated to the Memory and Legacy of Dr. Del SchalockMemory and Legacy of Dr. Del Schalock

Dr. Del Schalock, Professor, The Teaching Research Institute, Western Oregon University

Lead Developer and Advocate of the Teacher Work Sample Methodology (TWS)

Symposium OverviewSymposium Overview IntroductionIntroduction

Paper 1: Collaborative research design in teacher Paper 1: Collaborative research design in teacher preparation: Blueprints and processes preparation: Blueprints and processes

Paper 2: Evolution of policy implementation of teacher work Paper 2: Evolution of policy implementation of teacher work samples in Oregonsamples in Oregon

Paper 3: Perceptions of teacher educators and candidates Paper 3: Perceptions of teacher educators and candidates on the use of teacher work sampleson the use of teacher work samples

Paper 4: Teacher work sample effects on the learning of K-Paper 4: Teacher work sample effects on the learning of K-12 students 12 students

DiscussionDiscussion

IntroductionIntroduction

Paper 1Paper 1

Collaborative research design in Collaborative research design in teacher preparation: Blueprints and teacher preparation: Blueprints and

processesprocesses

Ronald A. Beghetto (University of Oregon)Ronald A. Beghetto (University of Oregon)

Linda Samek (Corban College)Linda Samek (Corban College)

Imagine…Imagine…

A focused, coherent multi-institutional collaborative A focused, coherent multi-institutional collaborative research initiativeresearch initiative Nearly a dozen institutions (public and private)Nearly a dozen institutions (public and private) Engaged in meaningful and sustainable teacher Engaged in meaningful and sustainable teacher

education research.education research.

This vision is a realityThis vision is a reality With several partnerships across the nationWith several partnerships across the nation And, most recently, a new network in the state of OregonAnd, most recently, a new network in the state of Oregon

The Oregon Collaborative Research Initiative (OCRI).The Oregon Collaborative Research Initiative (OCRI).

Blue Prints and ProcessesBlue Prints and Processes

Under what conditions do such partnerships Under what conditions do such partnerships develop? develop?

How do collaborative groups develop and How do collaborative groups develop and carry-out their research agendas? carry-out their research agendas?

What seems to support such efforts? What seems to support such efforts? What challenges do such efforts face? What challenges do such efforts face?

Historical ContextHistorical Context

The Diversity of OACTE InstitutionsThe Diversity of OACTE Institutions 19 institutions19 institutions

6 public universities6 public universities 3 research and/or urban3 research and/or urban 3 regional3 regional

12 private non-profit colleges and universities12 private non-profit colleges and universities 6 religious6 religious 6 secular6 secular

1 for profit university1 for profit university Small to largeSmall to large

From newly approved with no completers toFrom newly approved with no completers to Hundreds each yearHundreds each year

New to oldNew to old Multnomah Bible College program to begin fall 2007Multnomah Bible College program to begin fall 2007 Western Oregon University, preparing teachers since 1882Western Oregon University, preparing teachers since 1882

Precursors to OCRIPrecursors to OCRI

All subject to rules governing programs from the Oregon All subject to rules governing programs from the Oregon Teacher Standards and Practices CommissionTeacher Standards and Practices Commission

Mid 1990s – Oregon Collaborative for Excellence in Mid 1990s – Oregon Collaborative for Excellence in Preparation of Teachers, a 5-year, $5 million NSF Preparation of Teachers, a 5-year, $5 million NSF collaborative that included public, private, and community collaborative that included public, private, and community collegescolleges

1998-2001 – Oregon Quality Assurance in Teaching, a 4-1998-2001 – Oregon Quality Assurance in Teaching, a 4-year, multi-million dollar grant that involved the 16 four-year year, multi-million dollar grant that involved the 16 four-year institutions with teacher preparation programsinstitutions with teacher preparation programs

Common interests in the education of children through Common interests in the education of children through OACTEOACTE

Perfect Storm that Spawned OCRIPerfect Storm that Spawned OCRI

Leaders from all institutions have common goalsLeaders from all institutions have common goals Oregon TSPC requires evidence of competent Oregon TSPC requires evidence of competent

performance and student learningperformance and student learning All student teachers are required to produce All student teachers are required to produce

Teacher Work SamplesTeacher Work Samples More Oregon institutions interested in NCATE More Oregon institutions interested in NCATE

accreditationaccreditation Decentralized education enterprise leaves Decentralized education enterprise leaves

everyone under-fundedeveryone under-funded OACTE had treasury funds that needed to be usedOACTE had treasury funds that needed to be used

Sowing the Seeds of Sowing the Seeds of CollaborationCollaboration

Building a Flexible BlueprintBuilding a Flexible Blueprint ApproachApproach

Listen, synthesize, and verify Listen, synthesize, and verify WorkflowWorkflow

An iterative, multistage process of planning and activityAn iterative, multistage process of planning and activity

Research Designer

CORT

Project Strand Leaders

Local Participants & Partner Sites

Primary Goals of OCRIPrimary Goals of OCRI

The The Oregon Collaborative Research Oregon Collaborative Research InitiativeInitiative (OCRI) (OCRI) was established with was established with the purpose of:the purpose of: Engaging Oregon's teacher preparation Engaging Oregon's teacher preparation

institutions in the collaborative study and institutions in the collaborative study and dissemination of a dissemination of a meaningfulmeaningful and and sustainablesustainable set of research projects. set of research projects.

MeaningfulMeaningful

MeaningfulMeaningful in that the projects address in that the projects address questions of common interest, are questions of common interest, are aimed at improving teacher aimed at improving teacher preparation, and have real-world preparation, and have real-world implications for teacher development implications for teacher development and PK-12 student learning. and PK-12 student learning.

SustainableSustainable

SustainableSustainable in that the research in that the research projects do not overburden projects do not overburden participants but rather offer feasible participants but rather offer feasible and rewarding opportunities to engage and rewarding opportunities to engage in and disseminate the findings of in and disseminate the findings of collaborative inquiry.collaborative inquiry.

Collaborative FocusCollaborative Focus

Our initial collaborative endeavor has Our initial collaborative endeavor has focused on examining the efficacy of focused on examining the efficacy of teacher work sample methodology in teacher work sample methodology in preparing teachers in the state of preparing teachers in the state of Oregon. Oregon.

TWSM: An Initial ProjectTWSM: An Initial Project

Three interrelated questions:Three interrelated questions: How has the TWSM How has the TWSM evolvedevolved? ? What is the What is the experienceexperience of teacher of teacher

educators and candidates with the TWSM?educators and candidates with the TWSM? What What effecteffect has the TWSM had on PK-12 has the TWSM had on PK-12

students?students?

Example Planning ToolsExample Planning Tools

Evolution StudyEvolution Study

Goal: Goal: Examine how TWSM first became a Examine how TWSM first became a mandate and how implementation has mandate and how implementation has evolved over time.evolved over time.

Brief overview: Brief overview: Using a “Theory of Action” Using a “Theory of Action” framework, juxtapose the enactment of framework, juxtapose the enactment of TWSM with its intended and espoused TWSM with its intended and espoused theory.theory.

Experience StudyExperience Study

Goal: Goal: Document the perceptions of teacher Document the perceptions of teacher educators and candidates regarding TWSM.educators and candidates regarding TWSM.

Brief overview: Brief overview: Participants’ perceptions Participants’ perceptions (representing public and private institutions) (representing public and private institutions) regarding the meaningfulness, usefulness, regarding the meaningfulness, usefulness, and applicability will be examined and and applicability will be examined and reported on.reported on.

Effect StudyEffect Study Goal: Goal: Examine how TWSM has impacted PK-Examine how TWSM has impacted PK-

12 learners.12 learners. Brief overview: Analysis of completed work Brief overview: Analysis of completed work

samples to examine how work samples are samples to examine how work samples are impacting K-12 learning. This effort also impacting K-12 learning. This effort also focuses on the development and refinement focuses on the development and refinement of research/measurement tools most of research/measurement tools most appropriate for analyzing the impact of appropriate for analyzing the impact of TWSM.TWSM.

Supports and Supports and ChallengesChallenges

Key MetaphorsKey Metaphors

““Cyclical scaffolding” and “loose coupling” of Cyclical scaffolding” and “loose coupling” of parallel efforts as metaphors for supporting parallel efforts as metaphors for supporting collaborative research designcollaborative research design Key challenges:Key challenges:

Anticipating and providing “just in time” assistanceAnticipating and providing “just in time” assistance Helping to navigate research strand “autonomy” and Helping to navigate research strand “autonomy” and

“relatedness”“relatedness” Issues and challenges with respect to intellectual Issues and challenges with respect to intellectual

property (e.g., datasets), secondary analyses, and property (e.g., datasets), secondary analyses, and dissemination of findings.dissemination of findings.

Concluding ThoughtsConcluding Thoughts

Even though the Oregon group is still in the early Even though the Oregon group is still in the early stages of this process, stages of this process, the project has already attained no small measure of the project has already attained no small measure of

success. success. Only time will tell what the Oregon group is able to Only time will tell what the Oregon group is able to

accomplish in this ambitious endeavor.accomplish in this ambitious endeavor. W are highly optimistic and, indeed, energized by W are highly optimistic and, indeed, energized by

the idea thatthe idea that multiple institutions can come together multiple institutions can come together

To forge a strong link between the research and practice of To forge a strong link between the research and practice of teacher education, and teacher education, and

Work towards enhancing teacher preparation and, in turn, PK12 Work towards enhancing teacher preparation and, in turn, PK12 student learning. student learning.

Evolution studyEvolution study

Paper 2 – Evolution StudyPaper 2 – Evolution Study

Evolution of Policy and Implementation of Evolution of Policy and Implementation of Teacher Work Sample in OregonTeacher Work Sample in Oregon

Presenter:Presenter:Hilda Rosselli - Western Oregon UniversityHilda Rosselli - Western Oregon University

Research Team:Research Team:Dew Anna Brumley - Warner Pacific UniversityDew Anna Brumley - Warner Pacific University

Maria Cirielllo - University of PortlandMaria Cirielllo - University of PortlandHilda Rosselli - Western Oregon UniversityHilda Rosselli - Western Oregon University

Research QuestionsResearch Questions

1. What led to teacher work samples becoming mandated?

2. How have teacher work samples evolved over time within the state of Oregon?

3. To what extent do dominant patterns of implementation run counter to the major intended premises of the state’s policy?

(espoused theory and theory in use)

Research DesignResearch Design

Phase 1Phase 1 Historical--4 qualitative interviewsHistorical--4 qualitative interviews Key players involved with the early stages of Key players involved with the early stages of

teacher work samples policy developmentteacher work samples policy development Phase 2Phase 2

Survey data collected from 8 institutionsSurvey data collected from 8 institutions Early implementation, evolution of practice, challenges that have Early implementation, evolution of practice, challenges that have

evolvedevolved

Phase 3Phase 3 Document analysisDocument analysis

MethodologyMethodology

Four individuals selected for interviewing Four individuals selected for interviewing representing varying perspectivesrepresenting varying perspectives Del Schalock—Teacher researcher Dave Myton— Teacher education policymaker Kathy Wiper—Teacher on state commission Holly Zanville—Higher education official

MethodologyMethodology

Conducted 4 semi-structured interviewsConducted 4 semi-structured interviews Transcribed interview notesTranscribed interview notes Conducted membership checksConducted membership checks Coded transcripts individuallyCoded transcripts individually Compared coding schemesCompared coding schemes Reviewed and refined themesReviewed and refined themes

Theme 1: Original IntentTheme 1: Original Intent

What were teacher work samples designed What were teacher work samples designed originally to accomplish? What were they a originally to accomplish? What were they a reaction to or remedy for?reaction to or remedy for? An eAn evidence-based process designed to demonstrate

that student teachers can teach and that students learn under the guidance of a student teacher.

Legislators were looking beyond teacher mandated standardized tests for more accountability from teacher education programs about their graduates.

Theme 1: Original IntentTheme 1: Original Intent

What were teacher work samples designed What were teacher work samples designed originally to accomplish? What were TWSs a originally to accomplish? What were TWSs a reaction to or remedy for?reaction to or remedy for? Oregon Teacher Standards and Practices Commission

were looking to output standards for program review purposes

Going beyond what do teachers know and do to producing evidence that indicated student learning.

Theme 1: Original Intent Theme 1: Original Intent

“Document teacher performance beyond knowledge and connected to the standards”

Del Schalock

“…analogy of the driver’s test. Not enough to just take a written driver’s test. Need to demonstrate the skill to drive.”

Holly Zanville

Theme 1: Original Intent Theme 1: Original Intent

[It also] “ arose out of a dissatisfaction with the practice of using tests to demonstrate teacher competence”

Holly Zanville

“At Western it was clearly to go beyond the wall we had hit with competency based teacher ed where the issue is not what teachers know and can do. By 1974 we knew we had to go somewhere else because that mode didn’t help us so we realized we needed to connect teaching and learning.”

Del Schalock

Theme 2: Roots and ContextTheme 2: Roots and Context

What else was happening at the same time that influenced the development and adoption of TWSM? National, regional, state contexts? Intersection of key events

Development of competency based programs Creation of state curriculum standards in K-12 Increased accountability for teacher prep

Theme 2: Roots and ContextTheme 2: Roots and Context

Other factorsOther factors Elementary Secondary Act and engagement with Elementary Secondary Act and engagement with

schools (late 60’s and 70’s)schools (late 60’s and 70’s) Com-Field- a study funded through NWREL offered a Com-Field- a study funded through NWREL offered a

model for competency based, field-centered, and model for competency based, field-centered, and personalized teacher preparation (65 - 74)personalized teacher preparation (65 - 74)

Formation of TSPC, the nation’s first independent Formation of TSPC, the nation’s first independent commission for teacher education (1971-72)commission for teacher education (1971-72)

Commitment to context for research: Teaching Commitment to context for research: Teaching Research Institute and College of Education at WOUResearch Institute and College of Education at WOU

TSPC revision of programs to standards-based (1973)TSPC revision of programs to standards-based (1973)

Theme 2: Roots and ContextTheme 2: Roots and Context

Other factors Other factors Mid-Valley Consortium to assist districts with goal-based Mid-Valley Consortium to assist districts with goal-based

education and integration of Curriculum, Instruction, and education and integration of Curriculum, Instruction, and Assessment (1975 - 1984)Assessment (1975 - 1984)

Publication of Publication of Nation at RiskNation at Risk Report to Joint Boards referencing work samples (1986) Report to Joint Boards referencing work samples (1986) 21st Century Schools Act in Oregon (1991) 21st Century Schools Act in Oregon (1991) Analysis of this act and its implications for teachers.Analysis of this act and its implications for teachers. Recasting of teacher education standards to outcome Recasting of teacher education standards to outcome

based (1997)based (1997)

Theme 2: Roots and ContextTheme 2: Roots and Context

“ The school context in which teacher ed was being fashioned became much more outcome oriented, e.g. what we want our kids to know and be able to do and hold ourselves to responsibility for assessing that. Wasn’t that big a jump to apply this to teacher ed which led to TWSM.”

Del Schalock

Theme 2: Roots and ContextTheme 2: Roots and Context

“ComField was field centered—a graduated set of field experiences—moving from observations, to small group teaching, to full responsibility for teaching. That was a frame of reference that was firmly in the faculty minds so we married that with the goal based curriculum instruction demands for Oregon’s standards based movement.”

Del Schalock

Theme 2: Roots and ContextTheme 2: Roots and Context

“Behind closed doors [they] were able to pinpoint right to the classroom door where students were meeting the expectations….what they were getting at with the student test data that they were beginning to be able to look at effective schools.”

Dave Myton

Theme 2: Roots and ContextTheme 2: Roots and Context

Differing opinion

““I don’t think TWSMs were intended to address the credibility of the institutions but it did allow conversation with legislators in which Colleges could say we are using multiple measures to assess quality.”

Holly Zanville

Theme 3: Instrumental PlayersTheme 3: Instrumental Players

What individuals or agencies or political issues were instrumental in the start up and then evolution of TWSM? Del SchalockDel Schalock Theory and practice context at WOUTheory and practice context at WOU TSPCTSPC

Theme 3: Instrumental PlayersTheme 3: Instrumental Players

Who sat down and crafted the first draft? “I think I did.”

Del Schalock

“…75% probability that the language of TWSM came out of the research team in the mid 70s. The last half of the 70s was where it was being formulated. Tested at WOU in the late 70s or early 80s. Had sufficient testing within WOU before it went to TSPC.”

Del Schalock

Theme 3: Instrumental PlayersTheme 3: Instrumental Players

“They [TRI/WOU] built and still maintain a huge database of student test results…got the superintendents at the table looking at the data and it was all over the place. So their concern was how to ensure that kids are getting not only a good education.

Gerald????so he created a formula. It filled up the whole front of a classroom…but what it was to see if there were correlations with any of the inputs.”

Dave Myton

Theme 3: Instrumental PlayersTheme 3: Instrumental Players

“The 21st Century Schools Act didn’t mention teacher licensing but the Board of Ed and the Board of Higher Ed both indicated that they were going to do something, they were going to take Teacher Ed back, if the Commission didn’t do something pretty drastic.”

Dave Myton

Theme 3: Instrumental PlayersTheme 3: Instrumental Players

“So in the end, it kind of turned out to be the commission taking hold of it and saying we want them [student teachers] to have a work sample, you can waive anything else.”

Dave Myton

Theme 4: Evolution of IntentTheme 4: Evolution of Intent

What other purposes did teacher work What other purposes did teacher work samples attempt to serve over time?samples attempt to serve over time? Stronger clinical supervision modelStronger clinical supervision model Follow-up on teacher candidates performanceFollow-up on teacher candidates performance Quartile learning gains for groups of studentsQuartile learning gains for groups of students Shift of emphasis from learning gains to also Shift of emphasis from learning gains to also

more focus on student teacher performancemore focus on student teacher performance

Theme 4: Evolution of IntentTheme 4: Evolution of Intent

“Maybe unintended but one of the purposes was to create a closer link between student teachers and mentor teachers in the classrooms. Became a tool for tightening the expectations for student teaching and an opportunity to work together with the school personnel. Mentoring a student teacher couldn’t be a time to go sit in the lounge.”

Holly Zanville

Theme 4: Evolution of IntentTheme 4: Evolution of Intent

““The responsibility for the evaluating of work samples never became the CTs [cooperating teachers’] responsibility because of the OEA opposition.”

Kathy Wiper

Theme 4: Evolution of IntentTheme 4: Evolution of Intent

“And people were thinking, ‘now we know we have good people coming in, and we can see them in practice, but what we want to know though, is how are they when they leave? Not only the door of the university, but how are they 1, 2, or 5 years out? And frankly, we never get it, partly because the turnover rate in teaching is so high. And second because nobody had resources to go out and observe them in the field.”

Dave Myton

Theme 4: Evolution of IntentTheme 4: Evolution of Intent

““ In the early iterations of the work sample, we were talking about quartiles of students. If we were to go back and collect the old graphs of the standards…The goal of the work sample was to show that students had learned by quartiles, as opposed to each student.”

Dave Myton

Theme 4: Evolution of IntentTheme 4: Evolution of Intent

“Work sample purposes have evolved from teaching. Looking for student gains first…the meaningful gains that were focused on the CCGs (Common Curriculum Goals), then shifted to looking at the student teaching…demonstrating knowledge and skills.”

Kathy Wiper

Theme 4: Evolution of IntentTheme 4: Evolution of Intent

“The language that went into the TSPC standard was faithful to the idea of TWS but what didn’t go into the implementation were performance demands around learning gains. Became an exercise. Can you do it? TSPC never really put any quality control mechanisms in place. How those standards were interpreted and used? TSPC believes that this is partly handled through the program review.”

Del Schalock

InsightsInsights

1. Del Schlock is the “father” of work sample methodology - he had access to the research based on his position at

Teaching and research - he had funds through the Early Models program - he worked with the mid valley consortium

2. The state and national context in which TWSs first emerged was essential to understanding the full intent of the espoused theory as was the state of affairs in BOTH K-12 schooling and teacher preparation.

InsightsInsights

3. Although TWSM is required in order to complete a teacher education program and receive a license, the specific content and implementation process has been left to the discretion of a particular teacher education program. It has never been exactly prescribed by TSPC. Therefore different kinds of work samples and evolutions have flowed naturally from each institution which tailored the requirements to fit the philosophy and needs of the specific program.

Maria Ciriello

InsightsInsights

4. “In 2002-03, the detractors started coming out and telling the legislators that they were being burdened with this extensive TWS project requirement. And it’s more than any teacher should ever need. They had the opportunity to make teacher education a real profession and a self-governing profession. And by running back to the legislature saying TSPC is going to make it too hard to be a teacher, it undercut a lot of work.”

Dave Myton

InsightsInsights

5. ““I thought one of the changes that went into place that was helpful was when it was mandated that all campuses were to do TWSMs. Then a collaborative approach emerged. There was sharing and that was an important tool around which campuses, both private and public, could collaborate.”

Holly Zanville

InsightsInsights

6. “In a perfect world, we would have spent 6. “In a perfect world, we would have spent the energy required to get solid observations the energy required to get solid observations in place and quality measures of student in place and quality measures of student learning and its interpretation. But that not learning and its interpretation. But that not the case. So, in some respects, it’s all been the case. So, in some respects, it’s all been a little vacuous but faculties in Oregon have a little vacuous but faculties in Oregon have continued to play with that concept over continued to play with that concept over time.”time.”

Del Del SchalockSchalock

Experience studyExperience study

Paper 3 – Experience StudyPaper 3 – Experience StudyPerceptions of teacher educators and Perceptions of teacher educators and candidates on the use of teacher work candidates on the use of teacher work

samplessamples

Presenters:Presenters:Blaine C. Ackley – University of PortlandBlaine C. Ackley – University of Portland

Melina Dyer – Lewis & Clark CollegeMelina Dyer – Lewis & Clark College

Research Team:Research Team:Blaine C. Ackley – University of PortlandBlaine C. Ackley – University of Portland

Roy Bunch – Corban CollegeRoy Bunch – Corban CollegeEmily de la Cruz – Portland State UniversityEmily de la Cruz – Portland State University

Melina Dyer – Lewis & Clark CollegeMelina Dyer – Lewis & Clark CollegeMollie Galloway – Lewis & Clark CollegeMollie Galloway – Lewis & Clark College

Experience Team MembersExperience Team Members Roy BunchRoy Bunch, Corban College, Chair, Corban College, Chair Blaine AckleyBlaine Ackley, University of Portland, University of Portland Janine AllenJanine Allen, Northwest Christian College, Northwest Christian College Emily de la CruzEmily de la Cruz, Portland State University, Portland State University Melina DyerMelina Dyer, Lewis & Clark College, Lewis & Clark College Mollie GallowayMollie Galloway, Lewis & Clark College, Lewis & Clark College Mary JohnsonMary Johnson, George Fox University, George Fox University Ann MatschinerAnn Matschiner, Pacific University, Pacific University Tisha MorrellTisha Morrell, University of Portland, University of Portland Paula PolittePaula Politte, Pacific University, Pacific University Amanda SmithAmanda Smith, Western Oregon University, Western Oregon University

Survey InstrumentsSurvey Instruments Twelve quantitative response questions using a Likert Twelve quantitative response questions using a Likert

scale from 1 (strongly disagree to) to 5 (strongly agree)scale from 1 (strongly disagree to) to 5 (strongly agree) Three questions about the TWS methodologyThree questions about the TWS methodology Nine questions about elements of TWSNine questions about elements of TWS

Two qualitative questions:Two qualitative questions: In what ways is the work sample process and product beneficial in In what ways is the work sample process and product beneficial in

the development of an effective teacher?the development of an effective teacher?

What, if anything, would you change about the work sample What, if anything, would you change about the work sample requirements to make the process and product more beneficial in requirements to make the process and product more beneficial in the development of effective teachers?the development of effective teachers?

Survey Data ResponsesSurvey Data Responses

8 Teacher Education Institutions8 Teacher Education Institutions 33 Faculty Respondents33 Faculty Respondents 242 Candidates who had completed at least 242 Candidates who had completed at least

one Teacher Work Sample (TWS)one Teacher Work Sample (TWS)

Quantitative Survey ResultsQuantitative Survey Results No statistically significantNo statistically significant

differencesdifferences in candidates in candidates perceptions of the work perceptions of the work sample by:sample by: Institution type (public or Institution type (public or

private)private)

Graduate and Graduate and undergraduate teacher undergraduate teacher education programseducation programs

GenderGender

Number of work samples Number of work samples completed (1 vs. more completed (1 vs. more than 1)than 1)

For candidates there was a For candidates there was a statistically significant moderate statistically significant moderate correlationcorrelation between elements of between elements of TWS and the overall work sample TWS and the overall work sample (.627, p<.01)(.627, p<.01)

Faculty rated elements of the TWS Faculty rated elements of the TWS more highly than candidates:more highly than candidates: t t (316)=5.38, p<.001(316)=5.38, p<.001 Faculty M=4.20, SD=.65Faculty M=4.20, SD=.65 Candidate M=3.66, SD=.65Candidate M=3.66, SD=.65

Faculty and candidate felt the TWS Faculty and candidate felt the TWS best aided them in lesson planning best aided them in lesson planning and assessment:and assessment: M=4.07, SD=.93 for planning itemM=4.07, SD=.93 for planning item M=3.88, SD=.77 for assessment itemsM=3.88, SD=.77 for assessment items

Qualitative Analysis of TWS BenefitsQualitative Analysis of TWS Benefits

Faculty find the TWS more beneficial overall than Faculty find the TWS more beneficial overall than candidates.candidates.

Candidates focus on the specific elements of the TWS Candidates focus on the specific elements of the TWS while Faculty focus on the entire process.while Faculty focus on the entire process.

TWS Benefits

0

20

40

60

80

Planning Assessing Reflecting Other Teaching “BigPicture”

%Candidates Faculty

Respondents’ Exemplars: Respondents’ Exemplars: Benefits of TWSBenefits of TWS

TeachingTeaching CandidateCandidate: “… Writing and putting the work sample : “… Writing and putting the work sample

together was a necessary evil, but the experience together was a necessary evil, but the experience teaching is what made the work sample important to teaching is what made the work sample important to me.”me.”

FacultyFaculty: “It helps them understand the intricacies of : “It helps them understand the intricacies of teaching…teaching…

““Big Picture”Big Picture” Candidate:Candidate: “It helped me to think about the large picture “It helped me to think about the large picture

as well as the small details in my teaching.”as well as the small details in my teaching.”

Faculty:Faculty: “The entire process forces the student teacher “The entire process forces the student teacher to plan, teach, and assess a unit of instruction. “to plan, teach, and assess a unit of instruction. “

Qualitative Analysis of TWS ImprovementsQualitative Analysis of TWS Improvements

Faculty and candidates agree improvements Faculty and candidates agree improvements should be made to the logistical implementation of should be made to the logistical implementation of the TWS.the TWS.

Faculty to a greater degree than Candidates Faculty to a greater degree than Candidates indicated that TWS modifications are needed.indicated that TWS modifications are needed.

TWS Improvements

020

4060

80100

120140

Planning Assessing Reflecting Other Teaching “BigPicture”

Logistics Modify theProcess

NewMethod

%

Candidate Faculty

Respondents’ Exemplars: Improvements to TWSRespondents’ Exemplars: Improvements to TWS LogisticsLogistics

Candidate:Candidate: “The handbook that was supposed to guide us “The handbook that was supposed to guide us and provide requirements was scarce, poorly organized, and provide requirements was scarce, poorly organized, with bad examples.”with bad examples.”

Faculty:Faculty: “We seem to be forcing them to concentrate “We seem to be forcing them to concentrate more on the wording than the teaching process, and they more on the wording than the teaching process, and they put a lot of time into duplicating pages of information from put a lot of time into duplicating pages of information from the lesson plans…”the lesson plans…”

Modify the ProcessModify the Process Candidate:Candidate: “The work sample does not allow for “The work sample does not allow for

flexibility.  There were some days I had to completely flexibility.  There were some days I had to completely change my lesson, and this has no place in the work change my lesson, and this has no place in the work sample other than reflections sample other than reflections

Faculty:Faculty: “The work sample process often becomes one of “The work sample process often becomes one of jumping through hoops for students. Is it pretty enough? jumping through hoops for students. Is it pretty enough? Is their grammar correct? “Is their grammar correct? “

Respondents’ Exemplars: Improvements Respondents’ Exemplars: Improvements to TWS (2)to TWS (2)

New MethodNew Method Candidate:Candidate: “Work sample requirements could “Work sample requirements could

be more flexible, perhaps shorter.”be more flexible, perhaps shorter.”

FacultyFaculty: “The work sample both as a process : “The work sample both as a process and product is valuable, if and only if, the and product is valuable, if and only if, the students get a chance to discuss it with friendly students get a chance to discuss it with friendly cooperating teachers and peers. Otherwise, it cooperating teachers and peers. Otherwise, it tends to remain superficial.”tends to remain superficial.”

Next StepsNext Steps

Spring SurveySpring Survey Most candidates will have completed a second Most candidates will have completed a second

work sample.work sample. We will add a code to distinguish between We will add a code to distinguish between

elementary and secondary candidates.elementary and secondary candidates. We will administer the qualitative questions We will administer the qualitative questions

before the quantitative questions.before the quantitative questions.

Effect studyEffect study

Paper 5 – Effect StudyPaper 5 – Effect Study

Teacher work sample effects on the learning of Teacher work sample effects on the learning of K-12 studentsK-12 students

Presenters:Presenters:Tom Greene, Co-Chair, University of Portland

Mark Girod, Co-Chair, Western Oregon Jan Albrecht, Concordia UniversityDana Barbarick, Cascade College

Karen Buchanan, George Fox UniversityMickey Caskey, Portland State University

Jane Kovisto, Concordia UniversityLinda Samek, Corban College

Jacqueline Waggoner, University of Portland

The Effect GroupThe Effect Group

Consists of self-selected volunteers from participating Consists of self-selected volunteers from participating public and private institutionspublic and private institutionsCo-Chairs, one public, other privateCo-Chairs, one public, other private

Guiding PrincipleGuiding Principle:: What kinds of information do teachers need to support and What kinds of information do teachers need to support and

measure student progress toward learning in standards-measure student progress toward learning in standards-based systems of schooling most effectively?based systems of schooling most effectively?

Judging Instruction and Judging Instruction and Assessment…Assessment…

Candidate level variables - gender, type of program, Candidate level variables - gender, type of program, endorsements, authorization levelendorsements, authorization level

School and classroom level variables - SES, mobility, School and classroom level variables - SES, mobility, race/ethnicity, program eligibility….race/ethnicity, program eligibility….

Instructional methods by lessonInstructional methods by lesson Type of assessment strategies by outcomeType of assessment strategies by outcome

Judging Alignment…Judging Alignment…

Judgments about alignment Judgments about alignment betweenbetween::

State StandardsState Standards District StandardsDistrict Standards GoalsGoals ObjectivesObjectives InstructionInstruction Pre/Formative/Post Pre/Formative/Post

AssessmentsAssessments

Judging Learning Gains…Judging Learning Gains…

By individual studentBy individual student By categorization of By categorization of

student services - ELL, student services - ELL, IEP, TAG, None…IEP, TAG, None…

By Race/ethnicityBy Race/ethnicity By type of learning By type of learning

outcomeoutcome Met, Progress, Not metMet, Progress, Not met

Pilot StudiesPilot Studies

How will TWS data gathering process How will TWS data gathering process function at different institutions?function at different institutions?

Are there instrumentation and procedural Are there instrumentation and procedural issues across institutions that need to be issues across institutions that need to be modified prior to full implementation? modified prior to full implementation?

Two Pilot StudiesTwo Pilot Studies

Pilot StudyPilot Study II IIII

TimeTime Spring, 2006Spring, 2006 Fall, 2006Fall, 2006

Number of Number of TWSTWS

8888 99

Number of Number of P-12 P-12 StudentsStudents

19071907 214214

Results: Pilot StudiesResults: Pilot Studies

TWSs addressed the following subjects: math, language arts, social TWSs addressed the following subjects: math, language arts, social studies, science, world languages, health and PEstudies, science, world languages, health and PE

Teaching methods - Most frequent: Student- centered (62%); Teaching methods - Most frequent: Student- centered (62%); Teacher-centered (27%)Teacher-centered (27%)

Assessment Methods - Most Frequent: Selected Fixed Response at Assessment Methods - Most Frequent: Selected Fixed Response at the knowledge level; Student-constructed response at the the knowledge level; Student-constructed response at the knowledge levelknowledge level

Results: Pilot StudiesResults: Pilot Studies

92% of the P-12 students had learning gains92% of the P-12 students had learning gains

There were no significant difference in the learning gains or There were no significant difference in the learning gains or no learning gains based on race, TAG, SPED, ESL, or in no learning gains based on race, TAG, SPED, ESL, or in schools with poverty designationschools with poverty designation

Greater frequency of learning gains in high school…Greater frequency of learning gains in high school…Function of instrumentation?Function of instrumentation?

What is Next?What is Next?

Use revised measuresUse revised measures Gathering faculty supportGathering faculty support Consider consequences for Consider consequences for

pedagogypedagogy Local point people will clean and Local point people will clean and

organize data and submit to a organize data and submit to a central databasecentral database

Full implementation across Full implementation across seven institutionsseven institutions

Continue cross validationContinue cross validation

ChallengesChallenges

Ownership of data - Institutions, Authors, Associations? IRB process across all institutions Communication - frequent management meetings Institutional integrity - compliance with protocols Congruity of definitions - developing statewide common

terminology Support from stakeholders Time, Money, Resources Theory in action—differences in perspectives

General DiscussionGeneral Discussion