Teach For America EVALUATION REPORT · 2018. 7. 30. · Teach For America (TFA), an organization...

REACH FURTHER.Global competitiveness starts here.

Center for Research & EvaluationOffice of Accountability

Teach For America

EVALUATION REPORTMay 2011

EVALUATION OF TEACH FOR AMERICA IN CHARLOTTE-MECKLENBURG SCHOOLS

May 30, 2011

Prepared by: Jason A. Schoeneberger

Senior Analyst, Charlotte-Mecklenburg Schools

In collaboration with the Departmental of Educational Leadership, College of Education

University of North Carolina at Charlotte

Center for Research & Evaluation Office of Accountability

Charlotte-Mecklenburg Schools

For more information, contact: Center for Research & Evaluation

(980) 343-6242

©2011, Charlotte-Mecklenburg Schools

Dr. Christian FriendDirector of the Center

for Research and Evaluation

Dr. Lynne TingleExecutive Director of

Performance Management

Robert AvossaChief Accountability Officer

Dr. Peter GormanSuperintendent

TABLE OF CONTENTS

Executive Summary ....................................................................................................... 1

Introduction ................................................................................................................... 3

Method .......................................................................................................................... 11

Results ........................................................................................................................... 16

Discussion .................................................................................................................... 92

May, 2011 Teach for America Evaluation | 1

Office of Accountability | Center for Research and Evaluation

EXECUTIVE SUMMARYThe current evaluation report, prepared by the Center for Research and Evaluation (CRE), is the second of two annual reports for Charlotte-Mecklenburg Schools (CMS) on the local Teach for America (TFA) program. Some typical critiques of the TFA program and recruits include the lack of traditional training on pedagogy and the limited two-year service commitment. Some of the research literature challenges these critiques by asserting that the TFA program is a worthy endeavor that shows better performance for math and science classes in TFA-led classrooms (Schoeneberger, Dever, & Tingle, 2009; Xu, Hannaway, & Taylor, 2009; Decker, Mayer, & Glazerman, 2004). Other research has shown some negative or mixed results (Darling-Hammond, Holtzman, Gatlin, & Heilig, 2005). This study assumed a mixed-methods approach, including both quantitative and qualitative compo-nents. A manuscript of the first report was submitted for publication in a peer-reviewed journal. As part of that process, the authors obtained valuable feedback from anonymous reviewers about ways to improve that work. As such, the current study took into account the suggested changes, particularly for the quantitative analyses. This study aimed to compare student achievement outcomes between TFA and Comp-TFA teachers within CMS in an effort to examine the success of the local TFA program against recent state and national research findings. Further, we set out to collect qualitative information from principals and TFA teachers about their experience with the TFA program as a whole.

Major Study Findings � Elementary level math proficiency rates for TFA teachers were similar to Comp-TFA

teachers assigned to TFA schools (TFA-Comp), but fell short of Comp-TFA teachers assigned to Comp-TFA schools (Comp-TFA). Middle school math proficiency rates for TFA teachers were greater than TFA-Comp rates, but were still lower than Comp-TFA rates.

� Elementary reading and science proficiency rates for TFA teachers were similar to TFA-Comp teachers, but fell short of Comp-TFA teachers. Middle school reading and science proficiency rates for TFA teachers were similar to TFA-Comp rates, but were still lower than Comp-TFA rates.

� EOC proficiency rates for TFA teachers were similar to TFA-Comp rates in Algebra I, Biology, English I in 2009-10, and U.S. History in 2009-10. EOC rates for TFA were greater than TFA-Comp rates in Algebra II in 2009-10, Civics & Economics in 2008-09, and Geometry in 2009-10. TFA rates were higher than TFA-Comp and Comp-TFA rates in English I in 2007-08 and Physical Science in 2008-09 and 2009-10.

� Elementary math growth for TFA teachers was similar to TFA-Comp and Comp-TFA teachers, exhibiting an increase across the three years of analysis. Middle school math growth for TFA teachers was greater than TFA-Comp and Comp-TFA teacher growth.

� Elementary reading growth for TFA teachers was much lower in 2007-08, but was comparable to both TFA-Comp and Comp-TFA teachers in 2008-09 and 2009-10. Middle school reading growth for TFA teachers was similar to TFA-Comp and Comp-TFA teacher growth across all three years.

2 | Teach for America Evaluation May, 2011


� EOC growth for TFA teachers was similar to TFA-Comp growth in Civics & Economics in 2008-09, Chemistry in 2009-10, Geometry in 2008-09 and 2009-10 and U.S. History in 2009-10. EOC growth for TFA was greater than TFA-Comp growth in Algebra I in 2009-10 and Biology in 2007-08. TFA growth was higher than TFA-Comp and Comp-TFA growth in Algebra I in 2008-09, Algebra II in 2009-10 and Physical Science in 2008-09 and 2009-10.

� A significant, positive effect was found for TFA teachers when examining math achievement in 2009-10 through multilevel analyses, with non-significant effects noted in 2008-09 and 2007-08.

� A significant, positive effect was found for TFA teachers when examining reading achievement in 2008-09 through multilevel analyses, with a non-significant effect noted in 2009-10. A significant, negative effect was found for TFA teachers when examining reading in 2007-08.

� A significant, positive effect was found for TFA teachers when examining science achievement in 2008-09 through multilevel analyses, with non-significant effects noted in 2007-08 and 2009-10.

� Significant, positive effects obtained through multilevel analyses were noted for Geometry in 2008-09, Algebra II in 2008-09 and Biology in all three years.

� TFA and Comp-TFA classrooms exhibited similar levels of student engagement and differentiated instruction based on observation data, while TFA teachers spent more time controlling the classroom than their Comp-TFA peers.

� TFA teachers were more likely to utilize paired or small group work, while Comp-TFA was more likely to utilize whole-group instruction. Additionally, TFA teachers were more likely to encourage collaborative learning and were more likely to have a clearly stated objective for each lesson.

� Principals generally maintained positive views of the TFA program and its impact at their schools, commenting that TFA teachers are eager and willing to learn more about their role. Many principals, however, also had suggestions for improving the TFA program.

� Many principals felt as though TFA teachers tended to have strong leadership skills, actively seek out leadership roles, and provide services to students beyond traditional school hours.

� All but one TFA teacher expressed positive views about their experience in the TFA program, though most also offered suggestions for ways to improve the experience for other recruits.

� All but one TFA teacher felt as though they received adequate support from staff at their assigned school, stating that fellow faculty and administrators were very support-ive and helpful.

� TFA teachers reported relatively high confidence in their ability to engage students, facilitate learning, and gain comfort with subject material.



INTRODUCTIONStudent academic achievement is an output generated from a number of complex, interrelated inputs including teachers (Mendro et. al., 1998; Rivkin, Hanushek, & Kain, 2000; Rowan, Correnti, & Miller, 2002; Sanders & Rivers, 1996; Shkolnik et al., 2002; Wright, Horn, & Sanders, 1997), parental involve-ment (Houtenville, 2008; Jeynes, 2005; Walberg, 1984) and school climate (Freiberg, 1999; Lee & Smith, 1999; Sherblom, Marshall, & Sherblom, 2006). The research literature suggests that teachers have a strong influence on the academic outcomes of their students. This makes intuitive sense, as teachers directly provide the instruction of concepts that students need to succeed and move forward with their education. However, research has also shown that a great deal of variation exists between teachers, with some teachers performing more effectively than others (McAffrey, Lockwood, Koretz, & Hamilton, 2003). The recent phenomenon, generally referred to as strategic staffing, involves iden-tifying the teacher most successful in helping students achieve and placing them where they can serve the students most in need. Teach For America (TFA), an organization whose sole purpose is to recruit top-flight graduates from the nation’s premier schools to teach in the most challenging K-12 settings, assists districts in meeting the goals of their strategic staffing initiatives (Azimi, 2007).The TFA program has seen dramatic growth in both applicants and placements since its inception in 1990 (Xu, Hannaway, & Taylor, 2009). TFA initially received 2,500 applications and placed 500 teach-ers in 1990. Most recently, in 2005, TFA received 17,000 applications and expects to place over 4,000 teachers by 2010 (Xu et al., 2009). These figures alone suggest that TFA has been successful in meeting its mission to help staff America’s most needy schools. However, skeptics still question whether TFA recruits, without traditional education training, can have the same impact on their students as tradi-tionally trained teachers. Despite the increased applications and interest of customer districts, the amount of educational research investigating the impact of TFA is limited. Mathematica researchers conducted a random assignment study published in 2004 to compare student achievement outcomes between students taught by TFA teachers and non-TFA teachers in the same schools and grade levels (Decker, Mayer, & Glazerman, 2004). Decker and colleagues (2004) found that students taught by TFA teachers outperformed stu-dents taught by comparison teachers based on mathematics assessment scores. No differences were noted between the two groups of teachers with respect to reading achievement. Of particular interest, when TFA teachers were analyzed with novice comparison teachers, the effect associated with TFA was even larger than when analysis was conducted with the full sample. Quasi-experimental studies conducted by Raymond, Fletcher, and Luque (2001), Darling-Hammond, Holtzman, Gatlin, and Heilig (2005) and Schoeneberger, Dever, and Tingle (2009) have also shown positive effects for students taught by TFA teachers with respect to mathematics achievement. Schoeneberger, Dever, and Tingle (2009) found positive effects associated with TFA teachers at the high school level; and found positive effects for TFA reading teachers when compared only with teach-ers with similar years (less than two) of experience. Darling-Hammond and colleagues (2005) did find negative effects associated with TFA teachers in subjects other than math. Other larger-scale, quasi-experimental studies have been conducted with data from New York City



schools. Making use of longitudinal data, Kane, Rockoff, and Staiger (2006) found positive effects for TFA with respect to student mathematics achievement while controlling for years of teaching experi-ence, with effects somewhat larger at the middle school level compared to the elementary level. Boyd, Grossman, Lankford, Loeb, Michelli, and Wyckoff (2006) also found middle school math TFA teachers had a significant advantage when compared to other middle school math teachers. Kane et al. and Boyd et al. both found no significant differences or effects in favor of comparison teachers with respect to reading achievement. Qualitative research on the TFA program is even more limited, though some recent findings indicate support for the program. Veltri (2008) conducted a longitudinal study incorporating the perceptions of TFA corps members, mentors, and administrators to the TFA literature. Data integrated interviews and “teacher-researcher” field notes were gathered over eight years from more than 300 participants who discussed their TFA teaching experiences. Despite noted concerns regarding TFA’s current model, mission, and goals, Veltri’s (2008) findings suggested that TFA corps members actively work to learn the culture of their assigned school and the surrounding community, as well as learning the complexi-ties of teaching during their TFA commitment (Veltri, 2008). Schoeneberger, Dever, and Tingle (2009) interviewed principals and TFA teachers about their expe-rience with the TFA program. Some principals reported dissatisfaction with certain aspects of the TFA program, but, in general, expressed high levels of satisfaction with TFA teachers’ ability in the classroom. TFA teachers reported personal satisfaction with their experiences in the TFA program and school placement, though nearly all had suggestions for ways to improve the program for future recruits. Classroom observations of TFA and non-TFA teachers revealed similarities with respect to levels of teacher confidence, evidence of pre-planned activities and equivalent amounts of culturally appropriate quotations and pictures. Differences observed included the level of respect observed in student-teacher dynamics, where TFA teachers demanded higher levels of respect to be shown in their classrooms. Additionally, TFA teachers were observed to utilize more types of classroom management strategies, ask more open-ended questions, and emphasized more real-world connections to material.Another important study found alternative-route teachers (including TFA corps members) were more likely than traditionally-trained teachers to believe that good teachers can help children learn, even those from disadvantaged backgrounds. Moreover, alternative-route teachers were more likely to say that wanting to help underprivileged students was their main reason for becoming teachers (National Comprehensive Center for Teach Quality & Public Agenda (NCCTQ), 2007). This study also noted how alternate-route teachers were more likely than traditionally-trained teachers to cite a lack of support by administrators as the major drawback of teaching (NCCTQ, 2007). Furthermore, NCCTQ (2007) found that new alternative-route teachers were more likely than traditionally-trained teachers to give their administrators low ratings for their instructional leadership and support on discipline issues. They were also more likely to give their fellow teachers lower marks for supporting and advising them (NCCTQ, 2007). The current project seeks to improve up on earlier work conducted by Schoeneberger, Dever, and Tingle (2009) that was submitted for publication in a peer-reviewed journal. As part of that review process, the authors were provided valuable feedback about how the study could be improved and warrant publication. Specifically, reviewers suggested making use of more sophisticated analyses to



account for the clustering of students within classes, teachers and schools and to take into account more than a single prior achievement score. To accomplish this, we made use of two years’ worth of prior achievement and modeled the data using multilevel modeling methods, taking into account class, teacher, and school level effects. The purpose of this project was twofold: to better understand what impact TFA teachers have on student achievement and to ascertain the perceptions of principals and TFA participants about their experience with the TFA program. To accomplish these goals, the Center for Research (CRE) in CMS made use of a mixed methods design incorporating both quantitative and qualitative data collection and analysis strategies. Quantitative strategies were geared toward answering the question regarding student achievement, while the qualitative strategies were generally geared toward the question regard-ing participant perceptions. Specific descriptions of the methods used follows.



METHOD

QuantitativeA total of 294 TFA teachers were placed in CMS over the last three years based on a data file sup-plied by the Human Resources office (102 in 2007-08, 130 in 2008-09 and 62 in 2009-10). A match against a database containing student schedule records for EOG/EOC tested courses identified 169 TFA matches (meaning 125 non-matches). Initial analyses involved a basic comparison of proficiency rates among three distinct subgroups: students taught by TFA teachers, students taught by non-TFA teachers working in schools where TFA teachers were assigned, and non-TFA teachers working in other CMS schools where no TFA teachers were assigned. To isolate records of interest, courses taught by TFA teachers were flagged across all teacher groups so only records reflecting TFA-taught courses in each of the three years were retained. Student proficiency designations were obtained for Reading, Mathematics and Science for the 2007-08, 2008-09 and 2009-10 school years. Rates of proficiency by these subgroups were calculated for presentation in bar graphs in the Results section. To provide context to these results, we provided the number of teachers within each teacher group for each of the three years in Tables 1 and 2.

Table 1. Teacher Count by EOG Subject, Type and Year.

2007-08 2008-09 2009-10Subject Teacher Type n % n % n %Math Non-TFA 1,756 90.66 1,358 69.61 1,402 72.87

TFA-Comp 168 8.67 545 27.93 470 24.43TFA 13 0.67 48 2.46 52 2.7

Reading Non-TFA 1,752 90.26 1,352 68.98 1,403 72.54TFA-Comp 176 9.07 553 28.21 475 24.56

TFA 13 0.67 55 2.81 56 2.9Science Non-TFA 503 91.45 385 70 390 73.03

TFA-Comp 39 7.09 149 27.09 125 23.41TFA 8 1.45 16 2.91 19 3.56



Table 2. Teacher Count by EOC Subject, Type and Year.

2007-08 2008-09 2009-10Subject Teacher Type n % n % n %ALG1 Non-TFA 120 82.76 113 63.13 132 59.73

TFA-Comp 23 15.86 60 33.52 75 33.94TFA 2 1.38 6 3.35 14 6.33

ALG2 Non-TFA 73 83.91 57 61.96 58 58.59TFA-Comp 14 16.09 34 36.96 38 38.38

TFA . . 1 1.09 3 3.03BIOL Non-TFA 104 82.54 76 65.52 66 55.46

TFA-Comp 19 15.08 34 29.31 43 36.13TFA 3 2.38 6 5.17 10 8.4

C & E Non-TFA 85 82.52 46 50.55 58 52.25TFA-Comp 17 16.5 43 47.25 48 43.24

TFA 1 0.97 2 2.2 5 4.5CHEM Non-TFA 41 83.67 41 69.49 39 60

TFA-Comp 8 16.33 16 27.12 22 33.85TFA . . 2 3.39 4 6.15

ENGL Non-TFA 123 81.46 90 61.64 102 58.96TFA-Comp 25 16.56 51 34.93 66 38.15

TFA 3 1.99 5 3.42 5 2.89GEOM Non-TFA 70 84.34 54 67.5 45 62.5

TFA-Comp 10 12.05 22 27.5 23 31.94TFA 3 3.61 4 5 4 5.56

PHYS Non-TFA 23 85.19 13 68.42 17 65.38TFA-Comp 4 14.81 6 31.58 7 26.92

TFA . . . . 2 7.69PSCI Non-TFA 40 86.96 31 63.27 33 70.21

TFA-Comp 6 13.04 17 34.69 11 23.4TFA . . 1 2.04 3 6.38

USHI Non-TFA 72 81.82 51 60 46 55.42TFA-Comp 16 18.18 34 40 36 43.37

TFA . . . . 1 1.2

The same pattern is apparent across both tables where the non-TFA group is significantly larger than both the TFA-Comp and TFA groups and the TFA-Comp is larger than the TFA group. In examining all of the results that follow, note that the mean values or estimated effects from inferential analyses are based on information attributable to the teachers contained in counts above. Given the small number of teachers comprising the TFA group, particularly at the EOC level, confidence intervals around esti-mates will generally be wider reflecting our uncertainty about what the true value is in the population.



Although the proficiency comparisons provide a succinct, easy-to-understand synopsis of student per-formance based on teacher group, those summaries do not take into account any of the measured differences that may exist between students assigned to the non-TFA, TFA-Comparison, or TFA teach-ers, classrooms or schools, including prior performance, demographic backgrounds, or absenteeism. As a second method of analysis to partially address this concern, North Carolina ABC student growth values were obtained for each student across the subjects areas. These values inherently take into account student prior performance as growth values are based on a linear regression model incorpo-rating up to two years’ worth of data. Finally, multilevel regression analyses were conducted, taking into account the nested structure of student data (students nested in classrooms, within teachers and within schools) to yield more accurate estimates of an effect associated with instruction by a TFA teacher. To adjust the outcomes for pre-existing differences between students, classrooms and schools for TFA and non-TFA teachers, student demographics, two years of prior performance (for science, the mean of available prior reading and mathematics scores was used), attendance and incident information were entered into the models at various levels. Ultimately, the models yield an estimated effect of being instructed by TFA teachers, controlling for differences in student, classroom composition (aggregate values of students), teacher experience and school differences (aggregate values of students). These effects, in standardized units along with confidence intervals, are presented using forest plots in the Results section.

QualitativeRecently, mixed methods research has begun to assume a more prominent role in the applied researcher’s toolkit. Despite firm historical lines between purists from both the quantitative and qualitative camps, many researchers recognize that both paradigms provide important and useful benefits (Onwuegbuzie & Leech, 2004a). In an attempt to draw on the strengths and minimize the limitations of a single-scope research study, a qualitative component was incorporated to capture information not represented by the quantitative data generally available in public education. The qualitative component included observations of both non-TFA and TFA classrooms, as well as interviews of TFA teachers and principals of schools where TFA teachers were on staff. Due to limited resources in the form of dedicated staff, the CRE solicited assistance from the Department of Educational Leadership at the University of North Carolina at Charlotte (UNCC). UNCC recruited five students from the Principal Fellows program funded by the North Carolina General Assembly where entry is based upon academic merit. Students of this program receive a scholarship loan in exchange for two years of full-time study toward a Master’s degree in educational administration. These stu-dents agreed to conduct the classroom observations and interviews on a single day in April during the 2009-10 school year. In addition, a faculty member within the Department of Educational Leadership served as a sixth observer/interviewer. Through this effort, observations and interviews of teachers and principals were conducted by independent, objective personnel with no embedded interest in the outcome of the evaluation.

SamplingA total of nine schools were selected as potential sites for observations and interviews, with three schools representing each of the elementary, middle, and high school levels. CRE staff attempted to



identify schools within each level that had varying levels of overall academic success based on Annual Yearly Progress (AYP) status. UNCC Principal Fellows were allowed to select which school they would visit, though all levels had to be represented. Two each of the elementary, middle and high schools were chosen by the UNCC Principal Fellows and faculty member. Within each school, two TFA teachers were randomly selected to be observed. If a school had TFA teachers with varying levels of experience (i.e., in their first or second year), a teacher was selected rep-resenting each level of experience. Non-TFA teachers serving as comparisons for observation purposes were identified by matching the type of course that the selected TFA teachers taught and the number of years of experience. Perfect matches were not available in every instance, but teachers teaching the most similar courses with the closest number of years of experience were flagged as the comparison teachers.

Observation ProtocolCRE staff developed an observational protocol instrument designed to guide the UNCC observers in gathering information while observing teachers in the classroom (see Appendix A). Basic information including the number of students present in the class, whether a teaching assistant or co-teacher was present, and a general description of the lesson were collected on the observation form. A five-minute interval checklist was also used to monitor the type of instructional activity engaged in by the teach-ers (and students), as well as a rating of the level of student engagement during the same time period. Observers estimated how much class time was spent in whole class instruction versus small group work or work conducted individually by students. In addition, observers recorded how much time they thought the teacher spent attempting to maintain control of the class, the types of interactions that occurred between the students and teacher, and the frequency and nature of any differentiated instruction. Finally, observers recorded their perceptions of the Lesson Design, Content of the Lesson, Implementation of the Lesson, Teacher-Student Interactions, Classroom Culture, Likely Impact of the Lesson, and the Overall Lesson Quality using a rating scale format.

Teacher Interview GuideCRE staff also created a semi-structured teacher interview guide to assist observers during the inter-view process (see Appendix B). Though specific questions were included on the guide, observers were instructed to follow paths during the interview process if a particularly pertinent topic arose. In general, the interviews were conducted to determine TFA teachers’ general experience with the TFA program, the school they were assigned to, and their perceptions about their own efficacy in the classroom. TFA teachers were also solicited for suggestions on how to improve their experience and whether or not they planned to continue teaching beyond their two-year TFA commitment.

Principal Interview GuideA semi-structured principal interview guide was also provided to each observer (see Appendix C). Similar to the teacher interview guide, the principal guide was intended to gather general principal perceptions and experiences. Principals were asked to comment on the impact that the TFA program has had on their school, their satisfaction with the TFA teachers assigned to their school, and their perceptions of TFA teachers’ abilities in the areas of Leadership Skills/Classroom Management and Curriculum/Subject Proficiency. Principals were also asked to provide suggestions for ways to improve their TFA experience.



RESULTS

QuantitativeProficiency ComparisonBasic descriptive analyses using data from all three TFA cohort years were conducted to provide a general picture of TFA performance. The following bar graphs summarize the proficiency attained by students taught by three sub-groups of teachers within the constructed data file: “Non-TFA” teachers who taught a TFA-instructed subject but were assigned to a school that did not have any TFA teachers, “TFA-Comp” teachers who taught a TFA-instructed subject in a school with assigned TFA teachers and “TFA” who are TFA teachers. These sub-groups were created to provide a more equal comparison for TFA teachers assigned to high-poverty, low-performing schools. In addition, the analyses were disag-gregated by level, where Elementary includes 3rd, 4th and 5th grade data while Middle includes 6th, 7th and 8th grade data.

Figure 1. Elementary Math Proficiency by Group for 2007-08, 2008-09 and 2009-10.

Figure 1 above shows the rate of proficiency in mathematics attained by elementary students taught by TFA teachers was less than that of non-TFA teachers teaching in schools where no TFA teachers were assigned; this phenomenon held true across all three years. However, when comparing the TFA teachers to non-TFA teachers teaching in schools where TFA teachers were placed, the differences were dramatically reduced for 2007-08 and were reversed for 2008-09 and 2009-10 where TFA rates were slightly higher than TFA-Comp teachers. Note the wider confidence interval error bars for the TFA group, reflecting our uncertainty about the estimate due to a smaller sample (less students instructed by TFA) than in the other groups.



Figure 2. Middle School Math Proficiency by Group for 2007-08, 2008-09 and 2009-10.

Figure 2 above shows the rate of proficiency in mathematics attained by middle grade students taught by TFA teachers was less than that of non-TFA teachers teaching in schools where no TFA teachers were assigned; this phenomenon held true across all three years. However, when comparing the TFA teachers to non-TFA teachers teaching in schools where TFA teachers were placed, the differences disappeared and students of TFA teachers performed better than students of comparison non-TFA teachers.



Figure 3. Elementary Reading Proficiency by Group for 2007-08, 2008-09 and 2009-10.

Figure 3 above shows the rate of proficiency in reading attained by elementary students taught by TFA teachers was less than that of non-TFA teachers teaching in schools where no TFA teachers were assigned; this phenomenon held true across all three years. However, when comparing the TFA teach-ers to non-TFA teachers teaching in schools where TFA teachers were placed, the differences were dramatically reduced, particularly in 2008-09 and 2009-10.



Figure 4. Middle School Reading Proficiency by Group for 2007-08, 2008-09 and 2009-10.

Figure 4 above shows the rate of proficiency in reading attained by middle grade students taught by TFA teachers was less than that of non-TFA teachers teaching in schools where no TFA teachers were assigned; this phenomenon held true across all three years. However, when comparing the TFA teachers to non-TFA teachers teaching in schools where TFA teachers were placed, the differences disappeared and students of TFA teachers performed as well as students of comparison non-TFA teachers.



Figure 5. Elementary Science Proficiency by Group for 2007-08, 2008-09 and 2009-10.

Figure 5 above shows the rate of proficiency in science attained by elementary students taught by TFA teachers was less than that of non-TFA teachers teaching in schools where no TFA teachers were assigned; this phenomenon held true across all three years. However, when comparing the TFA teach-ers to non-TFA teachers teaching in schools where TFA teachers were placed, the differences were either reduced (2007-08) or reversed (2008-09 and 2009-10).



Figure 6. Middle School Science Proficiency by Group for 2007-08, 2008-09 and 2009-10.

Figure 6 above shows the rate of proficiency in science attained by middle grade students taught by TFA teachers was less than that of non-TFA teachers teaching in schools where no TFA teachers were assigned; this phenomenon held true across all three years. However, when comparing the TFA teachers to non-TFA teachers teaching in schools where TFA teachers were placed, the differences disappeared and students of TFA teachers performed as well as or slightly better than students of com-parison non-TFA teachers.



Table 3. EOC Proficiency Summarized by Subject and Teacher Group.

2007-08 2008-09 2009-10Subject Teacher Type Lower Mean Upper Lower Mean Upper Lower Mean UpperALG1 Non-TFA 0.54 0.56 0.57 0.74 0.76 0.77 0.78 0.79 0.80

TFA-Comp 0.48 0.52 0.55 0.69 0.71 0.73 0.65 0.67 0.68TFA 0.08 0.22 0.36 0.64 0.69 0.74 0.63 0.66 0.69

ALG2 Non-TFA 0.67 0.69 0.70 0.76 0.77 0.78 0.80 0.81 0.82TFA-Comp 0.44 0.48 0.52 0.68 0.70 0.72 0.69 0.71 0.73

TFA . . . 0.32 0.45 0.58 0.70 0.78 0.86BIOL Non-TFA 0.68 0.69 0.70 0.78 0.79 0.80 0.80 0.81 0.82

TFA-Comp 0.56 0.59 0.62 0.65 0.67 0.69 0.65 0.67 0.68TFA 0.42 0.51 0.60 0.60 0.65 0.70 0.64 0.67 0.70

C & E Non-TFA 0.62 0.63 0.64 0.70 0.72 0.74 0.74 0.75 0.76TFA-Comp 0.49 0.52 0.55 0.64 0.66 0.68 0.63 0.65 0.67

TFA 0.21 0.32 0.44 0.63 0.69 0.74 0.60 0.65 0.70CHEM Non-TFA 0.67 0.69 0.71 0.82 0.83 0.84 0.89 0.90 0.91

TFA-Comp 0.42 0.47 0.52 0.71 0.74 0.77 0.74 0.77 0.79TFA . . . 0.43 0.51 0.59 0.60 0.66 0.71

ENGL Non-TFA 0.69 0.70 0.71 0.77 0.79 0.80 0.81 0.82 0.83TFA-Comp 0.61 0.64 0.66 0.66 0.68 0.70 0.67 0.68 0.70

TFA 0.67 0.74 0.80 0.53 0.60 0.67 0.62 0.69 0.76GEOM Non-TFA 0.71 0.72 0.73 0.81 0.83 0.84 0.82 0.83 0.84

TFA-Comp 0.64 0.67 0.70 0.71 0.74 0.76 0.72 0.74 0.76TFA 0.50 0.58 0.65 0.66 0.71 0.76 0.76 0.80 0.85

PHYS Non-TFA 0.76 0.78 0.81 0.88 0.91 0.93 0.93 0.95 0.98TFA-Comp 0.29 0.37 0.45 0.61 0.70 0.78 0.80 0.85 0.90

TFA . . . . . . 0.42 0.57 0.71PSCI Non-TFA 0.46 0.48 0.51 0.55 0.58 0.60 0.58 0.61 0.63

TFA-Comp 0.43 0.49 0.54 0.57 0.60 0.62 0.57 0.60 0.64TFA . . . 0.66 0.79 0.92 0.61 0.67 0.73

USHI Non-TFA 0.60 0.62 0.63 0.67 0.69 0.70 0.72 0.73 0.75TFA-Comp 0.40 0.43 0.47 0.60 0.63 0.65 0.65 0.68 0.70

TFA . . . . . . 0.56 0.66 0.76

Table 3 above summarizes proficiency information by EOC subject and teacher group. The columns labeled Lower and Upper represent the lower and upper limits of the 95% confidence interval about the mean, similar to the error bars depicted in the bar graphs for EOG subjects. Looking at the table across subjects, the general pattern emerges where students instructed by TFA teachers generally perform below students instructed by non-TFA or TFA-Comp teachers in 2007-08 (except for ENGL). Over the next two years, the performance of TFA-instructed students tends to improve. The TFA cells



highlighted mark instances where the TFA group performed similarly (or better) than either non-TFA or TFA-Comp students. Note the wider confidence intervals associated with the TFA mean estimates, reflecting the smaller sample size and subsequent uncertainty about the estimate. The proficiency results presented above provide a quick snapshot of student proficiency based on whether they were instructed by a non-TFA teacher, a TFA-Comparison teacher or a TFA teacher. These summaries, however, do not take into account the prior academic performance of students. North Carolina ABC Growth scores, calculated using a regression-based method incorporating at least one (two when available) years’ worth of prior scores, inherently take into account the prior perfor-mance of students. These scores are presented in a standardized format, where zero equates to ‘expected growth’, a positive value indicates outpacing expectations and a negative value suggests a student has fallen short of expected growth. Graphs of mean growth by TFA teacher group are presented for 2007-08, 2008-09, and 2009-10 below.

Figure 7. Elementary Math Growth by Group for 2007-08, 2008-09, and 2009-10.

Figure 7 above shows the mean mathematics growth for elementary students instructed by the three teacher groups of interest: “Non-TFA” teachers who taught a TFA-instructed subject but were assigned to a school that did not have any TFA teachers, “TFA-Comp” teachers who taught a TFA-instructed subject in a school with assigned TFA teachers and “TFA” who are TFA teachers. In 2007-08 and 2008-09, we see similar patterns where students instructed by non-TFA teachers attained the highest growth, followed by students instructed by TFA teachers. In 2009-10, however we see that students instructed by TFA teachers attained higher levels of growth than non-TFA and TFA-Comp students. Note the par-ticularly wide error bars for the TFA group, representing the 95% confidence interval around the mean growth value. Essentially, these wider bars reflect our uncertainty about the true mean value stemming from the smaller number of data points (students instructed by TFA teachers) compared to the other



two groups.Figure 8 below shows the mean mathematics growth for middle grade students instructed by the three teacher groups. Here we see that students instructed by TFA teachers outperformed both the non-TFA and TFA-Comp students across all three years.

Figure 8. Middle School Math Growth by Group for 2007-08, 2008-09, and 2009-10.



Figure 9. Elementary Reading Growth by Group for 2007-08, 2008-09, and 2009-10.

Figure 9 above shows the mean reading growth for elementary students instructed by the three teacher groups. In 2007-08, students instructed by TFA teachers attained much lower levels of growth com-pared to both non-TFA and TFA-Comp students. In 2008-09, TFA students attained a slightly better level of growth than the TFA-Comp students, but still exhibited less than expected growth. In 2009-10, TFA students were still behind both comparison groups, but exhibited a small portion of positive growth.



Figure 10. Middle School Reading Growth by Group for 2007-08, 2008-09, and 2009-10.

Figure 10 above shows the mean reading growth for middle grade students instructed by the three teacher groups. In 2007-08, students instructed by TFA teachers attained lower levels of growth com-pared to both non-TFA and TFA-Comp students. In 2008-09, TFA students attained a slightly better level of growth than the TFA-Comp students, but still trailed the non-TFA students. Finally, in 2009-10, TFA students attained the same level of mean growth as did the non-TFA students, and outpaced the TFA-Comp students.



Table 4. EOC Growth Summarized by Subject and Teacher Group.

2007-08 2008-09 2009-10

Subject Teacher Type Lower Mean Upper Lower Mean Upper Lower Mean UpperALG1 Non-TFA 0.03 0.05 0.07 0.17 0.19 0.21 0.20 0.21 0.23

TFA-Comp -0.04 0.00 0.04 0.16 0.19 0.22 0.13 0.15 0.18TFA -0.33 -0.01 0.31 0.16 0.26 0.35 0.16 0.20 0.25

ALG2 Non-TFA 0.13 0.15 0.18 0.07 0.09 0.12 0.08 0.10 0.12TFA-Comp -0.05 0.04 0.13 0.03 0.07 0.10 -0.05 -0.02 0.01

TFA . . . -0.24 -0.08 0.08 0.14 0.28 0.42BIOL Non-TFA 0.06 0.08 0.10 0.12 0.14 0.16 0.13 0.15 0.16

TFA-Comp -0.03 0.02 0.06 0.09 0.11 0.14 0.09 0.12 0.14TFA -0.07 0.08 0.22 -0.04 0.03 0.10 0.03 0.08 0.13

C & E Non-TFA 0.02 0.04 0.06 0.10 0.13 0.16 0.11 0.13 0.15TFA-Comp 0.03 0.10 0.16 0.11 0.14 0.17 0.13 0.16 0.18

TFA -0.53 -0.38 -0.24 0.03 0.11 0.19 -0.07 -0.01 0.06CHEM Non-TFA 1.01 1.16 1.31 -0.19 -0.16 -0.13 0.10 0.14 0.17

TFA-Comp 0.89 1.28 1.67 -0.08 -0.03 0.02 0.12 0.16 0.21TFA . . . -0.41 -0.29 -0.17 0.03 0.12 0.21

ENGL Non-TFA 0.11 0.13 0.15 0.10 0.12 0.14 0.09 0.11 0.13TFA-Comp -0.02 0.02 0.06 0.06 0.09 0.12 0.02 0.05 0.07

TFA -0.20 -0.10 0.00 -0.19 -0.04 0.11 -0.10 -0.01 0.09GEOM Non-TFA 0.02 0.04 0.06 0.02 0.04 0.06 0.08 0.10 0.12

TFA-Comp 0.05 0.13 0.20 0.04 0.07 0.10 0.08 0.11 0.14TFA -0.21 -0.02 0.18 0.00 0.07 0.14 0.07 0.13 0.20

PHYS Non-TFA 0.04 0.37 0.69 0.00 0.12 0.25 0.18 0.30 0.43TFA-Comp -0.50 0.76 2.02 -0.17 0.32 0.80 0.15 0.26 0.38

TFA . . . . . . -0.15 0.10 0.35PSCI Non-TFA 0.02 0.09 0.15 -0.13 -0.08 -0.03 -0.04 0.00 0.05

TFA-Comp 0.20 0.31 0.42 0.15 0.20 0.24 0.04 0.10 0.16TFA . . . 0.29 0.55 0.80 0.12 0.24 0.36

USHI Non-TFA 0.20 0.30 0.40 -0.20 -0.16 -0.13 -0.02 0.02 0.06TFA-Comp 0.09 0.26 0.42 0.04 0.08 0.13 0.10 0.15 0.19

TFA . . . . . . -0.11 0.12 0.34

Table 4 above displays the mean growth, along with the 95% confidence interval around the means by year and teacher group. Similar to the proficiency results, we see that the performance of students instructed by TFA teachers has tended to improve in more recent years. Those cells highlighted denote instances where TFA-instructed students performed similarly or better than students instructed by non-TFA or TFA-Comp teachers. Again, take note that the confidence intervals tend to be much wider for the TFA means, reflecting the smaller sample sizes the estimates are based upon.



The growth results presented above provide a slightly more accurate depiction of the TFA impact as measured differences in prior student performance are inherently taken into account when the growth scores are calculated. However, remaining measured differences may still exist between stu-dents assigned to the non-TFA, TFA-Comparison, or TFA teachers, classrooms or schools based on demographic background, absenteeism, mobility, etc. Analyses adjusting for other available, measured characteristics are presented next.

Multilevel Regression EstimatesFigure 11 below shows the estimated standardized effect and confidence intervals associated with instruction by a TFA teacher based on Mathematics EOG scores. Mathematics EOG standard devia-tions reported by NCDPI are approximately 8-9 scale score points, thus an effect size of .3 with a standard deviation of 9 would suggest a scale score difference of about 3 points (or 1/3 of a standard deviation). As seen below, there was a positive adjusted effect associated with instruction by a TFA teacher in 2008-09 and 2009-10, though only the 2009-10 effect was statistically significant (the entire 95% confidence interval does not contain zero). The point estimate for 2007-08 was negative, though we can see due to the smaller sample size the larger confidence interval, suggesting a wider range of possible values. Note for the 2009-10 effect, the 95% confidence interval shows that over repeated anal-yses with multiple samples, we can be 95% confident that the analysis would have yielded an estimate falling somewhere within the range of interval (.01 to .14).

Figure 11. Mathematics EOG TFA Estimate for 2007-08, 2008-09, and 2009-10.



Figure 12 below shows similar results based on Reading EOG scores. Positive effects were noted for 2008-09 and 2009-10, though only the 2008-09 estimate was statistically significant. The negative 2007-08 estimated was also statistically significant, again estimated with a wider confidence interval reflecting the smaller sample and greater uncertainty about the estimate.

Figure 12. Reading EOG TFA Estimate for 2007-08, 2008-09, and 2009-10.



Figure 13 below shows similar results based on Science EOG scores. Positive effects were noted across all three years, though only the 2008-09 estimate was statistically significant. The 2009-10 estimate nearly attained statistical significance, as the majority of the 95% confidence interval was above zero. Note the wider confidence intervals for science scores compared to reading or mathematics as these estimates are based only on data from 5th and 8th grade students.

Figure 13. Science EOG TFA Estimate for 2007-08, 2008-09, and 2009-10.



Figure 14 below shows the estimated TFA effect for a subset of EOC subjects. Examining the confi-dence intervals in this graph and noting the teacher counts presented in Table 2 above, we can see that we have wider confidence intervals for instances where there is less data available to base our estimate upon. Both Geometry and Algebra II in 2008-09 yielded significant positive effects, while the remain-ing estimates all crossed zero (suggesting non-significance).

Figure 14. Algebra I, II and Geometry TFA Estimates for 2007-08, 2008-09 and 2009-10.



Figure 15 below shows the estimated TFA effect for a second subset of EOC subjects (English, C & E, and Biology). All subjects yielded non-significant effects across all years, except for Biology which was significant in all three years. Note from Table 2, however, that these estimates are based only on 3, 6, and 10 teachers in 2007-08, 2008-09, and 2009-10, respectively.

Figure 15. English, C &E, and Biology TFA Estimates for 2007-08, 2008-09, and 2009-10.

Quantitative SummaryQuantitative analyses revealed generally positive results associated with TFA teachers. For purposes of informing policy, the multilevel analyses were conducted to determine whether any statistically sig-nificant differences occurred among comparison groups. In effect, this result would state that students instructed by TFA teachers are performing no better (or worse) than students instructed by tradition-ally trained teachers. Aside from the significant, negative effect found for reading in 2007-08, this is what was found. In a number of instances (math in 2009-10, reading, science, geometry and Algebra II in 2008-09, and in Biology across all three years) students instructed by TFA teachers significantly outperformed their peers instructed by traditionally trained teachers (controlling for student back-grounds, prior performance and contextual variables). More simplistic depictions of the impact of TFA teachers, based on proficiency and NC ABC growth values tended to show that students instructed by TFA teachers perform similarly to students instructed by traditionally trained teachers, aside from a few instances. Overall, it appears that TFA teachers are more likely to have a positive impact on student performance when instructing math and science courses, compared to reading or English. At the very least, little quantitative evidence generated here suggests that TFA teachers, due to their limited pedagogical training or experience, are impacting stu-dents in a negative way.



QUALITATIVEClassroom ObservationsDescriptive analyses were conducted on the basic information collected with the Observation Protocol form to determine whether the Comp-TFA and TFA classrooms observed were ‘similar’. Table 5 below displays basic descriptive information disaggregated by Comp-TFA and TFA subgroups. Included are the average number of students in the classrooms and ratings of the mean percent of students engaged throughout the observation period. It appears as though Comp-TFA and TFA classrooms were rela-tively equivalent with respect to these two measures.

Table 5. Summary of Descriptive Classroom Observation Information.

Comp-TFA TFAVariable N Mean Std Dev Min Max N Mean Std Dev Min MaxStudent Count 7 16.57 4.39 10 20 10 16.70 3.80 8 20Percent Engaged 7 89.39 8.48 76.92 100 10 89.87 13.26 69.44 100Control Rating 7 2.29 0.49 2 3 10 1.60 0.97 0 4% Whole Class 7 54.29 45.86 0 100 10 28.20 22.73 0 60% Pairs/Small Groups 7 28.57 48.80 0 100 10 55.90 24.57 20 100% Individual 7 17.14 29.42 0 80 9 17.67 20.35 0 50Differentiation Rating 5 2.80 0.84 2 4 8 3.50 1.07 2 5Extent of Differentiation Rating 6 2.67 1.63 1 5 10 3.00 1.63 1 5

The Control Rating summary provides the mean rating supplied by observers based on how much time was spent maintaining control of the class, where 0 equals “A Lot” and 3 equals ”Not at All”. Based on this scaling, we can see that the Comp-TFA teachers appeared to spend less time controlling the class-room than their TFA peers, based on observer ratings.The next three items provide information on the general types of instructional groups teachers used during the observations. Observers provided their estimation of the percent of time each type of structure was utilized during the observation period. We can see, on average that Comp-TFA teachers were engaged in Whole Class instruction about 54% of the time, while TFA teachers did so only 28% of the time. Conversely, only 28% of Comp-TFA teachers utilized paired or small group work while 55% of TFA teachers made use of this same strategy. Both subgroups made use of individualized instruction in equivalent proportions.The second to last entry in Table 5 is related to observer judgments about the types of students with whom teachers engaged in differentiated instruction. This entry (Differentiation Rating) was measured on a scale of 1 to 5, where 1 equals Low-Performing and 5 equals Advanced. We can see from the mean values that Comp-TFA teachers focused on providing differentiated instruction to lower-per-forming students while the mean for TFA teachers was slightly more to the advanced end of the scale. Note, however, that at least one TFA teacher was observed providing instruction to Advanced students (Max=5), which may influence the mean values given the small sample sizes. Also, a rating of ‘9’ was available to observers to reflect that students worked at their own level during the period. These values were removed from these analyses to accurately reflect ratings of differentiation.



The last entry in Table 5 is related to the extent to which differentiated instruction was witnessed during the classroom observation time. This entry was measured on a scale where 1 equaled “Not at All” and 5 equaled “Very Much So”. We can see from the mean values that little difference existed between Comp-TFA and TFA classrooms, with both providing moderate to slightly less than moderate levels of differentiated instruction during the observation periods.Summaries of the rating scales included as part of the observation protocol were generated by Comp-TFA and TFA groups. Readers should note the small sample sizes represented in these summary tables, limiting the generalizability of the ratings. Table 6 below displays the tabulated data with respect to the Lesson Design ratings supplied by the observers. Generally, the ratings with respect to Lesson Design were clustered in the positive end of the scale for both Comp-TFA and TFA teachers. The one trend worthy of note was the difference between Comp-TFA and TFA teachers with respect to whether the lesson encouraged collaborative learning. TFA teachers were rated more favorably for encouraging col-laborative learning than were their Comp-TFA counterparts.



Table 6. Summary of Lesson Design Ratings by Comp-TFA/TFA Subgroups.

Item/Teacher Type

A Great Extent 4 3 2 Not At All

Can’t Assess

n % n % n % n % n % n %

The lesson was organized around a big-picture, or main concept.

Comp-TFA 5 71.4 1 14.3 1 14.3 . . . . . .TFA 4 40 4 40 1 10 1 10 . . . .

The organization of the lesson engaged students.

Comp-TFA 3 42.9 3 42.9 1 14.3 . . . . . .TFA 3 30 6 60 . . 1 10 . . . .

The resources utilized in this lesson enhanced instruction.

Comp-TFA 1 14.3 4 57.1 1 14.3 . . 1 14.3 . .TFA 4 40 2 20 3 30 . . 1 10 . .

The lesson incor-porated hands-on, inquiry-based activities requir-ing application of knowledge.

Comp-TFA 2 28.6 3 42.9 . . . . 2 28.6 . .TFA 3 30 3 30 3 30 . . 1 10 . .

The lesson encour-aged collaborative learning.

Comp-TFA 1 14.3 1 14.3 1 14.3 1 14.3 3 42.9 . .TFA 3 30 6 60 1 10 . . . . . .

The instructional strategies/activities in this lesson are individualized to student needs.

Comp-TFA 1 14.3 1 14.3 2 28.6 1 14.3 1 14.3 1 14.3TFA 3 30 1 10 1 10 3 30 1 10 1 10

Table 7 provides a summary of the ratings associated with Lesson Content witnessed in both Comp-TFA and TFA classrooms. Ratings were generally more varied with respect to Lesson Content than they were in the Lesson Design table. Ratings appeared to be relatively equivalent across Comp-TFA and TFA teachers, though TFA teachers had more positive ratings regarding whether objectives were clearly identified at the beginning of each lesson.



Table 7. Summary of Content of Lesson Ratings by Comp-TFA/TFA Subgroups.

Item/Teacher Type


Can’t Assess

n % n % n % n % n % n %

Important ideas of the lesson intellectu-ally engaged students.

Comp-TFA 1 14.3 4 57.1 2 28.6 . . . . . .TFA 2 20 6 60 1 10 1 10 . . . .

Connections were made with other subject areas and/or real-world contexts.

Comp-TFA 2 28.6 2 28.6 . . 1 14.3 1 14.3 1 14.3TFA 3 30 1 10 2 20 3 30 . . 1 10

Objectives were clearly identified in the beginning of the lesson.

Comp-TFA 3 42.9 . . 2 28.6 . . . . 2 28.6TFA 6 60 1 10 . . 1 10 . . 2 20

Students dem-onstrated an understanding of the lesson.

Comp-TFA 2 28.6 3 42.9 1 14.3 1 14.3 . . . .TFA 4 40 1 10 5 50 . . . . . .

Students explained their answers/reason-ing to the class.

Comp-TFA 2 28.6 1 14.3 1 14.3 . . 2 28.6 1 14.3TFA 2 20 2 20 1 10 2 20 1 10 2 20

Table 8 provides a summary of the ratings associated with Lesson Implementation witnessed in both Comp-TFA and TFA classrooms. Generally, the ratings with regard to implementation were even across both Comp-TFA and TFA classrooms.



Table 8. Summary of Implementation of Lesson Ratings by Comp-TFA/TFA Subgroups.

Item/Teacher Type

A Great Extent 4 3 2

Not At All

Can’t Assess

n % n % n % n % n % n %

Teacher appeared confident in his/her ability to teach this lesson.

Comp-TFA 7 100 . . . . . . . .TFA 8 80 2 20 . . . . . .

The teacher’s classroom management style/strategies enhanced lesson quality.

Comp-TFA 3 42.9 2 28.6 2 28.6 . . . .TFA 5 50 2 20 1 10 1 10 1 10

The lesson pace was appropri-ate for the levels/needs of the students.

Comp-TFA 2 28.6 2 28.6 2 28.6 1 14.3 . .TFA 5 50 2 20 2 20 1 10 . .

The teacher was able to iden-tify student knowledge (i.e., ‘read’ the students’ level of understanding).

Comp-TFA 3 42.9 2 28.6 2 28.6 . . . .TFA 5 50 2 20 1 10 2 20 . .

The teacher’s instruction created a student-centered learning experience, pro-viding opportunities for questions and exploration of concepts.

Comp-TFA 4 57.1 . . 2 28.6 1 14.3 . .TFA 5 50 1 10 3 30 1 10 . .

Table 9 provides a summary of the ratings associated with Teacher-Student Interactions. Generally, the ratings with regard to interactions were evenly distributed across both Comp-TFA and TFA class-rooms. TFA teachers tended to have more ratings on the lower end of the scale with respect to whether talk is centered around learning versus controlling behavior. In addition, TFA teachers also were assigned slightly more negative ratings about their ability to respond to non-verbal cues of confusion or boredom.



Table 9. Summary of Teacher-Student Interactions Ratings by Comp-TFA/TFA Subgroups.

Item/Teacher Type

A Great Extent 4 3 2 Not At AllCan’t

Assess

n % n % n % n % n % n %

There is a balance of give and take discus-sion between student and teacher.

Comp-TFA 4 57.1 1 14.3 . . 1 14.3 . . 1 14.3TFA 5 50 3 30 1 10 1 10 . . . .

Talk is centered on what students are learning rather than on controlling behavior.

Comp-TFA 2 28.6 3 42.9 . . 1 14.3 . . 1 14.3TFA 2 20 4 40 1 10 3 30 . . . .

There is a conver-sational tone to interactions in the classroom.

Comp-TFA 4 57.1 . . 2 28.6 . . . . 1 14.3TFA 5 50 2 20 1 10 2 20 . . . .

The timing for teaching points is appropriate (e.g., teacher does not inter-rupt learning)

Comp-TFA 3 42.9 2 28.6 1 14.3 . . . . 1 14.3TFA 5 50 2 20 2 20 1 10 . . . .

Teacher responds to non-verbal cues of confusion, boredom, and curiosity.

Comp-TFA 4 57.1 . . 1 14.3 1 14.3 . . 1 14.3TFA 2 20 3 30 2 20 2 20 . . 1 10

Students probe the teacher for more in-depth knowledge about concepts.

Comp-TFA 2 28.6 1 14.3 1 14.3 1 14.3 1 14.3 1 14.3TFA 3 30 3 30 1 10 1 10 1 10 1 10

Students answer and/or explain questions asked by teacher.

Comp-TFA 5 71.4 . . . . 1 14.3 . . 1 14.3TFA 4 40 4 40 1 10 1 10 . . . .

Table 10 provides a summary of the ratings associated with Classroom Culture. Generally, the ratings with regard to culture in the classroom were evenly distributed across both Comp-TFA and TFA class-rooms. There was some evidence to suggest that Comp-TFA teachers were less likely to incorporate collaborative opportunities for students. With respect to whether the classroom environment was dis-ruptive, TFA teachers tended to have more ratings on the upper end of the scale, suggesting this may have been a problem.



Table 10. Summary of Classroom Culture Ratings by Comp-TFA/TFA Subgroups.

Item/Teacher Type


Can’t Assess

n % n % n % n % n % n %

Active participa-tion of ALL students was encouraged and valued.

Comp-TFA 4 57.1 2 28.6 . . . . 1 14.3 . .TFA 4 40 3 30 1 10 2 20 . . . .

There was a climate of respect for students’ ideas, questions, and contributions to the lesson.

Comp-TFA 4 57.1 2 28.6 . . 1 14.3 . . . .TFA 4 40 6 60 . . . . . . . .

Interactions reflected collaborative working relationships among students.

Comp-TFA 2 28.6 . . 1 14.3 1 14.3 2 28.6 1 14.3TFA 2 20 6 60 1 10 1 10 . . . .

The lesson encouraged students to gener-ate ideas, questions, conjectures, and/or propositions.

Comp-TFA 2 28.6 1 14.3 3 42.9 . . 1 14.3 . .TFA 4 40 3 30 3 30 . . . . . .

Intellectual rigor, constructive criticism, and the challenging of ideas were evident.

Comp-TFA 3 42.9 1 14.3 2 28.6 1 14.3 . . . .TFA 3 30 3 30 2 20 2 20 . . . .

Classroom environ-ment was disruptive.

Comp-TFA . . . . 3 42.9 1 14.3 3 42.9 . .TFA 1 10 2 20 2 20 1 10 4 40 . .

In general, students show a high level of interest.

Comp-TFA 3 42.9 2 28.6 1 14.3 1 14.3 . . . .TFA 2 20 4 40 4 40 . . . . . .

Table 11 provides a summary of the ratings associated with the Likely Impact of the Lessons. Generally, the ratings with regard to lesson impact were equivalent across Comp-TFA and TFA teacher observations.



Table 11. Summary of Likely Impact of Lesson Ratings by Comp-TFA/TFA Subgroups.

Item/Teacher Type

Positive Effect 4

No Net Effect/

Mixed Effect 2Negative

EffectCan’t

Assess

n % n % n % n % n % n %

Students’ under-standing of important concepts.

Comp-TFA 2 28.6 4 57.1 1 14.3 . . . . . .TFA 2 25 4 50 2 25 . . . . . .

Students’ ability to apply or generalize skills and concepts to other areas and/or real-life situations.

Comp-TFA 2 28.6 2 28.6 1 14.3 . . 2 28.6TFA 1 11.1 5 55.6 1 11.1 . . 2 22.2

Students’ self-confi-dence in doing work in the subject area.

Comp-TFA 2 28.6 4 57.1 . . 1 14.3 . .TFA 2 25 3 37.5 2 25 1 12.5 . .

Students’ engage-ment during class.

Comp-TFA 2 28.6 3 42.9 2 28.6 . . . .TFA 3 37.5 1 12.5 4 50 . . . .

Students’ interest in and/or appreciation for the discipline.

Comp-TFA . . 4 57.1 1 14.3 1 14.3 1 14.3TFA 1 12.5 2 25 2 25 2 25 1 12.5

Finally, Table 12 shows a summary of the Overall Lesson Quality ratings. Again, the ratings for both Comp-TFA and TFA teachers appeared to be roughly equivalent.

Table 12. Summary of Overall Lesson Quality Ratings by Comp-TFA/TFA Subgroups.

Item/Teacher Type

Exemplary Instruction

4 3 2 Ineffective Instruction

n % n % n % n % n %

Overall Lesson Quality

Comp-TFA 3 42.9 2 28.6 1 14.3 1 14.3 TFA 3 30 3 30 4 40 . .

Observers were also asked to report on the types of instructional activities and tasks occurring in the classroom in five-minute increments during their observation. Table 13 below shows a summary of the classroom tasks disaggregated by Comp-TFA and TFA classrooms. Based on the data collected, TFA teachers tended to use independent practice more often in their classrooms, while Comp-TFA teachers were observed using cooperative learning more often. The differences observed in the other categories were only minimally different across all classrooms observed, suggesting both groups engage in the various activities at a similar rate.



Table 13. Summary of Instruction/Tasks by Comp-TFA/TFA Subgroups.

Type of Instruction/Tasks

Comp-TFA

TFA

n % n %

L-lecture/presentation 8 8.8 13 9.8PM-teacher problem modeling 0 0.0 3 2.3SP-student presentation (formal) 0 0.0 0 0.0LWD-lecture with discussion 9 9.9 6 4.5D-student demonstration 0 0.0 1 0.8CD-class discussion 7 7.7 8 6.1IP-independent practice 4 4.4 20 15.2R-reading 7 7.7 13 9.8INV-investigative activity/manipulatives 6 6.6 2 1.5SGD-small group discussion 12 13.2 16 12.1MG-games/activities not investigative 0 0.0 4 3.0CL-cooperative learning 12 13.2 0 0.0LC-learning center/station 6 6.6 12 9.1TIS-teacher/faculty interacting with student 13 14.3 19 14.4UT-utilizing technology 2 2.2 6 4.5A-assessment 0 0.0 2 1.5AD-administrative tasks 0 0.0 1 0.8OOC-out of class experience 0 0.0 0 0.0I-interruption 5 5.5 6 4.5Total 91 100.0 132 100.0

Principal InterviewsContent analysis was conducted on the data collected by observers during the principal interviews to identify themes or trends in the responses provided by principals. An attempt was made to informally quantify themes or phrases to get a feel for the frequency with which interviewed principals expressed common ideas or perceptions. Summaries for each item included on the Principal Interview Guide are presented below.Overall, how has the Teach for America program impacted your school?In general, principals expressed positive sentiments regarding their TFA experience. Of the six principal interviews conducted, three mentioned that TFA teachers had a positive impact on their school. Three principals also mentioned that TFA teachers arrived at their school data-driven and eager to analyze and track student performance. A few principals remarked that TFA teachers were well-trained and that the professional development they received was excellent preparation for an urban, low-income setting. Principals also felt that TFA teachers were highly motivated, solid professionals who take the success of their students personally. One principal in particular noted that TFA teachers tend to be competitive, which they felt was a good influence on the typical non-competitive school culture. Still



another principal specifically mentioned that TFA teachers’ classroom management skills were better than most lateral entry teachers they had encountered.Are there any suggestions for ways your involvement with TFA could be improved?The feedback provided by principals regarding this question generated a wide array of responses. One principal commented on the natural tension they have seen between the traditional and TFA teachers, which is exacerbated by the generational (age) difference as well. Often, traditional teachers perceive the confidence of TFA teachers as arrogance. This principal felt that there needs to be better preparation for both of these groups to create a collegial atmosphere where both can learn from the experiences of the other. Another principal recommended that TFA recruits have the opportunity to be immersed in the urban school setting prior to being placed in the classroom. This principal remarked that many TFA teachers are ‘shocked’ the first day, which undermines their ability from the start. Another prin-cipal requested the ability to provide input about specific TFA teachers assigned to their location, and would like to provide TFA with insight about how to better prepare recruits with respect to particular areas, such as classroom management. Also, a principal was hopeful that better communication could be achieved between themselves and the local TFA supervisor. This principal felt out-of-touch with what the TFA organization was asking of their TFA teachers, and felt it would better serve everyone if they were all on the same page. Finally, a single principal wished that TFA would share their recruiting strategies and professional development practices with them.What has been the most positive aspect of your involvement with TFA thus far?One principal commented that TFA teachers are willing to listen to feedback and are excited about taking on tasks. In addition, the principal remarked that TFA teachers work well with their colleagues and other team members. Another principal felt as though TFA teachers bring a new perspective for teaching and learning and view their role in the schools through a different lens than the more tradi-tional teachers. In addition, this same principal stated that TFA teachers arrive with no pre-conceptions about student backgrounds and their impact on student potential for learning. Yet another principal remarked that, “(TFA) brings fresh teachers without a canvas that needs to be repainted”. Yet another principal mentioned TFA teachers’ ability to create learning-based activities in the classroom and their general excitement to be in the classroom teaching. Finally, a principal commented that TFA teachers bring a great deal of confidence and preparedness to the classroom. This principal felt that, despite their lack of formal coursework in education, TFA teachers have enough exposure to the practical aspects of teaching they need for a firm base from which to build.Overall, how satisfied are you with the Teach for America teachers assigned to your school?In general, principals are satisfied with TFA teachers assigned to their schools and feel as though TFA teachers arrive prepared to take on any obstacle standing in the way of their reaching students. Some principals noted that there is a certain amount of variance between individual TFA teachers, with some performing ‘about average’ while others are ‘performing above-average’. One concern of principals that did arise was their lack of perceived control over the placement of TFA teachers at their location. One principal noted that one of the TFA teachers they had ranked as the lowest with regard to fit for their school was assigned there anyway (and is subsequently going to be dismissed). Yet another concern expressed by principals was the relatively short commitment period of two years. Some principals are concerned that the large sum of money spent on both formal and informal professional development is a large expense when they are only committed to the profession for two years.



Overall, how satisfied are you with the ability of the Teach for America teachers with respect to Leadership Skills/Classroom Management Practices?Principals reported that TFA teachers tend to have strong leadership skills, take on active leadership roles, take risks in their classroom strategies, and provide service to students beyond traditional school hours in various forms. Principals did note, however, that TFA teachers do tend to struggle with the classroom management aspect and often cite a lack of clear and consistent training in this regard. However, one principal remarked that they did not see any difference in preparedness for classroom management between TFA and traditional teachers in their first year. Another principal mentioned that TFA teachers often need a bit of guidance and ‘pep-talking’ to keep them going and another princi-pal suggested that ‘TFA teachers need to develop their ability to appreciate the adolescent mind outside of their own experiences.’ Finally, one principal felt as though sometimes the TFA teachers are a bit sensitive and breakdown when they are not able to reach the goals they have set for themselves.Overall, how satisfied are you with the ability of the Teach for America teachers with respect to Curriculum/Subject Proficiency?Principals were generally satisfied with the basic level of curriculum/subject proficiency exhibited by TFA teachers. However, several principals did express some concern in particular areas such as align-ment and lesson design and one principal in particular remarked that TFA teachers are slow to develop in this area. One principal mentioned their perception that TFA teachers work harder to familiarize themselves with the curriculum, and that even some of the most experienced teachers in their school struggle with curriculum in certain areas such as alignment. Two principals specifically mentioned that the success of TFA teachers with regard to curriculum is directly related to their ability to forge relationships with traditional teachers. Those who have made good contacts through group or team planning have learned from their colleagues and made greater strides than teachers who have not developed relationships. One principal noted that many TFA teachers appear to have more creativ-ity with respect to lesson development and learning activities; and that TFA teachers seem to be at an advantage compared to traditional teachers as they begin their lesson planning over the summer and tend to be ‘ahead’ of their colleagues throughout the year.Overall, how satisfied are you with the ability of the Teach for America teachers with respect to Facilitation of Student Learning/ability to engage students?Principals expressed high levels of satisfaction with TFA teachers in regard to their ability to facilitate learning and engage students. TFA teachers tend to arrive with fresh ideas and open minds about how to function in the classroom. A few principals thought that the level of ability to engage students varied from teacher to teacher and was impacted by classroom management practices, but did note that all TFA teachers are accepting of feedback and are willing to make changes with their planning and deliv-ery. One principal in particular noted that TFA teachers at their school were instrumental in helping a student get a scholarship. These TFA teachers worked together to get the student a suit and worked with the student in preparation for an interview.

Teacher InterviewsIn general, how satisfied are you with your experiences in the Teach for America program?All but one TFA teacher expressed general satisfaction with their experience in the TFA program, though the lone holdout did not elaborate on why they were not satisfied. Several teachers mentioned



their appreciation for the preparation TFA provided through training and professional development. In fact, one TFA teacher felt as though they were more prepared for the classroom than some lateral entry teachers at their location. One TFA teacher placed in an English as a Second Language (ESL) environment felt that her initial TFA training was lacking in ESL preparation, but noted that a mid-year correction by TFA had since accounted for that discrepancy. Several other TFA teachers specifically mentioned that faculty and staff at their locations was particularly supportive of them and felt wel-comed at their schools. One teacher, who did express overall satisfaction with the TFA program, was not prepared for the level of attendance and behavioral issues at their location. Specifically, this TFA teacher did not realize the frequency with which students would be absent or that 2nd or 3rd time behav-ioral referrals really carried no weight in curbing student behavior.Are there any suggestions for ways your TFA experience could be improved?When asked whether there were ways to improve the TFA experience for recruits, TFA teachers offered a number of suggestions. TFA teachers encouraged the TFA organization to continue the teacher support through professional development and suggested that TFA find a way to target professional development to the specific needs of each teacher. Along the same lines with regard to preparation, TFA teachers suggested providing new TFA teachers with a few pre-planned lessons to begin the school year and also suggested that Sheltered Instruction Observation Protocol (SIOP) training occur before the academic year as opposed to during the academic year. Further, several TFA teachers mentioned more in-depth training around the North Carolina Standard Course of Study (NCSCOS) standards and how to make use of the standards for lesson planning and instructional purposes. One TFA teacher also suggested that recruits be provided with more time in the classroom prior to taking on their actual teaching assignment. Another TFA teacher said the relationship between TFA and the University of North Carolina at Charlotte was unclear. Two TFA teachers suggested that more encouragement and support from local program directors would be beneficial and commensurate with the level of support they had received from school-based personnel. A single TFA teacher mentioned that both fellow TFA teachers and Comp-TFA teachers need to recognize the intrinsic reward that goes along with teaching and helping students, regardless of whether growth in the form of test scores is ever realized.What has been the most positive aspect of your TFA experience thus far?Several of the TFA teachers stated that just the opportunity to become a teacher in a classroom was beneficial to them. Several specific teachers mentioned the satisfaction they got from seeing the tools they learned in their pre-training come to life in the classroom in a way that positively impacts stu-dents. One TFA teacher mentioned her specific opportunity to remain with her students two years in a row, and to be able to watch their growth over that entire time period. Yet another commented about the TFA-based training around data-driven instruction, noting that they feel comfortable talking to their students about data and have even been sought out by more seasoned teachers to learn about how to make better use of data for instructional purposes. Two TFA teachers mentioned the support they have received from the school-based personnel and the collegiality and collaboration they have experienced in working with Comp-TFA teachers. Finally, one TFA teacher specifically mentioned the success stories she has created by impacting individual students and helping them to achieve. In general, how satisfied are you with the school you’ve been assigned to?The majority of teachers expressed general or extreme satisfaction with the schools they have been



assigned to. Most TFA teachers mentioned their affection and respect for the children attending their assigned schools, with a few noting this affection despite the difficulty of educating the students. A few teachers mentioned that they have been treated fairly and respected by their colleagues at their loca-tion, have been acknowledged as a valuable team member and been led by excellent mentor teachers into other areas such as community involvement. Several of the teachers happened to be assigned to schools that had recently undergone a principal change. One TFA teacher commented that they felt supported under the administration of both principals, even though each had very distinctive leader-ship styles. Two of the TFA teachers who had experienced a principal change mentioned that the time period during the change was ”chaotic”, creating some obstacles that made teaching somewhat difficult; but no specific examples of difficulties encountered were offered. How satisfied are you with the level of support you have received from the staff at your school?In general, TFA teachers expressed high levels of satisfaction with the support provided by staff at their schools. Several teachers mentioned that their school had a structured system in place to help support new teachers and they found that system beneficial to their adjustment in the classroom. One TFA teacher specifically mentioned the level of support provided to the Algebra team at her school and another mentioned his/her planning team as resources providing high levels of support while their skills develop. Another teacher specifically mentioned that the North Carolina New Schools project had also been a good resource and system of support. Two TFA teachers again specifically mentioned their mentor teachers as great resources and patient individuals helping them figure out how to become their own teachers. One TFA teacher offered that it has been easier to find and access support in their second year because they are more aware of what their needs are in order to develop. Another teacher mentioned that observations conducted by the assistant principal have been extremely constructive and helpful from an instructional development point of view. A TFA teacher that has experienced a principal change remarked that they are more satisfied now, as the new principal has made their pro-fessional development strategies more inclusive so everyone is involved. Only one teacher expressed a somewhat negative sentiment, suggesting that there were support issues regarding classroom manage-ment, but failed to elaborate any further. How satisfied are you with the level of support you have received from the local TFA contact?Again, TFA teachers generally expressed satisfaction with the support they received from their local TFA contacts. Several teachers mentioned that their TFA contact is available when needed, but yet is distant enough to not appear overbearing. One TFA teacher specifically mentioned their positive expe-rience with the local Program Director who has been open to hearing concerns expressed by teachers. This same Program Director has also been helpful in problem-solving, always redirecting issues back to how they potentially impact student learning, helping the teacher to maintain focus on what is “most important”.Several teachers mentioned that they recognize that their local TFA contact is available for support, but they rarely access this resource. One TFA teacher stated that they are comfortable with their classroom management skills and they acquire their content support from staff in the schools. Another teacher suggested that as a second-year TFA teacher, they now are more targeted in seeking their support and know who to go to for filling certain needs, so there is less reliance for guidance from the local contact. Finally, one TFA teacher was generally satisfied with the level of TFA support; but commented that the



support from her mentor teacher is much more important as that person is available and present on a daily basis. How satisfied are you with your ability in the classroom in the following areas? – Leadership/Classroom ManagementIn general, TFA teachers are confident and satisfied with their abilities to be leaders within their schools and classrooms, with several mentioning skills and abilities from previous experiences that have enabled them to take on leadership roles. Several TFA teachers did acknowledge that their classroom manage-ment skills are an area in need of improvement, but also followed up that they actively incorporate feedback from administrators and veteran teachers regarding management strategies. An additional TFA teacher mentioned taking advantage of every additional professional development opportunity available to bolster their skills and abilities in the classroom. Finally, one TFA teacher mentioned that the Teaching Education Leadership Framework provides an entire rubric system to help them develop their leadership abilities and has found that to be helpful.How satisfied are you with your ability in the classroom in the following areas? – Curriculum/Subject ProficiencyMany of the TFA teachers sampled for interviews reported high levels of comfort in the areas they were assigned to teach, reporting undergraduate degrees in the same or closely related fields. One TFA teacher did report putting in some extra work to ensure their coverage of content lined up with the expectations for the International Baccalaureate (IB) program, while another mentioned having to develop knowledge around ESL strategies. A teacher placed out of their subject area mentioned that the adjustment was more difficult because of the lack of comfort, but that the support and assistance pro-vided by faculty at their school helped tremendously. One TFA teacher specifically mentioned that the professional development provided by TFA gave them a high level of comfort with their ability to plan and execute a lesson and ensure that students are mastering content. Another teacher also reported a high level of comfort in their subject area, but also mentioned that they partner with colleagues to help with EOG review sessions in areas outside of their expertise. They specifically mentioned that they are not intimidated by their lack of comfort and view it as a growth opportunity for themselves and to show their students that they do not have to know everything. How satisfied are you with your ability in the classroom in the following areas? – Facilitation of Student Learning/Ability to Engage StudentsThe majority of the TFA teachers reported relatively high levels of ability to engage students. Some specifically mentioned that their ability to stimulate students is a growth opportunity for them. Some teachers recognized that students have a limit to their attention span and that each student learns in different ways, making the ability to reach every student all the time nearly impossible.TFA teachers reported several different strategies they use to engage students, including the incorpora-tion of lab work and other hands-on activities, providing stimulating or challenging topic areas, and making use of multiple methods of inquiry. Another teacher reported attempts to develop a vested interest in learning in their students by making use of the Socratic Method to tap students’ passion for a topic. Finally, one TFA teacher incorporates SIOP strategies to reduce the amount of talking they do while increasing the amount of talking on the part of students. This teacher has found that student participation has helped to keep students engaged during class time.



One TFA teacher reported not feeling very good about their ability to motivate students, rating them-selves a 6 or 7 out of 10. This particular teacher specifically mentioned the difficulty in reaching high school age students who are repeaters. This teacher stated that attempts to build relationships with students have failed thus far.Do you plan to continue teaching (or remain in education) after your Teach for America two-year com-mitment ends?All of the TFA teachers interviewed mentioned a desire to stay in education in one form or another, with several planning on staying in the classroom. Several TFA teachers have been accepted into CMS professional development programs including CMS Leaders for Tomorrow and Aspiring Leaders. Another TFA teacher has plans to work for TFA as a recruiter, and another is pursuing a master’s degree in education policy with the intent of exploring broader issues such as education accountability, pay-for-performance, and charter versus public schools.One TFA teacher plans to continue teaching, but would prefer to teach Biology, Chemistry, and Physics in a different school. Two TFA teachers mentioned their desire to attain their teaching certificates, but reported struggling to do so in the two-year timeframe. Specifically, one teacher stated obtaining a certificate was going to take longer as they needed to take credits in the subject area they are teaching. Still another mentioned that the partnership between TFA and UNCC had not been set up to allow completion of the program within two years. Subsequently, the teacher reported that this has changed, but not in time to impact them.

Qualitative SummaryA review of descriptive data for Comp-TFA and TFA classrooms suggested comparable classroom compositions. Generally, both Comp-TFA and TFA teachers had the same number of students in their rooms and were observed to attain similar levels of student engagement. One noted difference, however, was the division of instructional activities. Comp-TFA teachers tended to engage in whole-class instruction more often, whereas TFA teachers were observed to use pairs/small group instruction and independent practice activities in their lessons. This was confirmed through Lesson Design ratings collected by observers, where TFA teachers were found to more actively promote collaborative learning in comparison to their Comp-TFA counterparts. Another notable difference was the greater propensity for TFA teachers to clearly state lesson objectives at the outset of a class. Overall, with regard to lesson impact and quality, no notable differences were observed between TFA and Comp-TFA teachers.Not surprisingly, TFA teachers were rated more negatively than Comp-TFA teachers with respect the amount of talk focused on controlling behavior, their ability to respond to non-verbal cues of confusion or boredom, and the increased level of disruptiveness. This finding was confirmed by the TFA teachers themselves in the interviews, as most openly acknowledged their lack of preparation for dealing with classroom behavior issues.Principals reported high levels of satisfaction with TFA teachers placed at their schools, commenting that TFA teachers were highly professional, motivated, and data-driven teachers who were eager to learn and develop their classroom skills. TFA teachers are thought to have great potential for classroom and school leadership, and have shown the willingness and dedication to work hard at overcoming deficits they possess with respect to curriculum mastery or classroom management abilities. Often,



principals noted, TFA teachers are going the extra mile to develop themselves, improve the schools, and help students achieve at a high level. Principals provided suggestions for improving the TFA experience, including better direct communica-tion with the TFA organization and more classroom experience for TFA recruits before being formally assigned their own students; a sentiment echoed by some TFA teachers themselves. Principals also mentioned the development of strategies for increasing the collaboration between TFA and Comp-TFA teachers, to help overcome the natural tensions that arise between these two groups. Overall, principals reported a great deal of satisfaction with the TFA teachers in their schools and viewed the TFA program as a valuable resource for school staffing.Generally, most TFA teachers are satisfied with their experience in the TFA program. Several teachers mentioned struggles unique to their own situations or schools, yet most expressed desires to remain in education in one capacity or another. Most TFA teachers reported a welcoming setting when arriv-ing at their schools, complete with supportive leadership and colleagues willing to help them adjust to classroom life. One or two TFA teachers reported a difficult culture under prior leadership, but subsequently mentioned that a recent principal change injected a more healthy culture into the school environment.TFA teachers self-reported generally high levels of ability to engage students and facilitate learning. They also expressed a great deal of comfort with the subject material they are asked to teach. While several teachers openly acknowledged the need for developing their classroom management skills, they viewed this as an opportunity to capitalize on the support from fellow teachers and leaders in their school. Overall, TFA teachers expressed a strong desire to remain involved in education, with about half of those interviewed expressing a specific interest in extending their classroom teaching beyond their two-year commitment.



DISCUSSIONThe dual data collection and analytic strategies used allowed for both formative and summative infor-mation to be collected simultaneously. The results provide summative information in the sense that there are some quantifiable impacts of TFA teachers on student achievement. Formative information is available in the form of the perceptions and feedback shared by principals and TFA teachers regarding their experiences with the TFA program. Their insights and suggestions can be used to improve the collaboration between CMS and TFA.The quantitative results revealed that students instructed by TFA teachers, when compared to students of traditionally trained teachers in schools where TFA teachers are assigned, tend to perform just as well if not better on standardized assessments. In some instances, TFA teachers actually outperform all other teachers, including those assigned to schools where there are no TFA teachers. When more complicated analyses were conducted to take into account a host of background characteristics and prior performance measures at the student, class, teacher and school level, estimates for the impact of TFA teachers yielded some positive effects (particularly in math and science), varying by subject and year. Only a single significant, negative impact was noted (reading achievement results in 2007-08). CMS wants to ensure that the lack of formal training and experience as teachers, a common criticism mentioned in the literature, does not negatively impact student achievement. The quantitative results found here do not seem to suggest a global, adverse impact on student achievement. The qualitative work provided information from principals and TFA teachers about the TFA program. It also provided classroom observations and ratings of both TFA and Comp-TFA teachers. Generally, ratings and observation perceptions were similar across the two teacher types, though some differences are worth noting. Comp-TFA teachers tended to favor whole group instruction, while TFA teachers observed tended to favor paired and small group instructional formats. This coincided with lesson design ratings revealing that TFA teachers more actively promoted collaborative learning despite con-tradictory evidence of collaborative learning in instructional summary counts (see Table 13). One area in which TFA teachers were observed to struggle was classroom management, a finding also docu-mented in the research literature and cited as a criticism of the TFA program. Many TFA teachers interviewed openly acknowledged this difficulty, stating they lacked adequate preparation for dealing with classroom behavioral issues.Principals generally reported high levels of satisfaction with TFA teachers place at their school, citing high levels of professionalism, motivation and a willingness to “go the extra mile”. That said, principals did make suggestions for improving the experience further, including more classroom experience for TFA recruits before they are assigned their own students and the development of strategies to increased collaboration between TFA teachers and their traditionally trained peers. Most TFA teachers reported high levels of satisfaction with their placement in the schools, reporting welcoming, collaborative facul-ties willing to guide them through their adjustment period. Further, TFA teachers reported high levels of ability to engage students, familiarized themselves with the curriculum, and facilitate learning. Many TFA teachers expressed an interest in remaining in education beyond their two-year commitment.



Overall, the results presented here suggest using TFA as a staffing strategy does not present any adverse effects on student achievement or the school environment in which they are assigned. In some instances, TFA teachers are assuming leadership roles within their schools and working directly with students to help them achieve both in and out of the classroom. Thus, the average TFA teacher appears capable of assimilating themselves into their schools and functioning much like their traditionally-trained peers.



REFERENCESAzimi, N. (2007). Why Teach For America? New York Times Magazine. September 30, 2007, p. 111.Boyd, D., Grossman, P., Lankford, H., Loeb, S., Michelli, N., & Wyckoff, J. (2006). Complex by design. Investigation pathways into teaching in New York City Schools. Journal of Teacher Education, 57, 2, 155-166.Darling-Hammond, L., Holtzman, D., Gatlin, S., & Heilig, J. (2005). Does Teacher Preparation Matter? Evidence about teacher certification, Teach for America, and Teacher Effectiveness. Education Policy Analysis Archives, 13, 42.Decker, P., Mayer, D., & Glazerman, S. (2004). The effects of Teach for America on students: Findings from a national evaluation. Princeton NJ: Mathematica Policy Research.Freiberg, H. J. (1999). School Climate: Measuring, Improving and Sustaining Healthy Learning Environments. Philadelphia, PA: Falmer Press.Helms, A. (2009, May 13). Rookie teachers will bump some CMS veterans. Charlotte Observer. Retrieved from the World Wide Web on July 29, 2009 from: Charlotteobserver.com. Houtenville, A. (2008). Parental effort, school resources, and student achievement. Journal of Human Resources, 43, 2, 437-453.Jeynes, W. H. (2005). A meta-analysis of the relation of parental involvement to urban elemen-tary school student academic achievement. Urban Education, 4, 3, 237-69. http://www.gse.harvard.edu/hfrp/publications_resources/publications_series/family_involvement_research_digests/parental_involvement_and_student_achievement_a_meta_analysis.Kane, T., Rockoff, J. & Staiger, D. (2006). What does certification tell us about teacher effectiveness? Evidence from New York City. Working Paper No. 12155, National Bureau of Economic Research. April.Lee, V. & Smith, J. (1999). Social support and achievement for young adolescents in Chicago: The role of social academic press. American Educational Research Journal, 36, 4, 907-945.McCaffrey, D., Lockwood, J. Koretz, D. & Hamilton, L. (2004). Evaluating Value-Added Models for Teacher Accountability. RAND report for the Carnegie Corporation.McCaffrey, D.F., Lockwood, J., Koretz, D., Louis, T.A., & Hamilton, L. (2004). Models for value-added modeling of teacher effects. Journal of Educational and Behavioral Statistics, 29, 67-101.Mendro, R., Jordan, H., Gomez, E., Anderson, M., & Bembry, K. (1998). An application of multiple linear regression in determining longitudinal teacher effectiveness. Paper presented at the 1998 Annual Meeting of the AERA, San Diego, CA.National Center for Alternative Certification. (2008). Response to recent Linda Darling-Hammond study: Letter from Abigail Smith, Vice President of Research and Policy. Retrieved from the World Wide Web on July 29, 2009 from: http://www.teachnow.org/newsdisp.cfm?newsid=77.National Comprehensive Center for Teacher Quality & Public Agenda (NCCTQ). (2007). Lessons learned: New teachers talk about their jobs, challenges and long-range plans. Report prepared by the National Comprehensive Center for Teacher Quality & Public Agenda.



North Carolina Department of Public Instruction ABC’s Technical Documentation. (2009). Retrieved from the World Wide Web on July 27, 2009 from: http://abcs.ncpublicschools.org/abcs/.Onwuegbuzie, A. J., & Leech, N. L. (2005). On becoming a pragmatic researcher: The importance of combining quantitative and qualitative research methodologies. International Journal of Social Research Methodology, 8, 5, 375-387.Raymond, M. Fletcher, S., & Luque, J. (2001). Teach for America: An evaluation of teacher differences and student outcomes in Houston, Texas. Stanford, CA: The Hoover Institution.Rivkin, S.G., Hanushek, E.A., & Kain, J.F. (2000). Teachers, Schools, and Academic Achievement. Cambridge, MA: National Bureau of Economic Research, NBER Working Paper # W6691.Rossi, P. H., Lipsey, M.W., & Freeman, H.E. (2004). Evaluation research: A systematic approach. Thousand Oaks, CA: Sage Publications. Rowan, B., Correnti, R., & Miller, R.J. (2002). What large-scale survey research tells us about teacher effects on student achievement: Insights from the Prospects study of elementary schools. Teachers College Record, 104, 1525–1567.Sanders, W. & Horn, S. (1998). Research findings from the Tennessee value-added assessment system (TVAAS) database: Implications for educational evaluation and research. Journal of Personnel Evaluation in Education, 12(3), 247–256.Sanders, W.L. & Rivers, J.C. (1996). Cumulative and residual effects of teachers on future student aca-demic achievement. Knoxville, TN: University of Tennessee Value-Added Research Center.Schoeneberger, J., Dever, K. & Tingle, L. (2009). Evaluation of Teacher for American in Charlotte-Mecklenburg Schools. Center for Research & Evaluation report prepared for the Charlotte-Mecklenburg Schools Board of Education.Sherblom, S., Marshal, J.C. & Sherblom, J. (2006). The relationship between school climate and math and reading achievement. Journal of Research in Character Education, 4, No. 1 & 2, 19-31.Shkolnik, J., Hikawa, H., Suttorp, M., Lockwood, J.R., Stecher, B., & Bohrnstedt, G. (2002). Appendix D: The Relationship between teacher characteristics and student achievement in reduced-size classes: A study of 6 California districts. In G. Bohrnstedt & B. Stecher (Eds.), What We Have Learned About Class Size Reduction in California: Technical Appendix. Sacramento, CA: California Department of Education, D1–D22.Veltri, B. T. (2008). Teaching or service?: Site-based realities of Teach for America Teachers in Poor, Urban Schools. Education & Urban Society, 40, 5, 511-542.Walberg, H. (1984). Families as Partners in Educational Productivity, Phi Delta Kappan, 65, 397-400.Webster, W. & Mendro, R. (1997). The Dallas value-added accountability system. In J. Millman (Ed.), Grading Teachers, Grading Schools: Is Student Achievement a Valid Evaluation Measure? Thousand Oaks, CA: Corwin Press, Inc., pp. 81–99.Wright, S., S. Horn, and W. Sanders (1997). Teacher and Classroom Context Effects on Student Achievement: Implications for Teacher Evaluation. Journal of Personnel Evaluation in Education, 11, 57-67.Xu, Z., Hannaway, J., & Taylor, C. (2009). Making a difference?: The effects of Teach forAmerica in high school. Durham, NC: The Urban Institute and CALDER.



APPENDIX A DRAFT

Appendix A

Charlotte-Mecklenburg Schools

Center for Research & Evaluation

Teach for America Project

Classroom Observation Tool

Background Information

Teacher Name: ____________________________

School: __________________________________

Grade/Subject:____________________________

Date: ____________________________________

Observer: ________________________________

Beginning/end time of observation: _______/_______

Length of observation: ____________ (minutes)

Scheduled length of class: ____________ (minutes)

Total number of students in class: ________

Was a Teaching Assistant or co-teacher present during the observation? Yes No

Lesson Information

Description: In a few sentences, describe the lesson observed. Include details such as the

concepts covered, skills focused on, etc. Was the lesson an introduction to a new topic, in the

middle of a sequence, or a review lesson? Were hands-on activities or demonstrations

incorporated into the lesson.


Office of Accountability | Center for Research and EvaluationDRAFT

Lesson Checklist

Please fill in the types of instruction/activities, student engagement, and cognitive activity used

in each 5-minute portion of this class in the boxes below. Observe (and write notes) for 5

minutes and then at the end of the interval, complete this checklist. Continue this process until

the observation has ended.

Note: there may be one or more strategies used in each category during each interval. For

example, SGD, HOA, and TIS often occur together in a five-minute period, but SGD and L do

not.

Types of Instruction

L-lecture/presentation MG –games/activities that are not investigative

in nature (students discovering subject area

matter)

PM-teacher problem modeling CL-cooperative learning (roles)

SP-student presentation (formal) LC-learning center/station

LWD-lecture with discussion TIS-teacher/faculty member interacting w/

student

D-student demonstration (e.g., student

demonstrates how to solve a problem)

UT-utilizing digital educational media and/or

technology

CD-class discussion (almost all student-student

talk)

A-assessment: Please describe below.

IP-independent practice (if in groups, add

SGD)

AD administrative tasks. Please describe

below.

R-reading (if in groups, add SGD) OOC-out-of-class experience. Please describe

below.

INV-investigative activity/manipulatives I-interruption. Please describe below.

SGD-small group discussion (this includes

pairs)

OTH-other: Please describe.

Student Engagement

Engaged students are represented by activities in the left-hand column. Dis-engaged students are

represented by activities in the right-hand column

Engaged Not Engaged

Listening/Watching Passively Off-Task (day-dreaming, doodling)

Writing Doing Work for Another Class

Speaking Listening to Others

Reading Disturbing Others

Hands-on Activity Playing



Cognitive Activity

1. Receipt of knowledge – Students are involved in the rote reception of information. This

generally includes listening to a lecture, going over homework, or watching the teacher

verify a concept through demonstration. The key feature of this category is that students

are receiving information but not significantly doing anything with the information

(passive learning).

2. Application of procedural knowledge – Students apply their knowledge. This typically

involves students using what they have learned, doing worksheets, practicing problems,

or building skills. The key feature of this category is that students are taking information

and applying it or practicing.

3. Knowledge representation – Students manipulate information. This is usually a step

beyond application. In knowledge representation activities students will typically

reorganize, categorize, or attempt to represent what they have learned in a different way.

For example, students might take the data from an activity and represent it graphically.

The key feature here is the reorganization or representation of information.

4. Knowledge construction – Students create new meaning. This typically involves

creating new understandings or making new connections. The key features of this

category is that students generate new knowledge or meaning, engage in higher order

thinking, and/or generate or invent problems based on new knowledge. For example,

students may use results of different activities to look for patterns that hold true.

5. Other – This category includes activities not included above, e.g., classroom disruptions.

These should be described.

Checklist Table

Time Period (in Minutes)

0-5 5-10 10-15 15-20 20-25 25-30

Instruction

Engagement

(#)

Cognitive

*If A, AD, OOC, I or OTH observed, please briefly describe here:



In question 1 below, please divide the total duration of the lesson into instructional and non-

instructional time using the information from the checklist on the prior page and your notes. In

question 2, make your estimates based only on the instructional time of the lesson.

Approximately how many minutes during the lesson were spent:

a. On instructional activities? ________ minutes

b. On housekeeping unrelated to the lesson/interruptions/other non-instructional

activities? ________ minutes

Considering only the instructional time of the lesson, approximately what percent of this time

was spent in each of the following arrangements?

a. Whole class _______ %

b. Pairs/small groups _______ %

c. Individuals _______ %

What did the interaction between students and teacher(s) look like, with respect to questions

posed?

Back and forth

Students generated questions

Teacher generates questions only

Teacher answers own questions

Teacher waits for responses

Teacher chooses different students to answer questions

Other: explain

How much time was spent on maintaining control of the class?

A lot Some A little Not at all

Differentiation

Generally, the instruction provided to this class was geared towards students at the following

level (circle one):

1 2 3 4 5 9

Low-performing Average Advanced Students

worked

at own

level



Which of the following student-groups were provided individualized, differentiated instruction?

(Check all that apply)

High-Performing

Average-Performing

Low-Performing

Overall, rate the extent in which differentiation (or individualized) instruction occurred.

1 2 3 4 5

Not at All Very Much So

On average, how did this differentiation occur? Check all that apply.

Error Intervention or re-teaching

Performance-based pairing of students for lab activities

One-on-One lab instruction

Additional homework

Supplemental worksheets/in-class assignments

Supplemental independent lab or hands-on investigations

More instruction

More questions directed at these students

Other: explain



DRAFT

Lesson Ratings

In this part of the form, you are asked to rate each of a number of key indicators in five different

categories, from 1 (not at all) to 5 (to a great extent). You may list any additional indicators you

consider important in capturing the essence of this lesson and rate these as well.

Note that any one lesson is not likely to provide evidence for every single indicator; use 9,

“Can’t assess” when there is not enough evidence for you to make a judgment or when you

consider the indicator inappropriate given the purpose and context of the lesson.

Lesson Design Not

at All

A

great

extent

Can’t

Assess

The lesson was organized around a big-

picture, or main concept.

1 2 3 4 5 9

The organization of the lesson engaged

students

1 2 3 4 5 9

The resources utilized in this lesson enhanced

instruction

1 2 3 4 5 9

The lesson incorporated hands-on, inquiry-

based activities requiring application of

knowledge

1 2 3 4 5 9

The lesson encouraged collaborative learning. 1 2 3 4 5 9

The instructional strategies/activities in this

lesson are individualized to student needs.

1 2 3 4 5 9



DRAFT

Content of Lesson Not

at All

A

great

extent

Can’t

Assess

Important ideas of the lesson intellectually

engaged students.

1 2 3 4 5 9

Connections were made with other subject

areas and/or real-world contexts.

1 2 3 4 5 9

Objectives were clearly identified in the

beginning of lesson.

1 2 3 4 5 9

Students demonstrated an understanding of the

lesson.

1 2 3 4 5 9

Students explained their answers/reasoning to

the class.

1 2 3 4 5 9

Implementation of Lesson Not at

All

A

great

extent

Can’t

Assess

Teacher appeared confident in his/her ability to

teach this lesson.

1 2 3 4 5 9

The teacher’s classroom management

style/strategies enhanced lesson quality.

1 2 3 4 5 9

The lesson pace was appropriate for the

levels/needs of the students.

1 2 3 4 5 9

The teacher was able to identify student

knowledge (i.e. “read” the students’ level of

understanding).

1 2 3 4 5 9

The teacher’s instruction created a student-

centered learning experience, providing

opportunities for questions and exploration of

concepts.

1 2 3 4 5 9

The teacher’s questioning strategies enhanced

student understanding/problem solving (e.g.

emphasized higher-order questions,

appropriately used “wait time,” identified

misconceptions, etc.).

1 2 3 4 5 9



Teacher-Student Interactions Not at

All

A

great

extent

Can’t

assess

There is a balance of give and take discussion

between student and teacher.

1 2 3 4 5 9

Talk is centered on what students are learning

rather than on controlling behavior.

1 2 3 4 5 9

There is a conversational tone to interactions in

the classroom.

1 2 3 4 5 9

The timing for teaching points is appropriate

(e.g., teacher does not interrupt learning)

1 2 3 4 5 9

Teacher responds to non-verbal cues of

confusion, boredom, and curiosity.

1 2 3 4 5 9

Students probe the teacher for more in-depth

knowledge about concepts.

1 2 3 4 5 9

Students answer and/or explain questions asked

by teacher.

1 2 3 4 5 9

Classroom Culture Not

at

All

A

great

extent

Can’t

assess

Active participation of ALL students was

encouraged and valued.

1 2 3 4 5 9

There was a climate of respect for students’

ideas, questions, and contributions to the lesson.

1 2 3 4 5 9

Interactions reflected collaborative working

relationships among students.

1 2 3 4 5 9

The lesson encouraged students to generate ideas,

questions, conjectures, and/or propositions.

1 2 3 4 5 9

Intellectual rigor, constructive criticism, and the

challenging of ideas were evident.

1 2 3 4 5 9

Classroom environment was disruptive. 1 2 3 4 5 9

In general, students show a high level of interest. 1 2 3 4 5 9



Likely Impact of Lesson Negative

Effect

No net

effect/Mixed

Effect

Positive

Effect

Can’t

assess

Students’ understanding of

important concepts.

1 2 3 4 5 6 7 9

Students’ ability to apply or

generalize skills and

concepts to other areas

and/or real-life situations.

1 2 3 4 5 6 7 9

Students’ self-confidence in

doing work in the subject

area.

1 2 3 4 5 6 7 9

Students’ engagement

during class

1 2 3 4 5 6 7 9

Students’ interest in and/or

appreciation for the

discipline.

1 2 3 4 5 6 7 9

Overall Lesson Quality

In this final rating, consider all available information about the lesson, its context and the

teacher’s purpose, and your own judgment of the relative importance of the ratings you have

made. Select the capsule description that best characterizes the lesson you observed. Keep in

mind that this rating is not intended to be an average of all the previous ratings, but should

encapsulate your overall assessment of the quality and likely impact of the lesson.

1 2 3 4 5

Ineffective Exemplary

Instruction Instruction



APPENDIX B

Teacher Interview Guide

1. In general, how satisfied are you with your experiences in the Teach for America program?a. Are there any suggestions for ways your TFA experience could be improved?b. What has been the most positive aspect of your TFA experience thus far?

2. In general, how satisfied are you with the school you’ve been assigned to?a. How satisfied are you with the level of support you have received from the staff at your

school?b. How satisfied are you with the level of support you have received from the local TFA

contact?

3. Overall, how satisfied are you with your ability in the classroom in the following areas?a. Leadership Skills/Classroom Management Practicesb. Curriculum/Subject Proficiency

Note: be sure to probe about their college degree and alignment with teaching assignment. If assigned outside degree area, how have they adjusted?

c. Facilitation of Student Learning/ability to engage students

4. Do you plan to continue teaching (or remain in the education field as a school administrator, or with a nonprofit that is involved in education reform) after your Teach for America 2-year com-mitment ends?



APPENDIX C

Principal Interview Guide1. Overall, how has the Teach for American program impacted your school?

a. Are there any suggestions for ways your involvement with TFA could be improved?b. What has been the most positive aspect of your involvement with TFA thus far?

2. Overall, how satisfied are you with the Teach for American teachers assigned to your school?

3. Overall, how satisfied are you with the ability of the TFA teachers in the following areas?c. Leadership Skills/Classroom Management Practicesd. Curriculum/Subject Proficiencye. Note: be sure to probe about their college degree and alignment with teaching assignment.

If assigned outside degree area, how have they adjusted? f. Facilitation of Student Learning/ability to engage students

An Evaluation Report Prepared by the

CENTER FOR RESEARCH & EVALUATION OFFICE OF ACCOUNTABILITY

May 2011

In compliance with federal law, Charlotte-Mecklenburg Schools administers all education programs, employment activities and admissions without discrimination against any person

on the basis of gender, race, color, religion, national origin, age or disability.

Teach For America EVALUATION REPORT · 2018. 7. 30. · Teach For America (TFA), an organization...

Documents

Transcript of Teach For America EVALUATION REPORT · 2018. 7. 30. · Teach For America (TFA), an organization...