Math Connects National Field Study - Amazon S3 · 2015-09-29 · National Field Study Final Results...

Math Connects National Field Study

Final Report: Student Learning, Student Attitudes and Teachers’ Reports on Program Effectiveness

October 2009

Macmillan/McGraw- Hill and Glencoe

Math Connects National Field Study: Student Learning, Student Attitudes and Teachers' Reports on Program Effectiveness

Evaluation Report

University of Cincinnati Evaluation Services Center

Debbie Zorn, Director

Report prepared by:

Jerry M. Jordan, Ph.D., Research Associate

Suzanne van Berlo-van den Hoogenhof Audra Morrison Courtney Brown

Martin Saperstein, Ph.D., Saperstein Associates Stephanie Groce, Saperstein Associates Emily Goldman, Saperstein Associates Holly Knight, Saperstein Associates

© 2009 Evaluation Services Center University of Cincinnati PO Box 210105 Cincinnati, OH 45221-0105 Tel: (513) 556-3900 Fax: (513) 556-5112 http://www.uc.edu/evaluationservices/ E-mail: [email protected]

http://www.uc.edu/evaluationservices/�

mailto:[email protected]�

Math Connects National Field Study Final Results 1

Table of Contents Purpose of the Evaluation ................................................................................................ 3

Evaluation Questions ......................................................................................................................... 3

Research Design .................................................................................................................................. 4

Basic Design .................................................................................................................................... 4

Sampling and Assignment to Conditions .................................................................................... 4

Measures ............................................................................................................................................ 10

Student Learning ........................................................................................................................... 10

Student Opinions .......................................................................................................................... 10

Teacher Characteristics ................................................................................................................ 11

Teacher Perceptions and Opinions ............................................................................................ 11

Results: Student Learning .............................................................................................. 12

Basic Student Learning ..................................................................................................................... 12

Math Connects Basic Effectiveness ............................................................................................. 12

Student Learning by Gender ..................................................................................................... 12

Student Learning by Race .......................................................................................................... 14

Student Learning by Reading Level .......................................................................................... 15

Learning For Special Needs Students ...................................................................................... 18

Teacher Characteristics, Teacher Practices and Student Learning ............................................ 20

Teachers’ Self Efficacy ............................................................................................................... 20

Teachers’ Math Beliefs ............................................................................................................... 22

Teachers’ Reform-Based Practices ........................................................................................... 24

Program Comparisons ..................................................................................................................... 26

Basic Comparisons Between Conditions ................................................................................. 26

Subgroup Comparisons .............................................................................................................. 28

General Summary of Student Learning Results ........................................................................... 32

Basic Effectiveness of Math Connects ........................................................................................ 32

Comparing Math Connects with Other Mathematics Programs ............................................. 33

Results: Evaluations of K-5 Curriculum Materials ........................................................ 35

Elementary Teachers’ General Assessments................................................................................. 36


Qualitative Comments about Program Strengths .................................................................. 36

NCTM Standards ........................................................................................................................ 39

Focused Assessments of Basic Characteristics ....................................................................... 42

Assessments of Primary Resources .......................................................................................... 48

Focused Assessments of Curriculum Materials ............................................................................ 51

Vertical Alignment ...................................................................................................................... 51

Balanced Instruction ................................................................................................................... 52

Differentiated Instruction .......................................................................................................... 60

Digital Resources ......................................................................................................................... 65

Assessment and Intervention .................................................................................................... 70

Mathematical Literacy ................................................................................................................. 73

Teacher Professional Development ......................................................................................... 77

Blended Instruction .................................................................................................................... 78

Results: Middle School Teachers’ Evaluations of Curriculum Materials...................... 81

Seventh Grade Teachers’ General Assessments .......................................................................... 82

Qualitative Comments about Program Strengths .................................................................. 82

NCTM Standards ........................................................................................................................ 84

Focused Assessments of Basic Characteristics ....................................................................... 86

Assessments of Primary Resources .......................................................................................... 89

Focused Assessments of Curriculum Materials ............................................................................ 91

Vertical Alignment ...................................................................................................................... 91

Balanced Instruction ................................................................................................................... 91

Differentiated Instruction .......................................................................................................... 97

Digital Resources ....................................................................................................................... 100

Assessment and Intervention .................................................................................................. 104

Blended Instruction .................................................................................................................. 107

Mathematical Literacy ............................................................................................................... 107

Teacher Professional Development ....................................................................................... 110

General Conclusions ..................................................................................................... 112

Student Learning ............................................................................................................................. 112

Teacher and Student Opinions of Math Connects ........................................................................ 113

References ..................................................................................................................... 114


Math Connects National Field Study

Purpose of the Evaluation

The University of Cincinnati Evaluation Services Center (UCESC), in collaboration with Saperstein Associates, conducted a quasi-experimental1

Evaluation Questions

field study of the effectiveness of the Macmillan/McGraw-Hill K-5 and Glencoe middle school mathematics program, Math Connects. The study was conducted during the 2008-2009 school year. It includes a total of 139 teachers and approximately 3,600 students nationwide. This research was designed to test the assumptions on which the Math Connects program is based, assess teacher and student attitudes about their mathematics programs and objectively assess the effectiveness of the Math Connects program in helping students learn mathematics. The design of this study incorporates valid quantitative assessment of student learning, student attitudes, teacher practices and teacher attitudes while also admitting the consideration of the teachers’ professional wisdom through the solicitation of qualitative responses to open-ended questions about the materials they used and their perceptions of their students’ attitudes and knowledge gains.

Five specific evaluation questions guided this assessment:

1) Are the Math Connects curriculum materials effective in helping students improve their mathematics abilities?

a) Is Math Connects effective with all types of students?

b) Can Math Connects be used effectively by all types of teachers?

2) Does student learning of mathematics differ between students whose teachers used Math Connects and students whose teachers used other curriculum materials?

3) Did the teachers who used Math Connects like those materials?

a) Did these teachers think Math Connects was effective?

b) Did these teachers think Math Connects was easy to use?

4) Do teacher assessments of Math Connects differ from teacher opinions of other curriculum materials?

5) What do the students whose teachers used Math Connects think about the Math Connects materials?

1 Quasi-experiment is a common technical term used to designate specific design features of a study. Some readers may want to consult Cook and Campbell (1979) or others such as Babbie (1992) if they are unfamiliar with research designs.


Research Design

Basic Design. This evaluation of Math Connects employed a quasi-experimental design (see Cook & Campbell, 1979; Shadish, Cook & Campbell, 2002). That is, functionally equivalent groups of teachers and students were created by the careful matching of teachers and schools on key variables (e.g., teacher experience, teacher practices, school locations and student social demographic characteristics). Details of how the teachers were assigned to conditions are provided further below. The quasi-experimental design was employed for several reasons. Shadish and Cook (2009) note that well designed quasi-experiments of this type can be as effective as randomized controlled experiments for making causal inferences. As Cook (2004) suggests, accepting the traditional research model of medical research and considering random control designs as the primary component of some “gold standard” of evaluation is misguided. Carpenter, Dossey and Koehler (2004) are very direct in their call for policy makers to recognize the limits of traditional, scientific experimental designs as they advocate for better research employing different methods. Schoenfield (2006) implies a similar point in his critique of the often-cited but seldom understood standards outlined by the What Works Clearinghouse of the United States Department of Education.

The standards driving the design of this study were not the standards of clinical medical research, but rather the standards accepted as ideal for the assessment of educational programs (see The Joint Committee on Standards for Educational Evaluation, 1994). To enhance the external validity of this research, a nationwide study was created that included as many geographical locations as possible. This design allowed the involvement of a total of 106 schools in 11 states.

The design of this study did not mix experimental conditions within schools. Often randomized control studies assign classrooms within the same school to different experimental conditions. Thus, for example, one second grade mathematics teacher within a school may be given a new program to use while another second grade teacher in the same school continues to use an older program. This is a procedure which, in the abstract, provides an effective control against confounding school effects or teacher effects. However, such designs result in a situation where students and teachers within the same grade within one school are actually using different programs. Such designs make treatment reliability highly questionable because they assume that teachers do not share practices across conditions and that students do not interact with each other. Thus they create learning environments that rarely, if ever, actually occur in schools. The quasi-experimental design was used to maximize ecological validity.

Sampling and Assignment to Conditions. Initially, participants were recruited via a mass emailing sent in April of 2008 to teachers known to teach mathematics in grades two, four or seven. The initial pool of voluntary respondents to this email was screened for geographical location, teachers’ characteristics and school characteristics. Secondary screening of study participants occurred as respondents from restricted states were eliminated or as teachers withdrew from consideration after receiving more information about the details of the study.

The remaining pool of teachers (N = 139) was initially randomly assigned to either an experimental or comparison condition. Ultimately, teachers in the experimental condition were provided with the grade appropriate version of the Math Connects mathematics curriculum


materials (copyright 2009 by Glencoe/McGraw-Hill) to use during the entire 2008-2009 school year. They were provided with the standard appropriate training in those materials by representatives from the publisher prior to, or very near the beginning of the 2008-2009 school year. Teachers in the comparison condition continued to use their current mathematics program.

However, the original group assignments were altered (prior to any training) to create more precise matching of teachers and schools on relevant variables after adjustments were made for pragmatic considerations. For example, some teachers could not get permission from their administration to implement a new program. Thus these teachers had to be placed within the comparison condition. Or, in some cases, teachers were given permission to participate only if they were provided with new curriculum materials. Thus, these teachers had to be placed in the experimental condition. Ultimately, a purposive matching process was employed to maintain group equivalency between the experimental and comparison groups.

This matching process created conditions balanced according to critical teacher and school characteristics. Teachers in the experimental condition (final n = 68) had an average of 14.00 years teaching experience (range 1-40) and teachers in the comparison condition (final n = 71) had an average of 13.55 years teaching (range 1-40). The difference in Means was not statistically significant (t = .32, NS). Teachers in the experimental condition had an average of 13.35 years of experience teaching mathematics specifically (range 1-40). The Mean for the teachers in the comparison condition was identical (13.35). Thus conditions were well balanced regarding teacher experience.

Since group equivalency is so crucial to causal inferences, additional steps were taken to allow for equivalency to be validated at the conclusion of the study. All teachers completed a measure of mathematics teaching self efficacy (Enochs, Smith & Huinker, 2000) in a pretest-posttest manner. This measure is designed to assess the extent to which teachers believe they are effective at teaching mathematics. Both within and across grade levels the teachers’ scores on this instrument were nearly identical across conditions (see Table below). Teachers also completed a math beliefs instrument (Capraro, 2005; based on Fennema, Carpenter & Loef, 1990). This instrument assesses teachers’ fundamental beliefs about how children best learn mathematics. Briefly, a higher score reflects the belief that children construct their own knowledge of mathematics as teachers help them explore mathematical concepts. A lower score reflects beliefs centered on the notion that students receive their knowledge of mathematics more or less directly from the teacher (Capraro, 2005). Again, both within and across grade levels, teachers’ scores were nearly identical across conditions (see Table below). Significance testing of all Means across conditions revealed no significant differences for any of these measures at any grade level.


Teacher Attitudinal Measures by Grade and Condition

Grade 2 Grade 4 Grade 7

Experimental Comparison Experimental Comparison Experimental Comparison Pretest Math Teaching Self Efficacy*

3.17 (n=24)

3.06 (n=27)

3.16 (n=24)

3.12 (n=27)

3.29 (n=14)

3.30 (n=9)

Posttest Math Teaching Self Efficacy

3.18 (n=24)

3.17 (n=30)

3.18 (n=25)

3.15 (n=27)

3.23 (n=14)

3.20 (n=9)

Pretest Math Beliefs*

2.55 (n=22)

2.63 (n=23)

2.66 (n=23)

2.66 (n=25)

2.67 (n=14)

2.58 (n=8)

Posttest Math Beliefs

2.65 (n=23)

2.59 (n=29)

2.63 (n=25)

2.65 (n=29)

2.65 (n=15)

2.70 (n=9)

*Teachers’ scores on the self efficacy scale could range from 1-4 with 1 indicating very low self efficacy. Teachers’ scores on the math beliefs instrument could range from 1-4 with 1 indicating a belief that students receive knowledge from teachers in a direct fashion and 4 indicating a belief in active construction of knowledge. Teachers were asked to provide the official state or district “gradecard” classification for their schools. These labels vary greatly across states, but they essentially classify schools as below average in performance on state test results (e.g., “Academic Watch” or “Need Improvement”), performing as expected (e.g., “Effective” or “Satisfactory”) or performing at a high level (e.g., “Excellent” or “Highly Performing”). A total of 95 teachers provided interpretable classifications for their schools. They also provided their community classification indicating the nature of their school district (City: large, midsize or small; Suburb: large, midsize or small; and Town: fringe or rural). The two Tables below provide summaries of school characteristics by condition. They suggest that the conditions were well balanced by school performance characteristics. However, the final assignments resulted in the experimental condition drawing students from more small suburban schools while the comparison condition included more large and midsize city schools.

Classifications of Participating Schools by Condition

State or District Performance Classification

Below

Standards Meets Standards Exceeds

Standards

Experimental 5 28 20

Comparison 5 26 19


Classifications of Participating Schools by Community

Community Classification

City: Large

City: Midsize

City: Small

Suburb: Large

Suburb: Midsize

Suburb: Small

Town: Fringe

Experimental 4 2 5 39 6 12 1

Comparison 13 10 8 35 2 1 2

Because of the potential implications of the lack of balance across conditions in community classifications of the schools, the final sample of students was examined on key characteristics. The Table immediately below displays the distribution of student gender and race within each condition for all the participating students who provided data in the final sample. The data indicate a reasonable balance of gender and race across conditions.

The student sample was also examined to check for the number of students who had special codes within their records indicating the student qualified for a Special Accommodation (typically called a “504 Accommodation Plan”), an Individual Education Plan (IEP) or a Literacy Plan (for English Language Learners). The data in the Table further below reveals that while the two conditions were reasonably balanced in terms of students with special accommodations, the experimental condition contained substantially more students with Literacy Plans and, to a lesser extent, IEPs. Therefore, all analyses of student learning data making comparisons across conditions were conducted separately for students having IEPs or Literacy Plans.

Gender and Racial Composition of Final Sample of Students (N=3,590)

Gender Race

Female Male White Black Asian Latino Other Races

Experimental (n=1,958) 48.2% 51.8% 64.8% 13.4% 3.3% 11.9% 6.6%

Comparison (n=1,632)

49.9% 50.1% 58.0% 15.4% 1.4% 19.6% 6.6%


Student Special Codes by Condition

Student Special Codes

504

Accommodation IEP Literacy Plan

Experimental 49 145 184

Comparison 37 103 54

Finally, group equivalencies were examined through a comparison of students’ pretest scores in mathematics and reading.2

First, the mathematics pretest for the second graders indicated that the students in the experimental group (n = 374) and the students in the comparison group (n = 534) achieved essentially identical Mean scores, (t = .74, NS). For the fourth graders, the students in the comparison group (n = 519) scored significantly higher than students in the experimental group (n = 470), (t = 3.2, p < .00). There was no statistically significant difference between the experimental group (n = 734) and the comparison group (n = 337) for the seventh graders (t = .83, NS). There were no statistically significant differences in the comparison of the reading pretest scores. The data in the Figures below reveals the Mean scores across conditions were nearly identical.

All students completed the TerraNova survey form (CTB/McGraw-Hill, 2008) in mathematics and reading. The TerraNova is a well known, respected and widely used standardized test in elementary and secondary education. The data displayed in the Figures below show that the average pretest scores of students in both conditions were very similar, and, some cases, nearly identical. However, t-tests of the Means between groups did indicate statistically significant differences in some cases.

The appearance of one statistically significant difference between groups might be seen to cast some doubt on the assumption of roughly equivalent groups across experimental conditions. However, it must be noted that statistical significance should not be confused with substantive importance (see, for example, Pedhazur & Schmelkin, 1991, chapter 9; or Hays, 1981, chapter 7).3

2All general comparative analyses were performed only with students who provided both a pretest and a posttest score in mathematics and reading and, as noted, only with students who did not require an IEP or a Literacy Plan. Analyses of these students are reported separately.

The large sample sizes in this study create a situation where even trivial differences in groups can be detected statistically even though the average scores of the two groups were very nearly identical in most cases.

3 Hubbard & Armstrong (2005) provide a full explanation of the common misunderstandings and applications of the implications of statistical significance.


An examination of the Normal Curve Equivalent (NCE) scores4

4 The NCE scores are normed to the National Percentiles of students who took the TerraNova across the country within the same testing period (Fall 2008 and Spring 2009). Thus they reflect how a student, or a group of students, scored in comparison to national trends. They are similar to percentile scores in that they reflect where a student scored in reference to other students once the scores are transformed into a normal curve (potentially, NCE scores range from 1-99).

of each condition suggests that the groups of children in this study were quite representative of children across the country and that no group in this study could be considered substantially different from other groups in terms of reading or mathematics ability. The average scores of all groups are all approximately in the middle of the range of scores. Clearly, the students in the two groups can reasonably be judged to be members of the same general student population.

535599

667

532

609666

0

100

200

300

400

500

600

700

800


Avera

ge S

core

s

Average Pretest Mathematics Scores by Grade and Condition(TerraNova Scaled Scores)

Math Connects Other Programs

578629

668

579632

670

0

100

200

300

400

500

600

700

800


Avera

ge S

core

s

Average Pretest Reading Scores by Grade and Condition(TerraNova Scaled Scores)



Normal Curve Equivalent Scores by Grade and Condition


NCE Reading

Score NCE Math

Score

NCE Reading

Score NCE Math

Score

NCE Reading

Score NCE Math

Score

Experimental (n=1,958) 47.6 47.0 50.2 43.3 54.3 50.7

Comparison (n=1,632) 47.8 45.3 51.6 48.4 54.9 50.6

In sum, basic group equivalency is verified by teacher measures of experience and attitudes toward teaching mathematics. It is also verified by student testing data in mathematics and reading. Additionally, an examination of group composition by race and gender is consistent with group equivalency. Finally, school level data on community types and state or district test performance validates that the experimental and comparison groups are basically equivalent on relevant dimensions.

Measures5

This study employed measures of student learning, student attitudes toward mathematics, student attitudes toward curriculum materials, teacher attitudes toward teaching mathematics, teachers’ practices, teachers’ perceptions of student learning and student engagement and teachers’ attitudes toward curriculum materials.

Student Learning. All students within the study were tested both on mathematical abilities and reading in a pretest-posttest fashion. All students completed the grade-level-appropriate version of the TerraNova Survey of reading and mathematics (CTB/McGraw-Hill; www.ctb.com).

Reading level was assessed because, especially at primary levels, reading ability is known to affect academic performance in mathematics in fairly robust ways (Adams & McKoy, 2007; van Garderen, 2004; Walker, Zhang and Surber, 2008). Thus reading is being assessed as a potential moderating variable in discerning the effects of the different materials being used by teachers.

Student Opinions. All students at the fourth and seventh grade levels were surveyed at the end of the year to assess basic attitudes. First, the students’ attitudes about the materials they used in their mathematics classes were assessed via a survey constructed by UCESC personnel. This survey was specific to the materials used and assessed the extent to which the students used each component and how helpful the students found each component.

5 Copies of all instruments (except TerraNova) are included in an appendix.


Teacher Characteristics. All teachers completed a measure of mathematics teaching self efficacy (MTEBI Enochs, Smith & Huinker, 2000) in a pretest-posttest manner. Self efficacy is the belief that one can actually enact the behaviors that one believes to be necessary to produce certain results (see Bandura, 1986, 1997). Teacher self efficacy has been shown to be related to a number of important outcomes in education including basic student achievement (see Aderman, 2004; Henson, Kogan, & Vacha-Haase, 2001).

All teachers also completed a math beliefs instrument (MBS, Capraro, 2005; based on Fennema, Carpenter & Loef, 1990). This instrument assesses teachers’ fundamental beliefs about how children best learn mathematics. A higher score reflects the belief that children construct their own knowledge of mathematics as teachers help them explore mathematical concepts. A lower score reflects beliefs centered on the notion that students receive their knowledge of mathematics more or less directly from the teacher (Capraro, 2005).

Finally, all teachers completed a teaching practices scale (McCaffrey, Hamilton, Stecher, Klein, Bugliari & Robyn, 2001) as a part of their online biweekly logs. This scale asked teachers directly about the extent to which they had been engaging in specific reform-based teaching practices (e.g. having students explain their reasoning for solving a problem or rearranging seating so that students could work in small groups) and also asked about specific traditional practices (e.g., lecturing to the entire class or having students work individually on practice problems).

Teacher Perceptions and Opinions. All teachers completed posttest questionnaires soliciting their evaluations of the materials they used during the school year. The basic content of this assessment questionnaire was identical for both the experimental and comparison teachers within each grade level. However, the questionnaire completed by the teachers using Math Connects was elaborated in places to allow for more detailed assessment of specific components within the Math Connects materials.


Results: Student Learning

Basic Student Learning

Math Connects Basic Effectiveness. Primary analyses of student test scores centered on verifying the fundamental effectiveness of the Math Connects materials. The Figure below displays the average pretest and posttest TerraNova mathematics scores for all students using the Math Connects materials. The data verifies that, as expected, students’ average scores increased from the beginning of the year to the end of the year and that knowledge gain was statistically significant for all grades (second grade n = 374, t = 23.7, p < .00; fourth grade n = 470, t = 14.8, p < .00; seventh grade n = 734,: t = 8.5, p < .00).

• The standardized student testing, using the TerraNova in a pretest- posttest fashion, indicates that Math Connects is effective at all grade levels. Students using Math Connects in both elementary school and middle school showed substantial, significant learning gains over the course of the 2008- 2009 school year.

Student Learning by Gender. Additional analyses were conducted to explore the effectiveness of Math Connects with important student subgroups. The Figures below present the data on student learning in mathematics by gender for each grade. Within each grade level, both males and females showed statistically significant gains from pretest to posttest. For grade two the average pretest score for females (n = 187) was 528, average posttest was 585 (t = 18.6, p < .00). For males in grade two (n =184) the average pretest score was 543, the average posttest was 592 (t = 15.1, p < .00). In grade four, the average pretest score for females (n=230) was 600, average posttest was 630 (t = 11.2, p < .00). For the males in grade four (n = 237) the average pretest score was 597, average posttest was identical to the females

535

588599

630

667681

520

540

560

580

600

620

640

660

680

700

Pretest Scale Score (TerraNova) Posttest Scale Score (TerraNova)

Avera

ge S

cale

Sco

re

Math Connects: Student Learning in Math



at 630 (t = 9.9, p < .00). In grade seven, the average pretest score for the females (n = 375) was 668, the posttest average was 684. For males in grade seven (n = 357) the average pretest was 665 and the average posttest was 679 (t = 4.8, p < .00). Thus the data verify that Math Connects was equally effective for both males and females in all grades.

528

585

543

592

500

510

520

530

540

550

560

570

580

590

600


Avera

ge S

cale

Sco

reMath Connects: Student Learning by Gender: Grade 2

Females (n=187) Males (n=184)

600

630

597

630

550

560

570

580

590

600

610

620

630

640

650


Avera

ge S

cale

Sco

re

Math Connects: Student Learning by Gender: Grade 4



• Research indicates that Math Connects is effective for both males and females. Across grade levels, both males and females showed statistically significant learning gains over the course of the 2008- 2009 school year. Math Connects is equally effective for both males and females in that both genders showed functionally equal knowledge gains at all grade levels tested.

Student Learning by Race. Student learning was also analyzed by race. The data in the Figures below supports the notion that Math Connects is effective for students of all races. All racial subgroups (Asian, Black, Latino and White) showed statistically significant gains from pretest to posttest in both elementary and middle school. One unexpected result occurred within the seventh grade data (see Figure) as the Latino students using Math Connects showed an unusually large knowledge gain over time. The 40 Latino students in this condition were spread across nine classrooms but more than half (23) were in two classrooms, both of which showed substantial average gains from pretest to posttest. This suggests that the impressive gains in this group may be attributable to a combination of teacher and program effects.

668

684

665

679

620

630

640

650

660

670

680

690

700

710

720


Avera

ge S

cale

Sco

re

Math Connects: Student Learning by Gender: Grade 7


570

609

499

561

516

572

544

596

470

490

510

530

550

570

590

610


Avera

ge S

cale

Sco

re

Math Connects: Student Learning by Race: Grade 2

Asian (n=24) Black (n=41) Latino (n=69) White (n=192)


• Research results indicate that Math Connects is equally effective across racial groups. At every grade level tested, students of all races using Math Connects showed statistically significant knowledge gains over time. This gain was particularly pronounced with middle school Latino students.

Student Learning by Reading Level. Data analyses confirmed that Math Connects is effective with students at all reading levels. All students in the study completed the survey version of the TerraNova reading test to assess their basic reading abilities. Students were categorized by their TerraNova Normal Curve Equivalent (NCE) scores in reading. Students scoring 33 or below on their NCE were categorized as low level readers, students with NCE scores of 34-66 were categorized as medium and students scoring between 67 and 99 were categorized as high level readers.

First, it should be noted that, as expected, student reading level showed a strong positive association with math abilities. For students across all conditions, reading level showed a significant positive association with both their pretest and posttest TerraNova math

634

672

577

592595

628

613

647

550

570

590

610

630

650

670

690


Avera

ge S

cale

Sco

re



692705

646660

616

668673685

600

620

640

660

680

700

720

740


Avera

ge S

cale

Sco

re




scores.6

Focused analyses showed that readers at all levels, in every grade, made nearly identical levels of progress using the Math Connects materials. The Figures below display the pretest and posttest TerraNova math scores for low, medium and high level readers (by grade level).

That is, across conditions and across grade levels, the better readers consistently scored higher on their math tests.

Second grade readers of all levels showed substantial, statistically significant knowledge gains in mathematics over the course of the year. Comparisons of average pretest scores with average posttest scores revealed significant gains for all groups (low readers: t = 8.5, p < .00; medium level readers: t = 19.5, p < .00; high level readers: t = 10.8, p < .00). Fourth grade readers of all levels also showed substantial, statistically significant knowledge gains in mathematics over the course of the year. Comparisons of average pretest scores with average posttest scores revealed significant gains for all groups (low readers: t = 18.4, p. < .00; medium level readers: t = 11.1, p < .00; high level readers: t = 15.2, p < .00). Fourth grade readers of all levels also showed substantial, statistically significant knowledge gains in mathematics over the course of the year. Comparisons of average pretest scores with average posttest scores revealed significant gains for all groups (low readers: t = 18.4, p < .00; medium level readers: t = 11.1, p < .00; high level readers: t = 15.2, p < .00). And, finally, the same pattern held for the seventh grade students. Readers of all levels in the seventh grade also showed substantial, statistically significant knowledge gains in mathematics over the course of the year. Comparisons of average pretest scores with average posttest scores revealed significant gains for all groups (low level readers: t = 2.9, p < 00; medium level readers: t = 5.1, p < .00; high level readers: t = 8.3, p < .00).

6 A series of oneway ANOVAs was conducted with student’s pretest reading level as the factor and their pretest reading scale score as the dependent variable. Post hoc comparisons indicated significant differences among all groups (low, medium and high reading levels) such that the high level readers scored significantly higher on their math pretest than students at other levels. A parallel analysis for posttest classifications and posttest scores showed the identical pattern. This was the case at all grade levels.

482

531534

587576

631

450

470

490

510

530

550

570

590

610

630

650


Avera

ge M

ath

em

ati

csSca

le S

core

Math Connects: Learning in Math by Reading Level for Grade 2

Low Readers (n=62) Medium Readers (n=228) High Readers (n=84)


• Students at all levels of reading ability made substantial, statistically significant knowledge gains in mathematics over the course of the school year. Reading level did not have an effect on amount of mathematical knowledge gained. There were no statistically significant differences between the knowledge gain scores of students at different reading levels. This was the case in both elementary and middle school grades.

Students of every reading level showed nearly identical gains in mathematical knowledge. For every student a knowledge gain score was calculated by simply subtracting the student’s pretest score from their posttest score (TerraNova scale scores). These gain scores were then compared to see if student reading level affected how much math the student had learned. Analysis of variance testing (ANOVA with gain scores as the dependent variable and reading level as the factor) indicated no significant differences in knowledge gains across reading level (grade 2: F= 1.2, NS; grade 4: F = .24, NS; grade 7: F = .39, NS).

555

585602

635631

662

500

520

540

560

580

600

620

640

660

680

700


Avera

ge M

ath

em

ati

csSca

le S

core



635647

670684

690708

550

570

590

610

630

650

670

690

710

730

750


Avera

ge M

ath

em

ati

csSca

le S

core




Learning for Special Needs Students. Student learning was analyzed separately for students requiring special accommodations due to some form of learning disability (requiring an Individual Education Plan) or students requiring a Literacy Plan (due to the fact that they are English Language Learners). These results were mixed. The elementary school students showed very substantial gains while middle school students showed little growth. For grade two, both the students with IEPs and the students with LPs made statistically significant progress over the year (see Figure below). (For IEP students, t = 1.9, p <.00; for the LP students, t = 11.4, p <.00.) In fact, both of these groups showed average knowledge gains nearly equivalent in magnitude to the students who did not require special accommodations. That is, students within the general population of second graders showed an average gain of 53 points in their scale scores over the course of the year. The students with LPs showed a gain of 54 points and the students with IEPs showed a gain of 45 points.

In grade four, both groups of students showed improvement from pretest to posttest (see Figure below), but the change for students with IEPs fell just short of conventional levels of statistical significance. (For IEP students, t = 1.9, p < .06; for the LP students, t = 8.2, p < .00.) Once again, these students showed knowledge gains similar to those of the general student population. The general group of fourth graders scored an average increase of 31 points over the course of the year. The fourth grade students with LPs showed an average gain of 45 points, while the IEP students showed an average increase of 36 points.

486

531

511

565

475485495505515525535545555565575


Avera

ge S

cale

Sco

re

Math Connects: Learning of Special Needs Students: Grade 2

Students with IEPs (n=32) Students with LPs (n=68)


In grade seven, this group of special needs students showed virtually no knowledge gain. (Significance testing revealed that neither group made statistically significant gains over time.) Overall then, results suggest that Math Connects was used to effectively help special needs students in the elementary grades, but that no substantial learning occurred with these students in grade seven. (See Figure below.)

• Elementary school students using the Math Connects materials who also had special needs (requiring either an IEP or a Literacy Plan) progressed at the same rate as students in the general student population. They showed substantial, statistically significant gains in mathematical knowledge over time. However, seventh grade students with special needs showed virtually no gains over the course of the year.

561

597

565

610

550

560

570

580

590

600

610

620

630

640

650


Avera

ge S

cale

Sco

re



648 652

648 651

600

610

620

630

640

650

660

670

680

690

700


Avera

ge S

cale

Sco

re




Teacher Characteristics, Teacher Practices and Student Learning

Most generally, effective curriculum materials empower teachers to maximize their strengths and to minimize the effects of any weaknesses they might have as teachers. A complete understanding of the effectiveness of curriculum materials must include a consideration of how those materials might function with teachers of different abilities or different perspectives on teaching mathematics. Significant effort was made in this study to explore how the Math Connects materials might function for teachers who had very different ideas about how children learn mathematics and also how the materials might work for teachers of varying levels of interest and confidence in their ability to teach mathematics.

Teachers’ Self Efficacy. Teacher self efficacy has been shown to be related to a number of important outcomes in education including basic student achievement (see Aderman, 2004; Henson, Kogan, & Vacha-Haase, 2001). Self efficacy is the belief that one can actually enact the behaviors that one believes is necessary to produce certain results (see Bandura, 1986, 1997). So, for teachers, teaching self efficacy is the belief that they can actually perform the teaching behaviors necessary to produce desired learning in students.

All teachers in this study completed a measure of mathematics teaching self efficacy (the MTEBI, Enochs, Smith & Huinker, 2000) near the beginning of the school year and again at the end. This measure assessed the extent to which teachers believed they are effective at teaching mathematics. An average level of mathematics teaching self efficacy was calculated for each teacher by simply averaging the MTEBI pretest score with the MTEBI posttest score. Teachers were then classified as either having low, medium or high levels of mathematics teaching self efficacy. This classification was determined by the location of the individual teacher’s score within the distribution of the average scores for each grade level. So, for example, if a second grade teacher’s average self efficacy score was in the lower third of all the scores for second grade teachers, they were classified as “Low.” Analyses were then conducted to see how the Math Connects materials functioned with teachers of varying levels of mathematics teaching self efficacy.

For the second grade, students using the Math Connects materials showed functionally equivalent gains in knowledge for all levels of teacher mathematics teaching self efficacy (see Figure below). There was no statistically significant difference in the student knowledge gain scores across levels of teacher self efficacy (ANOVA, F = .31, NS). That is, students using the Math Connects materials seem to have learned just as well regardless of how confident their teachers were about teaching math. While, strictly speaking, the average gain scores should be regarded as functionally equivalent, it is interesting to note that the students whose teachers showed moderate or low levels of mathematics teaching self efficacy showed slightly greater gains than the students whose teachers showed a high level of self efficacy.


Similar results occurred for fourth graders in that there were no statistically significant differences among the gain scores for varying levels of teacher self efficacy (see Figure below; ANOVA of group Means, F = 2.4, NS). In this grade, the students of teachers with highest levels of mathematics teaching self efficacy showed the numerically highest gains while the students of teachers with low levels of self efficacy showed gains slightly larger than the medium level group.

The pattern for the seventh graders was somewhat unusual (see Figure below) in that the students of teachers with the highest level of self efficacy actually showed the lowest level of gain. In this case, the analysis indicated there were statistically significant differences (ANOVA, F = 20.1, p < .00). Post hoc comparisons indicated that each of the group Means was significantly different from the other two. So, the largest gains were made by students of teachers with moderate levels of self efficacy, followed by the students of teachers with low levels of self efficacy and, finally, the students of teachers with high levels of mathematics teaching self efficacy.

52.555.8

51.6

0

10

20

30

40

50

60

Low Teacher SE (n=107) Medium Teacher SE (n=127)

High Teacher SE (n=122)

Avera

ge K

no

wle

dg

e G

ain

s(o

n T

err

aN

ova S

cale

Sco

re)

Student Knowledge Gains by Teachers' Mathematics Teaching Self Efficacy: Grade 2

Student Knowledge Gain

3226.4

37.2

0

10

20

30

40

50

60

Low Teacher SE (n=117) Medium Teacher SE (n=191)

High Teacher SE (n=162)

Avera

ge K

no

wle

dg

e G

ain

s(o

n T

err

aN

ova S

cale

Sco

re)




Teachers’ Math Beliefs. The teaching of any subject is always a reflection of the teacher’s basic beliefs about the subject matter and beliefs about how students best learn that subject matter (see Nespor, 1987). All teachers in this study completed a math beliefs survey (MBS, Capraro, 2005; based on Fennema, Carpenter & Loef, 1990). This instrument assesses teachers’ fundamental beliefs about how children best learn mathematics. Briefly, a higher score reflects the belief that children construct their own knowledge of mathematics as teachers guide them in their exploration (essentially a constructionist perspective). A lower score reflects a perspective centered on the notion that students receive their knowledge of mathematics more or less directly from the teacher (Capraro, 2005). These different beliefs about the teaching of math can be seen to distinguish orientations toward inquiry based instruction (where students construct knowledge interactively through exploration) and what is commonly labeled traditional methods (where teachers “deliver” knowledge to students directly via lecture or demonstration).

Once again, since the teachers completed the MBS once at the beginning of the year and again at the end, an average MBS score was calculated by simply averaging the pretest and the posttest. Then, teachers were assigned to one of three categories based on their position within the distribution of scores from all teachers in their grade. So, for example, a teacher whose MBS score was in the top third of all the teachers in her/his grade level would have been assigned to the “High” category. This procedure, then, placed teachers in a “Low” category indicating that they prescribe mostly to traditional attitudes about delivering knowledge to students in a direct fashion. Teachers in the “High” category indicated that they generally believed students learned best through inquiry based methods and those in the “Medium” category can be seen to hold a mix of both forms of beliefs.

The results for the second graders using the Math Connects materials indicate that student learning was unaffected by teachers’ math beliefs (see Figure below). There were no statistically significant differences among the student knowledge gain scores (ANOVA, F = 1.3, NS). Numerically, the students whose teachers were in the Medium MBS category showed the greatest knowledge gains.

13.7

34.4

2.70

10

20

30

40

50

60

Low Teacher SE (n=363) Medium Teacher SE (n=145) High Teacher SE (n=213)

Avera

ge K

no

wle

dg

e G

ain

s(o

n T

err

aN

ova

Sca

le S

core

)




The data for the fourth grade students indicated that students of teachers with the lowest scores on the MBS showed significantly greater knowledge gains than the students of teachers with medium or low scores (ANOVA, F = 3.7, p < .03). So, while students of teachers of all forms of beliefs showed substantial knowledge gains, the teachers with mostly traditional beliefs about student learning of mathematics showed the largest gains (see Figure below).

Finally, the knowledge gains for seventh grade students using Math Connects were mostly consistent across all forms of teachers’ math beliefs (see Figure below). There were no statistically significant differences among the Means of the three teacher groups.

50.1

57.948.1

0

10

20

30

40

50

60

Low Teacher MBS (n=94) Medium Teacher MBS (n=141)

High Teacher MBS (n=126)

Avera

ge K

no

wle

dg

e G

ain

s(o

n T

err

aN

ova S

cale

Sco

re)

Student Knowledge Gains by Teachers' Mathematics Beliefs:Grade 2


38.8

24.6

28.8

0

10

20

30

40

50

60

Low Teacher MBS (n=231) Medium Teacher MBS (n=98) High Teacher MBS (n=199)

Avera

ge K

no

wle

dg

e G

ain

s(o

n T

err

aN

ova S

cale

Sco

re)




Teachers’ Reform- Based Practices. Throughout the course of this study, teachers completed semimonthly activity logs via an online questionnaire. Logs were completed 16 times over the course of the school year. The first entry was between October 6 and October 17, 2008, and the final between May 25 and May 29, 2009. Twice each month teachers were required to log in to a specified Web site and respond to a brief set of questions about their recent activities in their mathematics classrooms. A teaching practices scale (McCaffrey, Hamilton, Stecher, Klein, Bugliari & Robyn, 2001) asked teachers directly about the extent to which they had been engaging in specific reform-based teaching practices (e.g. having students explain their reasoning for solving a problem or rearranging seating so that students could work in small groups). For every teacher, a reform practices score could be calculated for each two week period by simply summing their responses to each of these items and dividing by the number of items. Then, an overall reform practices score could be calculated for each teacher by averaging their bi-weekly scores. This overall reform score was a direct reflection of how often the teacher used specific reform-based practices over the course of the school year.

The student knowledge gain scores were analyzed to see if the teachers’ use of reform-based practices were related to the students’ TerraNova scores. Teachers were categorized as “Low,” “Medium” or “High” in the use of reform practices. This categorical assignment was based on where the individual teacher fell within the distribution of reform practices scores for all teachers in the same grade level. The Table below displays the data for the second grade students using the Math Connects materials. The data indicates that learning gains were consistent for these students regardless of teachers’ propensity to use reform-based practices (ANOVA, F = .62, NS). The same pattern appears for the fourth grade students. While, the students whose teachers reported the highest levels of reform-based practices showed the lowest gains in knowledge, the testing of group Means showed no statistically significant differences (ANOVA, F = 1.5, NS).

16.6 18.912.2

0

10

20

30

40

50

60

Low Teacher MBS (n=137) Medium Teacher MBS (n=229) High Teacher MBS (n=311)

Avera

ge K

no

wle

dg

e G

ain

s(o

n T

err

aN

ova S

cale

Sco

re)




The data pattern in the seventh grade indicated that Math Connects students whose teachers reported a medium level of reform-based practices showed significantly lower knowledge gains than teachers who reported high levels of reform-based practices (ANOVA, F = 3.6, p < .03; post hoc comparisons indicated significance only between the medium and high categories).

51.3 50.9

56.2

0

10

20

30

40

50

60

Teacher Reform Practices Low (n=142)

Teacher Reform Practices Medium (n=73)

Teacher Reform Practices High (n=159)

Avera

ge K

no

wle

dg

e G

ain

s(o

n T

err

aN

ova

Sca

le S

core

)

Student Knowledge Gains by Teachers' Use of Reform Practices:Grade 2


35.6 36.126.8

0

10

20

30

40

50

60




Avera

ge K

no

wle

dg

e G

ain

s(o

n T

err

aN

ova

Sca

le S

core

)




Program Comparisons

A second major set of analyses was performed to make direct comparisons between the student learning within the classrooms that used Math Connects during the school year (the experimental group) and the comparison classrooms that used their existing mathematics curriculum materials.7

Basic Comparisons Between Conditions. Basic group comparisons were made by an examination of the two groups’ average pretest and posttest scores on the TerraNova mathematics test. The Figures below display the average pretest and posttest scores for both the experimental and comparison groups by grade level.

For the second graders, the Math Connects students showed a larger knowledge gain over the course of the year than the students in the comparison group. The two groups scored nearly identically on the pretest (test of Means, NS), and both made statistically significant knowledge gains over the course of the year. But the Mean posttest score for the Math Connects students was significantly higher than the Mean posttest score for students using other programs (posttest Mean for the comparison group = 580; posttest Mean for the Math Connects group = 588). This difference in posttest scores is statistically significant (t = 2.3, p < .02).

7 While the 69 comparison group teachers who provided final data used a total of 21 different programs, more than half (37) used one of three major programs.

18.2

9.5

19.8

0

10

20

30

40

50

60




Avera

ge K

no

wle

dg

e G

ain

s(o

n T

err

aN

ova

Sca

le S

core

)




The data patterns in grades four and seven suggest that both the comparison and experimental groups progressed in nearly identical ways. Recall that the comparison group in the fourth grade began the year with significantly higher pretest scores than the Math Connects group with an average pretest score 11 points higher than the experimental group. On the posttest, the comparison group again averaged 11 points more than the Math Connects group (comparison group Mean score = 641, experimental group Mean score = 630). This difference was again statistically significant (t =3.7, p < .00). This pattern, of course, suggests that both groups made equal learning gains over the course of the year.

The seventh graders also showed nearly identical patterns in knowledge gained (see Figure further below). The Mean pretest score for the experimental group (667) was a single point higher than the Mean pretest score for the comparison group (666; test of Means NS). Both groups made statistically significant knowledge gains over the course of the year with just two points separating the average posttest scores (posttest Mean for experimental group = 682, posttest Mean for comparison group = 684, NS).

535

588

533

580

500510520530540550560570580590600


Avera

ge S

cale

Sco

re

Student Mathematics Learning by Condition: Grade 2

Math Connects (n=374) Other Programs (n=574)

599

630609

641

580590600610620630640650660670680


Avera

ge S

cale

Sco

re




• The Math Connects materials compared favorably to other programs. While

both groups (Math Connects and the comparison group) showed substantial, statistically significant knowledge gains over time, second grade students using the Math Connects materials made more progress than students using other materials. The average posttest score for this group of students was significantly higher than the average score for students using other programs even though the average pretest scores for those groups were nearly identical. There were no cases where the comparison group showed greater knowledge gains than the Math Connects group.

Subgroup Comparisons. The detailed comparisons between programs were achieved through a slightly different data analysis strategy. A knowledge gain score was calculated for every student by subtracting the student’s pretest TerraNova mathematics scale score from the student’s posttest TerraNova mathematics scale score. This gain score reflected the amount of improvement in students’ scores from pretest to posttest. Program comparisons were accomplished by comparing the magnitude of the knowledge gain scores across conditions. Detailed comparisons are presented first for the elementary grades, followed by parallel comparisons for the middle school students (grade 7).

Elementary Grades. The Figure immediately below displays the data on knowledge gains by gender for both groups for grades two and four.8

8 Sample sizes varied for each comparison, of course. For females in second grade, Math Connects n = 187, comparison group n = 263. For males in second grade Math Connects n = 184, comparison group n = 255. For females in the fourth grade Math Connects n = 230, comparison group n = 275. For males in the fourth grade Math Connects n = 237, comparison group n = 241.

While knowledge gains were essentially equal across conditions, female second graders using Math Connects showed a substantially higher knowledge gain than female second graders using other programs (t = 2.1, p < .04). No other comparisons approached statistical significance.

667 682

666

684

620

630

640

650

660

670

680

690

700

710

720


Avera

ge S

cale

Sco

re




• Results indicated that female students in the second grade who used the Math Connects materials gained more mathematical knowledge than second grade females who used other programs. All other comparisons at the elementary level indicated that both Math Connects and the other programs worked equally well.

For the elementary grades, significance testing of student gain scores across conditions by racial categories indicated that the Black students using Math Connects in grade two showed significantly greater gains than Black students using other programs (t = 2.6, p < .01).9

While no other differences met conventional levels of statistical significance, the Figure below reveals a data pattern where students using Math Connects had numerically greater gains in knowledge than students using other programs in every racial category except one (Black students in grade 4; this difference was not statistically significant either).

9 In grade 2, Asian students using Math Connects n = 24, comparison group = 7; Black students using Math Connects n = 41, comparison group n = 97; Latino students using Math Connects n = 69, comparison group n = 128 and White students using Math Connects n = 192, comparison group n = 252. For grade 4, Asian students using Math Connects n = 6, comparison group n = 11; Black students using Math Connects n = 104, comparison group n = 87; Latino students using Math Connects n = 55, comparison group n = 59 and White students using Math Connects n = 250, comparison group n = 325.

57.149.2

30.3 32.3

48.2 48.1

30.334.2

0

10

20

30

40

50

60

70

Females Grade 2 Males Grade 2 Females Grade 4 Males Grade 4

Avera

ge K

no

wle

dg

e G

ain

s(o

n T

err

aN

ova

Sca

le S

core

)

Knowledge Gains for Elementary Grades by Gender and Condition



• Black students using Math Connects in grade two showed significantly greater knowledge gains than Black second grade students using other programs. Technically, all other comparisons at the elementary level indicated that both Math Connects and the other programs worked equally well. No other differences met standards of statistical significance even though students using Math Connects in almost all other categories showed numerically higher knowledge gains than students using other programs.

Middle School. Comparisons of subgroups within the seventh grade generally indicated that both experimental and comparison groups performed equally well. The Figure below displays the knowledge gain scores by gender. Neither the difference between females’ scores nor the difference between the males’ scores were statistically significant for the seventh graders.10

Complete comparisons by race for the seventh graders were not possible because there were fewer than five Asian students in the comparison group for seventh graders. The Figure below shows the comparisons for Black, Latino and White students.

11

10 For females, n = 375 for the Math Connects group and 169 for the comparison group; t = .52, NS; for males, n = 357 for the Math Connects group and 188 for the comparison group; t = 1.1, NS).

Once again, the major difference was for the Latino students. The Latino students using Math Connects showed a significantly larger gain over the course of the year than students using other programs (n = 40 for the Math Connects group and 73 for the comparison group; t = 3.3, p < .00). None of the other differences in knowledge gain reached statistical significance even though the gains for other programs for Black and for White students were numerically larger for the comparison group than for the Math Connects group.

11 For Blacks, n = 70 for the Math Connects group and n = 29 for the comparison group; t = 1.0, NS; for Whites, n = 573 for the Math Connects group and n = 215 for the comparison group; t = 1.3, NS).

38.8

61.255.2 51.7

37.5

15.5

33.1 34.328

40.9

49.3 50.6

26.6 28.9 31.5 33.2

0

10

20

30

40

50

60

70

Asian Grade 2 Black Grade 2 Latino Grade 2 White Grade 2 Asian Grade 4 Black Grade 4 Latino Grade 4 White Grade 4

Avera

ge K

no

wle

dg

e G

ain

s(o

n T

err

aN

ova S

cale

Sco

re)

Knowledge Gains for Elementary Grades by Race and Condition



Latino students using Math Connects in grade seven showed significantly greater knowledge gains than Latino students using other programs. All other comparisons, by race and by gender at the seventh-grade level, did not meet conventional levels of statistical significance.

16.1 13.818 18.8

0

10

20

30

40

50

60

70

Females Grade 7 Males Grade 7

Avera

ge K

no

wle

dg

e G

ain

s(o

n T

err

aN

ova

Sca

le S

core

)

Knowledge Gains for Grade 7 by Gender and Condition


13.7

52

12

27.4

17.5 16.3

0

10

20

30

40

50

60

70

Black Grade 7 Latino Grade 7 White Grade 7

Avera

ge K

no

wle

dg

e G

ain

s(o

n T

err

aN

ova

Sca

le S

core

)

Knowledge Gains for Grade 7 by Race and Condition



General Summary of Student Learning Results

The data reported in this section of Results provides answers to the first two guiding evaluation questions.

Basic Effectiveness of Math Connects. The first evaluation question asked:

1) Are the Math Connects curriculum materials effective in helping students improve their mathematics abilities?

a) Is Math Connects effective with all types of students?

b) Can Math Connects be used effectively by all types of teachers?

Basic results. Most basically, the student standardized pretest-posttest evaluation of student learning via the TerraNova survey mathematics tests documents that the Math Connects materials can be used to produce substantial, significant gains in student learning of mathematics. Students whose teachers used Math Connects at grades two, four and seven all showed statistically significant knowledge gains from the beginning of the year to the end of the year. Thus, Math Connects appears to be very effective in helping students improve their mathematical abilities.

Student learning by gender. Additional analyses verified that Math Connects was equally effective for both males and females. At all grade levels, students whose teachers used Math Connects showed statistically significant gains from pretest to posttest. Math Connects was equally effective for students of both genders across grade levels.

Student learning by race. Math Connects also proved effective with students of all races. When analyzed by race, the TerraNova data showed that Asian, Black, Latino and White students who used Math Connects all showed substantial, statistically significant knowledge gains over the course of the school year. Unexpectedly, seventh grade Latino students showed an exceptionally large knowledge gain over the course of the year.

Student learning by reading level. All students were tested for reading ability using the TerraNova. Results showed that students of all reading levels who used the Math Connects materials had functionally equivalent knowledge gains in mathematics. In other words, the Math Connects materials proved to be effective for students regardless of reading ability. This was the case at all grade levels.

Students with special needs. Student learning was analyzed separately for students requiring special accommodations due to a learning disability requiring an Individual Education Plan (IEP) or requiring a literacy plan (LP) usually due to the fact that they are English Language Learners. At the elementary level, these special students using Math Connects showed gains in mathematical knowledge equal to, or slightly greater than other students. At the seventh grade level, these students showed virtually no knowledge gain over the course of the year.

Teachers’ self efficacy. Especially at the elementary levels, teachers may vary in their assessments of their own abilities to teach mathematics and it is reasonable to suspect that


their confidence level (or self efficacy) may affect student learning. All teachers in the study completed a mathematics teaching self efficacy scale to assess their level of confidence in teaching math. Analyses showed that, at the elementary levels, student learning was essentially equivalent regardless of teacher self efficacy. In other words, teachers of all levels of confidence about their abilities to teach math achieve essentially equivalent student outcomes. At the seventh grade level, students of teachers in the midlevel category of self efficacy showed the largest learning gains. Thus, at the middle school level, teachers’ self efficacy may mediate the effectiveness of the Math Connects materials.

Teachers’ math beliefs. Teachers have different philosophies of teaching and, of course, hold varying beliefs about how students learn mathematics and how mathematics can best be taught. All teachers in this study completed a math beliefs scale that identified their basic orientation toward teaching math. For the most part, students who used Math Connects showed functionally equivalent knowledge gains across forms of teachers’ math beliefs. For second graders and seventh graders, there were no statistically significant differences among Means. However, for the fourth grade students using Math Connects, the students of teachers with the lowest scores on the MBS showed significantly greater knowledge gains than the students of teachers with medium or high scores Thus at the fourth grade level, the Math Connects teachers with mostly traditional beliefs about student learning of mathematics showed the largest knowledge gains.

Teachers’ reform-based practices. Teachers engage in a variety of different practices in their efforts to teach mathematics. All teachers in this study completed regular activity logs in which they recorded how regularly they used common “reform-based” practices. Teachers’ reports were averaged over time to allow for teachers to be classified according to their tendencies to use reform-based practices. For both the second grade and fourth grade, the learning gains were consistent for students who used Math Connects regardless of teachers’ propensity to use reform-based practices. However, in the seventh grade, the Math Connects students whose teachers reported a medium level of reform-based practices showed significantly lower knowledge gains than teachers who reported high levels of reform-based practices.

Comparing Math Connects with Other Mathematics Programs. The second evaluation focused on how the effectiveness of Math Connects compares to the effectiveness of other programs:

2) Does student learning of mathematics differ between students whose teachers used Math Connects and students whose teachers used other curriculum materials?

Basic program comparisons. General comparisons between the experimental and comparison conditions indicate that second grade students using Math Connects achieved significantly higher knowledge gains in mathematics than students using other programs. The knowledge gains for the fourth and seventh grades were nearly identical between the Math Connects group and the comparison group. There were no cases where the comparison group students scored significantly higher than the Math Connects students.

Student subgroups. Program comparisons by gender revealed that female second graders using Math Connects made significantly larger knowledge gains than female second graders using other programs. All other comparisons focused on gender (at all grade levels) indicated


that both the Math Connects and the other programs performed equally well. There were no cases where the comparison group students scored significantly higher than the Math Connects students.

Comparisons by racial category revealed that Black students in the second grade who used Math Connects made significantly higher knowledge gains than Black second graders using other programs. While no other comparisons met conventional tests of statistical significance, the pattern in the data revealed numerically higher gain scores for elementary level Math Connects students than for elementary students who used other programs.

Analyses by gender at the middle school level indicated that both Math Connects and other programs worked equally well. Comparisons by racial category revealed that Latino students using Math Connects showed significantly higher gain scores in mathematics than Latino students using other programs.


Results: Evaluations of K- 5 Curriculum Materials

While the valid assessment of student learning outcomes is certainly an essential component of any evaluation of educational materials, it may not be reasonable to expect any program to yield robust changes in student learning scores after only its first year of implementation. Common sense suggests that teachers need time to learn how to use any program, especially a genuinely innovative program, to its fullest advantage. Thus, a second major dimension of this national field study was the direct assessment of teachers’ evaluations of program materials. Our assumption is that the teachers who use the materials are expert judges of the programs’ effectiveness. Their opinions should be considered along with the standardized testing (see U.S. Department of Education, 2008; The Final Report of the National Mathematics Advisory Panel). Existing research suggests that effective curriculum materials should have several key characteristics. Teachers were asked to provide their general assessments of their materials and also asked to assess the global characteristics of their curriculum materials including:

• The extent to which the materials addressed essential curriculum focal points.

• The degree to which the materials are vertically aligned.

• The effectiveness of the materials in facilitating balanced instruction.

o This would include effectiveness in teaching applied problem solving.

o This would also include materials to facilitate cross-curricular connections.

• The extent to which the materials supported differentiated instruction.

• The general effectiveness of the digital resources available with the materials.

• The effectiveness of the diagnostic assessment and intervention resources.

• The effectiveness of the formative and summative assessment resources.

• The effectiveness of the materials in helping students develop mathematical literacy.

• The value of the professional development resources provided with the materials.

• The effectiveness of the materials in facilitating blended instruction.

The results in this section are for the elementary grades only. The results for the seventh grade materials are reported in a separate section of this report. Within each major section, the most general assessment data is presented first, followed by more concrete evaluations of specific components.


Elementary Teachers’ General Assessments

All teachers were asked to provide their overall impressions of the effectiveness of the materials they used over the course of the school year. Teachers provided both open-ended qualitative data and closed-ended quantitative data. These general assessments are presented first, followed by more focused assessments of specific functions and components of their curriculum materials.

Qualitative Comments about Program Strengths. One strategy for identifying the strengths of any set of materials is to solicit opinions in a very open, general manner. The assumption is that people will express their most salient attitudes first and thus reveal what has made the most impact on their attitudes. Teachers’ assessments of the Math Connects materials began with a general open-ended question asking them to identify their most positive experiences over the course of the year. To avoid priming the teachers to comment on any specific components, this question was asked before any other specific questions were presented. The intention was to discern the teachers’ most salient impressions about the strengths of the Math Connects program. The Figures below present representative samplings of both second grade and fourth grade teachers’ most positive experiences with Math Connects. The full text of all comments from both the Math Connects teachers and the comparison group teachers is included in an appendix. Teachers were also asked to identify their most negative experiences with their materials. The full text of all of these comments is available in an appendix.

While the teachers’ comments covered a wide variety of topics, many commented on both the Teacher Edition and the Student Edition as being very useful. Many also indicated that the manipulatives included in the program were very helpful to their students and that the regular assessment materials were helpful to them. When prompted for negative feedback, many teachers indicated they did not have any complaints. The most common negative comments centered on typographical errors it the new materials or being overwhelmed with the variety of resources available to them. Teachers complained about not having time to use many of the resources. Several also mentioned having trouble implementing the digital resources provided.


Selected Math Connects Second Grade Teachers’ Positive Experiences

We would like to hear about your most positive experiences. Generally, what materials were most helpful, what did you like using and what truly added to your students’ learning? Please explain as much as you can.

“I loved it. The Teacher Edition was very easy to follow. The colorful student books were very helpful. I liked how each chapter had a workbook for extra worksheets. Overall, I plan to use this book instead of our school’s current book.”

“The curriculum correlated directly to the math standards so I did not have to create my own lessons to fill any gaps. The hands-on manipulatives gave students a way to understand using concrete items. Many children enjoyed using the items even after they could solve the problems on their own without the manipulatives…The workbooks were engaging and very practical; the students showed a lot of enthusiasm for the data unit where they collected their own data.”

“I loved using the Math Connects materials this year. It was a huge change from our school adopted math series. The lessons were not scripted as with (another program), but contain many ways to provide instruction. I liked the opportunity for teacher choice!...The inclusion of two problem-solving lessons per chapter was wonderful! We spent a lot more time exploring problem solving this year…The student textbooks were colorful, exciting and provided lots of practice. I think the manipulatives were a great addition.”

“Most of my students were able to master the base ten system up to 100. The previous year this was difficult for my students and this year with the student workbooks and the manipulatives my students were able to understand the tens better. I feel that the idea of adding concepts that we use in reading, such as the main idea and the vocabulary section to workbooks also helped my students understand that math has as much value as reading does in their educational process.”

“I liked the sequence of the curriculum. I have worked with other curriculums where you must jump around to provide students with some background knowledge / experience before they tackle another concept. Math Connects was user-friendly in the way that we could move from the beginning and move right through – I did not find any holes!”

“The Teacher Edition, Homework Practice Book and the Skills Practice Book were the materials that I utilized the most. I found the Teacher Edition to be comprehensive and extensive, and it provided me with a great deal of information beyond the initial lessons that I was teaching. The consumable books were a wonderful way to allow the children to practice at their own pace while feeling successful. I could differentiate the material by allowing the children to use either the reteach pages or the practice pages, or both as a challenge. In addition, I used the unit books to supplement material for my above-average students and was able to pull what I needed from each of the books. The material was clear, concise, and most times easy for the children to use independently. Overall, the materials that were provided allowed my students to thrive this year in their understanding of mathematical concepts.”

“I liked the ESL component, supplemental books and colorful textbooks.”


Selected Math Connects Fourth Grade Teachers’ Positive Experiences


“I have to start by saying I truly love this series. Compared to what I used in the past it is so much more teacher-friendly and student effective. One of my favorite things is the Are You Ready? pre-chapter check. It is nice to know what knowledge my students have before I begin a new concept. I also really find the Mid-Chapter Check useful. The alignment with (state) standards and seeing how deep a lesson hits them is great. The manipulatives are wonderful; I used them more than I ever had in the past. I also love the real-world connections this book brings to my students. We learned so many other things while doing math.”

“It was a great program!!! I had many compliments on the program from parents and other teachers. The textbooks were great for the kids and offered many opportunities for differentiated learning. I loved having the resource worksheets for each chapter.”

“I like the Teacher Edition and the layout is easy to use. I like the differentiation part and the real-world examples. I also found the alignment to the state standards page quite useful. The student books are adequate and colorful. I think they make math easy for students to understand.”

“I thought the textbook was effectively organized because if it had to be sent home, parents could easily follow what the lesson was, as could the child. Regarding the Teacher Edition, I liked the organization of Introducing, Teaching, Practicing and Assessing the lesson that are along the margins. I also appreciated the Common Error notations, as well as the Alternative Teaching Strategies. I thought the Get Ready to Learn at the beginning of each lesson was thorough, and the Are You Ready? gave me an indication of how I needed to structure the lessons and for whom.”

“The most effective component was the bag of student manipulatives. These inspired enthusiasm in the students and provided clear practice for their math concepts.”

“I was very pleased with all components of the Math Connects curriculum. I loved the manipulatives kits the most. I was able to use them to improve my teaching strategies and for my students to visualize how to do each type of problem. I used many of the teacher books with my students and really liked that they were differentiated. My favorite part of the homework practice worksheets was that there was a spiral review at the bottom of each worksheet.” “The technology pieces were most helpful. The students enjoyed the math games. The hands-on material was also helpful.” “I liked the pre-tests before each concept to get a general understanding of what the students knew. Real-world application with manipulatives excited my students, and the exploration helped them master curriculum…The parents appreciated that they could reference the book to understand the lessons taught in class.”


NCTM Standards. The National Council of Teachers of Mathematics (NCTM) has identified curriculum focal points for every grade level (NCTM, 2006). That is, they have identified three major objectives that mathematics curricula should address at each level. One primary objective of this evaluation was to assess the extent to which teachers perceived the Math Connects materials as grounded in these national standards. If teachers are expected to address NCTM standards and accomplish NCTM curriculum objectives (as expressed in the focal points) their materials must be written and structured in ways that facilitate these goals.

All teachers in this study were asked to evaluate the extent to which the materials they used addressed the NCTM focal points for their grade level. Specifically all teachers were asked:

“These questions ask you about the general content focus of the curricular materials that you used this year. Please take a moment and reflect on those materials overall and then respond to the items below. Remember, we are asking you about the materials themselves and how they facilitated your teaching in these different areas.”

Then the teachers responded to statements which asked directly about the substantive content of each curricular focal point for their grade level (verbatim items are displayed in Tables below). Teachers made a judgment on a scale of 1-4 (1=not at all effective, 4=very effective) of the extent to which their materials helped students achieve each curriculum objective.

Teachers’ responses indicate that Math Connects is very effective at addressing the NCTM curricular focal points. The Tables below display the frequency of responses for all teachers in both the experimental and comparison conditions. The responses indicate that more second grade teachers using Math Connects felt that their materials were effective at addressing the NCTM curriculum focal points than teachers using other materials. For the second grade focal point 1 (understanding base-ten and place values), 96% of all Math Connects teachers indicated their materials were either “mostly effective” or “very effective” with the majority giving the highest rating. Only 60% of the teachers using other programs gave one of the two highest ratings. For second grade focal point 2 (recall of addition and subtraction and fluency with multi-digit operations) an equal percentage of teachers in both groups indicated high levels of effectiveness. But, a substantially higher percentage of teachers using other programs gave the lowest ratings for their programs. Finally, for the third NCTM second grade focal point (linear measurement and facility in measuring lengths), 72% of the Math Connects teachers indicated high levels of effectiveness while only 43% of teachers using other programs did so.


Second Grade Teachers’ Assessments Regarding NCTM Focal Point 1

How effective were your materials in helping students to develop an understanding of the base- ten numeration system and place value concepts?

Not at all effective

Somewhat effective

Mostly effective

Very effective

Math Connects Teachers (n=25) 0% 4% 44% 52% Teachers using other programs (n=30)

10%

30%

40%

20%


How effective were your materials in helping students develop quick recall of addition facts and related subtraction facts and fluency with multi- digit addition and subtraction?


Somewhat effective

Mostly effective

Very effective

Math Connects Teachers (n=25) 8% 56% 28% 8%

Teachers using other programs (n=30)

33%

30%

13%

23%


How effective were your materials in helping students develop an understanding of linear measurement and facility in measuring lengths?


Somewhat effective

Mostly effective

Very effective



17%

40%

40%

3%

More fourth grade teachers also indicated that the Math Connects materials were effective at addressing the NCTM focal points for fourth grade than teachers using other materials. The response frequency data for each of the grade four NCTM focal points is displayed in the Tables below. For the first fourth grade focal point (fluency with multiplication and related division facts), 40% of the Math Connects teachers gave the highest ratings while only 31% of the other teachers did so. Nearly a third of the comparison group


teachers indicated that their programs were “not at all effective” in addressing this focal point while only 7% of the Math Connects teachers had that opinion. Regarding the second focal point for fourth grade (the understanding of decimals), 61% of the Math Connects teachers gave the highest ratings for effectiveness while only 38% of the comparison group teachers did so. Once again, 31% of the teachers using other programs felt their materials were “not effective at all” in addressing this focal point. None of the Math Connects teachers responded in that manner. The frequency data for the third NCTM focal point for fourth grade (understanding area) suggests that Math Connects is much stronger in this area than other programs. First, 74% of the Math Connects teachers indicated high levels of effectiveness in this area while only 24% of teachers using other programs rated their programs this highly. And, once again, 31% of teachers using other programs indicated that their programs were “not effective at all.” Once again, none of the Math Connects teachers had that opinion.

Fourth Grade Teachers’ Assessments Regarding NCTM Focal Point 1

How effective were your materials in helping students develop quick recall of multiplication facts and related division facts and fluency with whole number multiplication?


Somewhat effective

Mostly effective

Very effective



31%

38%

21%

10%


How effective were your materials in helping students develop an understanding of decimals, including the connections between fractions and decimals?


Somewhat effective

Mostly effective

Very effective



31%

31%

28%

10%



How effective were your materials in helping students develop an understanding of area and determining the areas of two- dimensional shapes?


Somewhat effective

Mostly effective

Very effective



31%

45%

14%

10%

• Elementary school teachers using Math Connects clearly felt that their

materials were more effective at addressing the curriculum focal points specified by the National Council of Teachers of Mathematics than did teachers using other materials. Both second grade and fourth grade teachers consistently indicated that Math Connects was effective at addressing specific focal points for their grade levels. Teachers using other programs frequently indicated their programs were not at all effective in addressing these focal points.

Focused Assessments of Basic Characteristics. A substantial part of this evaluation addressed some of the most basic characteristics of any set of curriculum materials. First, all teachers in the study completed an initial series of closed-ended items focused on general characteristics of curriculum materials that are known to be of central interest to teachers. Specifically, these items asked about pragmatic factors that have direct implications for teachers’ daily practices (ease of implementation, program pacing, assistance in lesson planning and the adequacy of practice materials). Also, in keeping with the priorities advanced by the NCTM, they were asked to make a general assessment of how well their materials helped them prepare students to learn algebra. Finally, teachers were asked to report on the extent to which their materials seemed to be engaging their students in the study of mathematics.

The Figures below present the Mean responses on the items centered on teachers’ pragmatic interests. Results indicate that, regarding the most practical issues for teachers, the teachers who used Math Connects were much more positive about their materials than teachers who used other programs (see Figures below). First, the teachers who used Math Connects were significantly more positive about ease of implementation than teachers who used other programs. In both second and fourth grade their average responses to the item asking about implementation were significantly more positive than those of teachers using other programs (Grade two: Math Connects n = 25, comparison group n = 31, t = 4.1, p < .00; for grade four: Math Connects n = 27, comparison group n = 29, t = 5.8, p < .00).

Second, the Math Connects teachers were also significantly more positive about the pacing of their program (Grade two: Math Connects n = 25, comparison group n = 31, t = 5.5, p < .00; for grade four: Math Connects n = 27, comparison group n = 29, t = 5.1, p < .00).


Finally, the Math Connects teachers were significantly more positive about the help they received in planning their lessons than were teachers who used other programs (Grade two: Math Connects n = 25, comparison group n = 31, t = 2.8, p < .00; for grade four: Math Connects n = 27, comparison group n = 29, t = 5.1, p < .00).

3.6 3.5

2.8

2.5

1

1.5

2

2.5

3

3.5

4

Grade 2 Grade 4

Avera

ge R

esp

on

ses

(1=

Str

on

gly

Dis

ag

ree...

4=

Str

on

gly

Ag

ree)

Elementary Grade Teachers: "The program I used to teach math this year was easy to implement."


3.4

2.9

2.4

2.1

1

1.5

2

2.5

3

3.5

4

Grade 2 Grade 4

Avera

ge R

esp

on

ses

(1=

Str

on

gly

Dis

ag

ree...

4=

Str

on

gly

Ag

ree)

Elementary Grade Teachers: "The pacing of the program I used to teach math this year is excellent."



Teachers responded to three additional general items that were more content oriented in nature (see Figures below). First, teachers provided a global judgment of the reading level of the materials they used. Second grade teachers who used Math Connects were significantly more positive about the appropriateness of the reading level than teachers using other programs (Math Connects n = 25, comparison group n = 30, t = 3.0, p < .00). While the average response for the fourth grade teachers using Math Connects was higher than the average for teachers using other programs, the difference was not statistically significant (Math Connects n = 27, comparison group n = 29, t = .9, NS).

Results also indicate that teachers in both grades two and four who used Math Connects were significantly more positive about the amount of practice provided for students than were teachers who used other materials (Grade two: Math Connects n = 25, comparison group n = 31, t = 3.1, p < .00; for grade four: Math Connects n = 26, comparison group n = 29, t = 4.3, p < .00).

3.63.3

3.1

2.5

1

1.5

2

2.5

3

3.5

4

Grade 2 Grade 4

Avera

ge R

esp

on

ses

(1=

Str

on

gly

Dis

ag

ree...

4=

Str

on

gly

Ag

ree)

Elementary Grade Teachers: "The program I used to teach math this year includes very helpful suggestions and guides for planning my lessons."


3.33

2.7 2.8

1

1.5

2

2.5

3

3.5

4

Grade 2 Grade 4

Avera

ge R

esp

on

ses

(1=

Str

on

gly

Dis

ag

ree...

4=

Str

on

gly

Ag

ree)

Elementary Grade Teachers: "The program I used to teach math this year was written at the appropriate reading level for my students."



Finally, teachers were asked directly about the extent to which they felt their materials helped them prepare their students to learn algebraic concepts. The teachers who used Math Connects were significantly more positive about their materials than teachers who used other programs (Grade two: Math Connects n = 24, comparison group n = 31, t = 5.0, p < .00; for grade four: Math Connects n = 25, comparison group n = 29, t = 4.9, p < .00).

• Elementary teachers’ opinions of Math Connects were very positive on several important pragmatic dimensions. Teachers using Math Connects were more positive about ease of implementation than teachers using other programs. They were more positive about the pacing of the program and the assistance provided to them in planning their lessons. Second grade teachers using Math Connects were significantly more positive about the reading level of their materials than teachers using other programs. The teachers using Math Connects were also more positive than teachers using other programs about the amount of practice problems provided for their students and, finally, they were significantly more positive about the preparation their students received for learning algebraic concepts.

3.3 3.3

2.7

2.4

1

1.5

2

2.5

3

3.5

4

Grade 2 Grade 4

Avera

ge R

esp

on

ses

(1=

Str

on

gly

Dis

ag

ree...

4=

Str

on

gly

Ag

ree)

Elementary Grade Teachers: "The program I used to teach math this year includes a sufficient amount of practice problems for students."


3.5 3.4

2.72.6

1

1.5

2

2.5

3

3.5

4

Grade 2 Grade 4

Avera

ge R

esp

on

ses

(1=

Str

on

gly

Dis

ag

ree...

4=

Str

on

gly

Ag

ree)

Elementary Grade Teachers: "The program I used to teach math this year is very effective in preparing my students to learn algebraic concepts."



Teachers also reported on their perceptions of their students’ affective reactions to studying mathematics during the school year. Specifically, one component of their biweekly log entries asked them to report on their perceptions of their students’ level of engagement in mathematics. That is, every two weeks the teachers responded on a 1-4 scale (strongly disagree to strongly agree) to a series of statements designed to assess how their students were reacting to the study of math. The Figures below display the data for the teachers’ average responses over the course of the year.12

Thus overall, teachers using Math Connects reported very positive perceptions of how their students were responding to learning mathematics. This was also generally the case with the teachers using other programs. While some specific comparisons did yield statistical significance, the actual difference in Mean scores was not particularly substantial.

The data is displayed for each item by grade level and by condition. Results are identical for the first two questions. For the item regarding student interest in mathematics the average response for the second grade Math Connects teachers was significantly higher than the average for teachers in the comparison group (t = 2.1, p < .04). This was also the case for the item regarding student enjoyment of math. The Mean response for the second grade Math Connects teachers was significantly higher than the Mean for the comparison group teachers (t = 2.0, p < .05). There was no significant difference at the fourth grade level.

12 For the “interest in math” item, grade 2 Math Connects n = 24, comparison group n = 29. Grade 4 Math Connects n = 22, comparison group n = 28. For the “enjoying math” item, grade 2 Math Connects n = 24, comparison group n = 29. Grade 4 Math Connects n = 22, comparison group n = 28. For the “actively engaged” item, grade 2 Math Connects n = 24, comparison group n = 29. Grade 4 Math Connects n = 22, comparison group n = 28. For the “more positive” item, Grade 2 Math Connects n = 24, comparison group n = 29. Grade 4 Math Connects n = 22, comparison group n = 28.

3.43.1

3.23.1

1

1.5

2

2.5

3

3.5

4

Grade 2 Grade 4

Avera

ge R

esp

on

ses

(1=

Str

on

gly

Dis

ag

ree...

4=

Str

on

gly

Ag

ree)

Elementary Grade Teachers: "My students seem very interested in learning mathematics."



3.43.1

3.23.1

1

1.5

2

2.5

3

3.5

4

Grade 2 Grade 4

Avera

ge R

esp

on

ses

(1=

Str

on

gly

Dis

ag

ree...

4=

Str

on

gly

Ag

ree)

Elementary Grade Teachers: "My students are enjoying their mathematics lessons."


3.43.2

3.33.2

1

1.5

2

2.5

3

3.5

4

Grade 2 Grade 4

Avera

ge R

esp

on

ses

(1=

Str

on

gly

Dis

ag

ree...

4=

Str

on

gly

Ag

ree)

Elementary Grade Teachers: "My students are actively engaged in their mathematics lessons."


3.43.1

3.23.1

1

1.5

2

2.5

3

3.5

4

Grade 2 Grade 4

Avera

ge R

esp

on

ses

(1=

Str

on

gly

Dis

ag

ree...

4=

Str

on

gly

Ag

ree)

Elementary Grade Teachers: "My students are becoming more positive about learning mathematics."



Assessments of Primary Resources. The basic assessment of curriculum materials included a straightforward set of questions centered on teachers’ general perceptions of the quality of the Teacher Edition of their basal program and also their perceptions of the effectiveness of the Student Edition of their basal program. All teachers in the study responded to direct questions about the effectiveness of the Teacher Edition and the Student Edition of the program they used. The Figure below displays the response data for teachers’ perceptions of their Teacher Editions.

The teachers who used Math Connects in both second grade and fourth grade were extremely positive about their Teacher Editions. Teachers’ ratings of helpfulness at both grade levels were significantly higher than the helpfulness ratings of teachers using other programs. For second grade, Math Connects n = 25, comparison group n = 31, t = 6.4, p < .00. For fourth grade, Math Connects n = 27, comparison group n = 29, t = 6.9, p < .00.

Teachers were also asked to rate how easy, or difficult, it was to use their Teacher Edition. The data displayed in the Figure below indicates that the teachers using Math Connects were significantly more positive about the ease of use of their Teacher Edition than teachers using other programs. For second grade, Math Connects n = 25, comparison group n = 31, t = 3.9, p < .00. For fourth grade, Math Connects n = 27, comparison group n = 28, t =4.5, p < .00.

3.8 3.7

2.9

2.5

1

1.5

2

2.5

3

3.5

4

Grade 2 Grade 4

Avera

ge R

esp

on

ses

(1=

No

t at

all

Help

ful…

4=

Extr

em

ely

Help

ful)

Elementary Grade Teachers: "All things considered, how helpful is the Teacher Edition of the program you used to teach math this year?"



Teachers using Math Connects were also very positive about the Student Edition of the basal Math Connects program. The Figure below displays the teachers’ ratings of the helpfulness of the Student Edition in the programs they used. For second grade teachers, Math Connects n = 25, comparison group n = 31, t = 7.2, p < .00. For fourth grade, Math Connects n = 27, comparison group n = 29, t =6.3, p < .00.

All teachers also rated the Student Edition of their programs to assess how easy, or difficult, it was for their students to use that textbook. The teachers using Math Connects were significantly more positive about the Student Edition of Math Connects than teachers using other programs were about their Student Editions (see Figure below). For second grade teachers, Math Connects n = 25, comparison group n = 30, t = 5.1, p < .00. For fourth grade, Math Connects n = 27, comparison group n = 28, t =4.5, p < .00.

3.5 3.6

2.8 2.8

1

1.5

2

2.5

3

3.5

4

Grade 2 Grade 4

Avera

ge R

esp

on

ses

(1=

Very

Dif

ficu

lt…

4=

Very

Easy

)

Elementary Grade Teachers: "All things considered, how would you rate the ease of use of the Teacher Edition of the program you used to teach math this

year?"


3.93.6

2.72.5

1

1.5

2

2.5

3

3.5

4

Grade 2 Grade 4

Avera

ge R

esp

on

ses

(1=

No

t H

elp

ful

at

all

…4

=Extr

em

ely

Help

ful)

Elementary Grade Teachers: "All things considered, how helpful is the Student Edition of the program you used to teach math this year?"



During the posttest period of the study, a very brief questionnaire was distributed to all the fourth grade students in the study. One item asked them directly about their opinion of their math textbook. Specifically, students responded on a 3-point scale (1=I do not agree,

2=I am not sure, 3=I agree) to the statement, “I like my math book.” The Figure below displays results for all fourth grade students in both conditions. Although the difference in ratings may not be considered substantial, results indicate that the fourth grade students whose teachers used Math Connects were significantly more positive about their math book than the students of teachers using other texts (Math Connects n = 626, comparison group n = 643, t = 6.3, p < .00).

Elementary school teachers using Math Connects were very positive about both the Teacher Edition and the Student Edition of the Math Connects program. Both second grade and fourth grade teachers’ ratings of the helpfulness of their Teacher Edition were significantly higher for teachers using Math Connects than teachers using other programs. Teachers using Math Connects also rated their Teacher Edition as easier to use than teachers using other programs. Teachers using Math Connects were very strongly positive about the helpfulness of the Student Edition in Math Connects. Their ratings of the helpfulness of this text were significantly higher than the ratings provided by teachers using other programs. This was also the case for the teachers’ ratings of how easy the Student Edition was to use. Teachers using Math Connects were significantly more positive about the ease of use of the Student Edition than teachers using other programs.

3.7 3.6

2.92.8

1

1.5

2

2.5

3

3.5

4

Grade 2 Grade 4

Avera

ge R

esp

on

ses

(1=

Very

Dif

ficu

lt…

4=

Very

Eeasy

)

Elementary Grade Teachers: "All things considered, how would you rate the ease of use for the Student Edition of the program you used to teach math this

year?"


2.52.3

1

1.5

2

2.5

3

Avera

ge R

esp

on

ses

(1=

I d

o n

ot

Ag

ree…

3=

I A

gre

e)

Fourth Grade Students: "I like my math book."



Focused Assessments of Curriculum Materials

The Math Connects curriculum materials were evaluated in detail through the online posttest questionnaire completed by all teachers in the study. Teachers were asked detailed questions about their perceptions of the each major component of their mathematics program.

Vertical Alignment. One of the most primary characteristics of any curriculum is its vertical alignment. In their articulation of principles for effective mathematics education, The National Council of Teachers of Mathematics (NCTM) asserts that “(a) curriculum is more than a collection of activities; it must be coherent, focused on important mathematics, and well articulated across grades.” (NCTM Principles and Standards; see www.nctm.org). They go on to assert that “(i)n a coherent curriculum, mathematical ideas are linked to build upon one another so that students’ understanding and knowledge deepen and their ability to apply mathematics expands.” The Teacher Reference Handbook of the Math Connects program states that Math Connects is vertically aligned into an articulated, coherent sequence of content. One important objective of this study was to test that assertion directly by asking all teachers in the study to evaluate the vertical alignment of the mathematics curriculum they used during the school year. They were asked their assessment directly.

Results indicate that elementary school teachers using the Math Connects program were more likely to agree that their program was vertically aligned in an effective manner than teachers using other programs (see Figure below). In both grades two and four, the average response for the Math Connects teachers was significantly higher than the average for teachers using other programs. For second grade, the Mean for the Math Connects teachers (n=25) was 3.2 while the Mean for teachers using other programs (n=29) was 2.6 (t = 3.0, p < .00). For the fourth grade, the Mean for the Math Connects teachers (n=25) was 3.0 and the Mean for teachers using other programs (n=25) was 2.4 (t = 2.2, p < .03).

Elementary school teachers using Math Connects were very positive about

the vertical alignment of the Math Connects program. Results suggest that, according to elementary school teachers, the vertical alignment of Math Connects is superior to the vertical alignment of other programs.

3.2 3

2.62.4

1

1.5

2

2.5

3

3.5

4

Grade 2 Grade 4

Avera

ge R

esp

on

ses

(1=

Str

on

gly

Dis

ag

ree...

4=

Str

on

gly

Ag

ree)

Elementary Grade Teachers: "The program I used to teach math this year is vertically aligned in a very effective way."



Balanced Instruction. Effective curricular materials enable teachers to provide balanced instruction. That is, to speak very generally, effective instruction requires that teachers provide a coherent blend of holistic learning activities along with direct instruction and other applications (see, for example Stanulis & Floden, 2009). The pre-development research that provided the foundation for Math Connects indicates that the program was intended to help teachers in their efforts to provide balance in their instruction (Edwards, 2009). The Teacher Reference Handbook of the Math Connects asserts that the program facilitates balanced instruction by providing resources for teaching conceptual understanding, computational and procedural skills and real-world problem solving within every unit. More generally, balanced instruction can be facilitated by integrating cross-curricular connections that link mathematical concepts and practices to other subjects of interest to students.

Teachers’ perceptions relevant to balanced instruction were assessed in a number of ways. First, three closed-ended items on their posttest questionnaire asked them to assess the extent to which their materials provided resources for teaching conceptual understanding, computational skills and applied problem solving. The assumption is that a balanced program would be evaluated as equally effective on all these dimensions. The Figures below present the teachers’ responses by grade level and by condition.

Perhaps the primary characteristic of balanced materials in mathematics is that they facilitate both the teaching of conceptual understanding and computational procedures. In both grades two and four teachers using Math Connects rated their materials very highly on both dimensions (near 3.4 on a 4-point scale). Thus both the second and fourth grade teachers who used Math Connects indicated it was very effective at teaching both conceptual understanding and computational procedures. Also, teachers using Math Connects rated their materials higher than teachers using other programs on both dimensions. These differences are substantial and statistically significant. For the item focused on conceptual understanding: in grade two, Math Connects n = 25, comparison group n = 31, t = 2.9, p < .00; for grade four, Math Connects n=27, comparison group n = 29, t = 4.5, p < .00. For the item focused on computational procedures: in grade two, Math Connects n = 25, comparison group n = 31, t = 4.0, p < .00; for grade four, Math Connects n = 27, comparison group n = 29, t = 5.1, p < .00.

3.4 3.32.9

2.5

1

1.5

2

2.5

3

3.5

4

Grade 2 Grade 4

Avera

ge R

esp

on

ses

(1=

Str

on

gly

Dis

ag

ree…

4=

Str

on

gly

Ag

ree)

Elementary Grade Teachers: "The program I used to teach math this year provides excellent resources for teaching conceptual understanding."



An important third dimension of balanced instruction is the teaching of how to apply mathematical concepts and procedures to solve practical, “real-world” problems. Results indicate that teachers using Math Connects felt that Math Connects provided excellent resources for teaching applied problem solving. For the question about real-world problem solving (Figure immediately below) the Math Connects teachers’ ratings were significantly higher than the ratings of teachers using other programs for both grades two and four (in grade two, Math Connects n = 25, comparison group n = 31, t = 5.3, p < .00; for grade four, Math Connects n = 27, comparison group n = 29, t = 5.4, p < .00.) For the item on hands-on activities the difference for the fourth grade was statistically significant (Math Connects n = 26, comparison group n = 29, t = 5.6, p < .00) while the difference in the second grade fell short of conventional levels of statistical significance (Math Connects n = 25, comparison group n = 31, t = 1.8, p < .07).

3.23.4

2.6 2.6

1

1.5

2

2.5

3

3.5

4

Grade 2 Grade 4

Avera

ge R

esp

on

ses

(1=

Str

on

gly

Dis

ag

ree…

4=

Str

on

gly

Ag

ree)

Elementary Grade Teachers: "The program I used to teach math this year provides excellent resources for teaching computational and procedural

skills."


3.5 3.4

2.62.3

1

1.5

2

2.5

3

3.5

4

Grade 2 Grade 4

Avera

ge R

esp

on

ses

(1=

Str

on

gly

Dis

ag

ree…

4=

Str

on

gly

Ag

ree)

Elementary Grade Teachers: "The program I used to teach math this year contains very effective 'real- world' problems that help students learn how to

apply mathematical knowledge."



Elementary school teachers using Math Connects indicated that Math Connects is effective at providing basic balanced instruction. These teachers indicated that Math Connects is very effective at helping them teach both conceptual understanding and computational procedures. Both the second grade and fourth grade teachers who used Math Connects also indicated that Math Connects provides effective resources for teaching practical, real- world problem solving by applying mathematical concepts. Teachers who used Math Connects provided significantly higher assessments of the Math Connects materials than teachers who used other programs when asked about effectiveness in teaching conceptual understanding, computational procedures and resources to teach applied problem solving.

The quantitative data reported above was supplemented by asking teachers to comment in an open-ended fashion about the effectiveness of the materials they had used in helping students develop problem solving skills. Twenty four of 25 second grade teachers who used Math Connects provided comments while all 27 fourth grade teachers who used Math Connects did so. All responses were generally categorized as “positive,” “negative,” or “irrelevant” (teachers would sometimes comment in ways that did not address the question) by two independent coders. Disagreements were resolved by discussion. Representative comments are displayed from both second and fourth grade teachers who used Math Connects in the Figures below. Positive and negative comments are included in proportion to their general frequency. The full text of all comments by all teachers is available in an appendix.

3.4 3.43.1

2.3

1

1.5

2

2.5

3

3.5

4

Grade 2 Grade 4

Avera

ge R

esp

on

ses

(1=

Str

on

gly

Dis

ag

ree…

4=

Str

on

gly

Ag

ree)

Elementary Grade Teachers: "The program I used to teach math this year contains very effective hands- on activities that really help the students learn

math."



Math Connects Second Grade Teachers’ Comments on Problem Solving

Please comment on the effectiveness of Math Connects in helping students develop real- world problem solving skills. You may include both positive and negative comments. Be as specific as you can.

“This was one of the BIG pluses for me. I liked the many problem-solving lessons, and I feel that my class was much more prepared for working out problems than in years past. The two lessons in each chapter allowed students multiple practice sessions using the many strategies that were taught and yet in each of these practices there was no one right way to solve the problems. We used the Problem of the Day to begin each day’s instruction. Sometimes we worked on them independently, sometimes with a partner and sometimes in small groups.” “I used the Problem of the Day frequently. Children wrote responses in their math journals. I can say with confidence that the children learned to use the problem-solving strategy of circling the question and underlining what is known in the Understand the Problem. I liked the various methods for teaching Polya's problem-solving strategies to young children.” “This is one of my favorite aspects of this program. There was so much varied practice and focus on strategies. I noticed a great difference in my teaching this year from last year, and problem-solving practice was still hard, but not frustrating.” “The real-world problem-solving questions were great for the students to understand why math is important in their world. They were able to see the importance of understanding different math concepts.” “The Problem of the Day really encouraged mathematical thinking. My students really learned from the Problem-Solving Strategies.” “The problem-solving lessons for each chapter were very instrumental in not only understanding the concept, but in applying it.” “I loved the pages of problem solving in each chapter. Most were very good and engaging. I loved, loved, loved the Problem of the Day.” “My students loved the problem-solving readers. In fact, I used them in after-school reading tutoring because they were of such high interest to the students. I felt the series provided many opportunities for problem-solving practice, and I liked the multi-step approach to give kids a format to solve problems…” My students enjoyed this, but they seemed to take a little longer to understand. I had to give more examples to help them. “Many of them were good examples. Although, I felt some examples for the Problem of the Day were simply basic computations.” “The Problem of the Day tended to be easier for my students. Estimation was more difficult and was good to use for the 5-Minute Check. The problem-solving strategy was great, and we used that procedure each time during problem solving.”


Math Connects Fourth Grade Teachers’ Comments on Problem Solving


“Math Connects was extremely effective in the development of real-world problem-solving skills for my 4th graders. They were always thrilled to see the problem-solving lessons and literally begged to work in teams to start the lesson. I've never experienced so much enthusiasm from a class in regard to math lessons. They quickly began discussing what method they would need to solve the problem and had the steps down after the first few tries.” “I really liked the Problem of the Day. I created a way for students to break down word problems that the students used in solving the Problem of the Day. The problem-solving skills were reiterated with the program’s problems.” “I witnessed huge gains in students' problem-solving abilities throughout the year. I used the Problem of the Day quite a bit, probably four days a week. Because we had so many transitions, it was a great method to get the students settled as they were coming into the room, with the overhead on and the lights low. Next year, I will use the Real World Problem Solving Readers a lot more because they are of high interest to the students. I always included the Problem Solving Strategy and Investigation in every unit. I feel it was very useful in guiding their thinking, even to the very questions they need to be asking, and also in identifying the information they already had.” “The students found the real-world problems interesting and relevant.” “The students enjoyed the real-world problems because they could relate to them.” “I used the overhead Problem of the Day every day for the morning routine and loved them! We also used the problem solving workbooks. We worked in teams to solve the problems and then discussed and demonstrated how they solved the problems. It was great test preparation.” “These were great for the on-level and above-level students, but were too difficult for the below-level students.” “Problem solving is hard for many students. I think the students who got this had no problems with it, but my lower-level students really struggled with the problem-solving skills section. I liked the way the book taught problem-solving steps of understand, plan, solve and check. We also taught problem solving at another time, so we didn't always do all the problem solving lessons.” “I didn't use the Problem of the Day. There is so much else that they really were not needed. Again, the examples that supported the concept really supported student learning!!! We liked the way the examples broke down how to solve a problem: understand, plan, solve and check.”


Teachers were also asked to assess the effectiveness of their curricular materials in facilitating cross-curricular connections. First, they were asked to evaluate it in a very direct manner and then they were invited to comment and explain their assessment in an open-ended manner. The Figure below displays the quantitative data teachers provided regarding the effectiveness of their materials in this regard. In grade four, teachers using Math Connects provided significantly higher ratings of effectiveness than teachers using other programs (Math Connects n = 25, comparison group n = 29, t = 3.6, p < .00). For second grade, the difference in Mean scores did not meet conventional levels of statistical significance (Math Connects n = 25, comparison group n = 31, t = 1.7, NS).

Teachers provided a substantial number of comments about the effectiveness of Math Connects in facilitating cross-curricular connections. Altogether, 24 second grade teachers and 27 fourth grade teachers offered comments. All responses were generally categorized as “positive,” “negative,” or “irrelevant” (teachers would sometimes comment in ways that did not address the question) by two independent coders. Disagreements were resolved by discussion. The Figures below display representative comments. The full text of all comments by all teachers is available in an appendix.

2.8 2.9

2.4

2.1

1

1.5

2

2.5

3

3.5

4

Grade 2 Grade 4

Avera

ge R

esp

on

ses

(1=

Str

on

gly

Dis

ag

ree…

4=

Str

on

gly

Ag

ree)

Elementary Grade Teachers: "The program I used to teach math this year is very effective at helping me make cross- curricular connections."



Math Connects Second Grade Teachers’ Comments on Cross-Curricular Connections

Please comment on the effectiveness of the guidance for making cross- curricular connections. Also include comments about the general effectiveness of Math Connects in helping you to make cross- curricular connections. You may offer both positive and negative comments. Please be as specific as you can.

“The Learning Station lessons, as well as the mini-books, were a wonderful way to integrate the different subject areas into the area of math. In addition, the actual real-world problem solving specific to our state made it possible for my students to recognize that there is more out there than simply the town they live in.” “The little booklets in the student journal that made cross-curricular connections were great. My students loved pulling them out, and they created many great conversations on other disciplines.” “I took advantage of the ideas and used the books provided, as well as the literature suggested to connect reading and social studies. It was great.” “There were some wonderful ideas and plans in the Teacher Edition, and I found them very helpful and useful. As I mentioned before, it helps me to have specific ideas that I can add onto, versus having too much information in the center ideas, for instance.” “We use stations for reading in my class. It was very easy to implement some of the math stations as needed and use the ideas to create different ones.” “The Learning Stations allow for differentiating, individualizing, creativity and integration. Every child needs that time of choice, self-reflection and self discipline.” “Having the Real-World Math in the workbook made it easy to use. The students looked forward to pulling out the books and completing the lessons. The multiple copies of science and social studies books on magnets, life, plants and other areas were of high interest. I would like to collect more of the literature books for more lessons. Cross-curricular connections, such as needs and wants, working with a calendar and money games were fun. There was a great variety of hands-on activities which the students enjoyed learning from; they were very effective. I needed to have an activity for the students to move to after completing their Learning Station so a student would not be bothering others after one student finished his or her work. They could move to reading and testing on BookAdventure.com or working on a computer math game which helped the students work quietly and refocus.” “I feel I did not get to use the connection effectively or at all. With too much already on our curriculum plate, it was something I didn't use.” “My students really enjoyed the areas that included other disciplines, but I only used them when I had a little extra time. They didn't feel like they were working on math. They had a lot of fun with this.” “They are great materials, but I lacked time to implement them.”


Math Connects Fourth Grade Teachers’ Comments on Cross- Curricular Connections


“The Learning Stations and Problem Solving lessons were wonderful. They really helped connect math to so many other teaching areas. We learned so much just going through each lesson with the real-world connections. These connected to so many other areas of our teaching curriculum.” “I liked the use of photographs and colorful pages to link to science and social studies. These activities were very popular with my group of students.” “The word problems often had science and social studies connections which made cross-curricular connections.” “This year, students completed the Problem Solving lessons as they progressed in the book. Daily, we are making real-world connections to the importance of math, why we learn math and how it will help us in our daily lives. I didn't set up the learning stations; instead, I would pick a couple of subjects and integrate them into our lessons. The more connections that can be made and across the disciplines, the more student learning increases! The ideas that I chose were easy to implement without a lot of prep work, which saves time.” “There wasn't enough time in the day to be able to do every Learning Station that was suggested. I did use some of the resources, but overall I did not. I would have if I could teach like I wanted to and didn't have to follow the strict state standards for every subject. I didn't have the time to research the connection of additional stations to the other curriculum standards to integrate the lessons. I wish I could have!” “I didn't utilize the Learning Stations as I could have, partly due to time and partly due to planning. Next year, I anticipate actively incorporating them throughout the year. I used the Problem Solving lessons in every chapter with groups one and four. Groups two and three were moving so slowly that we didn't always get to the material in the Problem Solving lessons, even though they were of high student interest.” “I did not use much of the learning center ideas, but I did use the Individual Readers and really liked them. The students did well with them.” “I did not get a chance to use those as often as I would like. I would like to do more next year.” “The materials are lovely. However, there was no way to effectively and continuously engage students in their use. We were sprinting through our days and through our curriculum. Only those students with free time were able to participate in the Learning Stations.”


Differentiated Instruction. In the background review of research for Math Connects, Papa and Brown (www.glencoe.com/glencoe_research/research_maths.html) summarize the voluminous research on differentiated instruction. It is widely recognized that effective instruction requires adapting both instructional content and instructional practices to meet the students at their individual level of readiness. This can be a major challenge for most teachers as they encounter classrooms that have, at the same time, students who are above grade level in ability and ready for enrichment activities, students who are below grade level for any of a variety of reasons and students for whom English is not their first language. According to the Teacher Reference Handbook of Math Connects, one major component of the Math Connects program is the provision of multiple options for providing differentiated instruction.

Results clearly and consistently suggest that teachers believe the Math Connects materials are effective at helping to provide differentiated instruction. First, the Figure below displays the response data for a global question asking all teachers about the effectiveness of their programs in the area of differentiated instruction. In both grades two and four, the teachers using Math Connects were very positive about this dimension of the materials and, in both grades, their responses were significantly more positive than those of the comparison group teachers. For second grade teachers Math Connects n = 24, comparison group n = 31, t = 5.8, p < .00. For fourth grade teachers Math Connects n = 26, comparison group n = 29, t = 5.1, p < .00.

Specific challenges to teachers regarding differentiating instruction were also explored

directly. The Figure below displays the data reflecting the teachers’ perceptions of their programs’ effectiveness at helping students who were below grade level in mathematics. The teachers using Math Connects were very positive about the materials, and they were significantly more positive than teachers using other programs. For second grade teachers Math Connects n = 24, comparison group n = 31, t = 4.6, p < .00. For fourth grade teachers Math Connects n = 26, comparison group n = 29, t = 4.7, p < .00.

3.63.4

2.7

2.3

1

1.5

2

2.5

3

3.5

4

Grade 2 Grade 4

Avera

ge R

esp

on

ses

(1=

Str

on

gly

Dis

ag

ree...

4=

Str

on

gly

Ag

ree)

Elementary Grade Teachers: "The program I used to teach math this year was very effective in helping me provide differentiated instruction."



At the same time, the Math Connects teachers also indicated that their materials were very effective in teaching students who were above grade level in mathematics. And, the Math Connects teachers were significantly more positive about this dimension of their materials than teachers using other programs. For second grade teachers Math Connects n = 25, comparison group n = 31, t = 2.2, p < .03. For fourth grade teachers Math Connects n = 27, comparison group n = 29, t = 4.3, p < .00.

Finally, teachers using Math Connects also indicated that their materials were effective at teaching students who were English Language Learners (see Figure below). They were again significantly more positive than teachers using other materials. For second grade teachers Math Connects n = 23, comparison group n = 30, t = 4.3, p < .00. For fourth grade teachers Math Connects n = 22, comparison group n = 29, t = 2.9, p < .00.

3.33.1

2.5

2.2

1

1.5

2

2.5

3

3.5

4

Grade 2 Grade 4

Avera

ge R

esp

on

ses

(1=

Str

on

gly

Dis

ag

ree...

4=

Str

on

gly

Ag

ree)

Elementary Grade Teachers: "The program I used to teach math this year was very effective for teaching students who were below grade level in

mathematics."


3.23.3

2.8

2.5

1

1.5

2

2.5

3

3.5

4

Grade 2 Grade 4

Avera

ge R

esp

on

ses

(1=

Str

on

gly

Dis

ag

ree...

4=

Str

on

gly

Ag

ree)

Elementary Grade Teachers: "The program I used to teach math this year was very effective for providing enrichment for students who were above

grade level in mathematics."



• The teachers who used Math Connects were very positive about its effectiveness in providing differentiated instruction. The Math Connects teachers were significantly more positive than teachers using other programs. This was the case in reference to students below grade level, above grade level and for students for whom English is not their native language. In all cases, at both grades two and four, the teachers who used Math Connects were significantly more positive than teachers who used other programs.

Teachers also provided open-ended comments explaining their assessments. Their comments support the positive closed-ended data but also suggest that the wide variety of materials made it a challenge to implement all the resources available. The Figures below display selected comments from both the second grade and fourth grade teachers who used Math Connects during the year. The comments presented are representative of all comments which are available in an appendix.

3.1 3

2.4 2.4

1

1.5

2

2.5

3

3.5

4

Grade 2 Grade 4

Avera

ge R

esp

on

ses

(1=

Str

on

gly

Dis

ag

ree...

4=

Str

on

gly

Ag

ree)

Elementary Grade Teachers: "The program I used to teach math this year was very effective for teaching students who are English Language

Learners."



Second Grade Math Connects Teachers’ Comments on Resources for Differentiating Instruction

Please comment on the overall effectiveness of the program that you used in helping you to differentiate instruction. You may include both positive and negative comments. Please be as specific as you can.

“I thought the differentiation of this program was wonderful. It allowed me to reach all learners at his / her own level, and either provide the help they needed, or give them problems to help them stretch even further than I thought possible. All of my children showed such wonderful growth this year, and I think it was due to the differentiation of this program.” “This is a true strength of this program. There are great ideas and resources for differentiating instruction… Two-thumbs up!” “…this program allowed me to differentiate my instruction on a daily basis. Although my ELL students struggled the most, I was able to provide them with the same experiences that my regular education students had. The use of the manipulatives by the whole class allowed for full participation and a clearer understanding of the concepts. In addition, using the daily problem-solving pages helped to reinforce the concepts that had been covered previously. In preparation for the recent testing, we even returned to the beginning of the problem-solving booklet as a quick review. The idea of differentiating lessons became clear, especially within the realm of problem-solving questions. My ELL learners needed to be walked through the problems and provided with extensive time to understand and solve the higher-level questions. I did not have to take away from the learning of my other students because the material provided me with numerous options for teaching, reteaching and supporting concept understanding.” “I found it easy to plan for differentiating instruction for my class. We were able to complete many activities as a whole class, but I had plenty of resources to reteach and provide additional practice for some students, as well as challenge some of the students who were above grade level. I found the Differentiated Instruction section of the Teacher Edition very helpful. I also made use of the differentiated practice suggestions.” “Math Connects was very effective with my ESL students and low achieving students." “Math Connects provided me daily instruction support for all of my students. I could use the many resources to teach the whole group, small groups and individual students at the level that they were ready for, and to stretch their current understandings to a higher level. I appreciated the extensions available for my higher-level students. I used the Problem Solvers and enrichments daily with these students, whom I feel are left out a lot of the time due to their capabilities.” “The program offered many excellent ways to help my students master concepts, and I tried to incorporate as much as I could. There was always the time issue involved because there are so many other parts of the curriculum that have to be covered. I especially liked the quiz at the beginning of each chapter because it gave me an idea on how to guide my instruction and address the needs of my students while using as much of the material as I could in the time I had to teach math.”


Fourth Grade Math Connects Teachers’ Comments on Resources for Differentiating Instruction


“It was great. I could pick-and-choose which problems I want the ELL kids to do, the below-levels to do and the norm of the class to do. There were activities and vocabulary glossaries that were necessary for all of my students. I really liked the vocabulary glossaries. We are SIOP school and vocabulary is important in all areas.” “I was very pleased with Math Connects and how it applied to my differentiated instruction. I had a large range of students at many different levels, and it was great to be able to provide a variety of options for them. I also loved the Data Driven Decision Making section because this is a huge push in my school. The guidelines for Reaching All Learners were helpful when planning each lesson.” “The only limitation in differentiating instruction was my time to explore the resources at my fingertips. Math Connects far surpassed (another program) in the resources available to the instructor. As I said in previous comments, the lowest level group struggled with their reading skills. But rather than abandon the textbook, we incorporated it into the lesson with pre-selected questions that were more within their capabilities. In giving them access to the textbook, their self confidence grew because they were reading 4th grade materials, as they would be expected for the (state) test.” “There are many options for differentiation. They are easily accessible and easy to implement. I did not have a wide range of skill levels this year, therefore not much was needed. At times, I experimented with the resources and liked what I found.” “Math Connects greatly helped me in differentiating my math instruction. I had a class with a wide range of needs, and I found lesson planning quite simple and always had ample amounts of resources to challenge, as well as reteach materials for those who struggled.” “I appreciated the Are You Ready? questions at the beginning of each chapter. It provided a basis for determining skills for each specific area. The wide variety of activities within each chapter offered more choices than any program I have ever worked with.” “Time was not on my side, especially with the levels of the students in my classroom. Many students came to me without having the skills for fourth grade, so I had to do a lot of backtracking. The materials provided the support to assist me with that. I was not able to get online with my students and would have liked to do that.” “It was difficult to do differentiated instruction with such a large class size. It was helpful for the resource teacher.” “All of the materials make it very easy to differentiate. The only problem is that there is so much stuff that I need more time and training on them.”


Digital Resources. The NCTM Principles for Teaching Mathematics state directly that technology is essential in the teaching and learning of mathematics. Advances in technologies of many types have had a great impact on the teaching and learning of mathematics in recent years (see Wong, 2003) and educators are being encouraged to continue to explore the potential of new digital resources (Martin & Spear, 2009). One major objective of this evaluation was to gather information from teachers regarding the effectiveness of the wide range of digital resources available to them.

All teachers in this study (experimental and comparison) were asked to evaluate the digital resources that are a part of their mathematics curricular materials, and the Math Connects teachers were asked direct questions about specific components of the Math Connects materials. These digital resources come in two basic forms. First, there are materials on a CD-ROM for both teachers and students. Also, most programs provide online resources available via the Internet. All teachers were asked directly about the effectiveness of these components and comparisons between conditions were made wherever appropriate.

CD-ROM resources. The data regarding CD-ROM resources from the Math Connects teachers is reported in the Tables below. Responses from all elementary grade teachers (grades two and four) are aggregated because the response rates to these questions were very low. Overall, about two thirds of the teachers indicated that they did not use many components at all. Given the rate with which technologies develop, it is reasonable to expect that some teachers may struggle to use these digital tools in an effective manner (Kysilka, Geary & Schepise, 2002). Teachers’ open-ended comments on this topic revealed that many intended to integrate the digital resources into their daily practices after becoming more familiar with the basic content of the program but they had not had the time to do so during the school year. Several teachers made comments of this nature:

“I think that it takes more than one year of using the curriculum to fully implement all of the resources. I feel I need more than one academic year to use it to its fullest potential. From what I was able to explore and utilize with my students, the CD-ROM resources were user-friendly and relevant.”

“I found myself not using the CD-ROM because I devoted my energy toward the books. I wish I would have forced myself halfway through the year, but the year went by so fast.”

“I would have loved to been able to incorporate more of these resources. They looked fantastic. I was just not able to find the time to use them correctly.”

Other teachers cited technical problems that represented an additional roadblock to using some of the resources right away. For example, two comments were:

“I am sure the materials are very helpful, but our computers are very outdated, and we don't have enough computers to be able to have the students use these materials.”

“I don't have easy access to classroom computers; therefore, my classes didn't spend the time with the technology resources. How unfortunate!”

The objective data from teachers who presumably at least attempted to use the CD-ROM resources indicated that these resources could be implemented easily. When asked directly, the majority of the Math Connects teachers indicated that the CD-ROM resources


included in Math Connects were either “easy” or “very easy” to use. The Table below displays the frequency data for teachers who responded to this item. .

Math Connects Elementary Teachers’ Assessments of Ease of Use for CD- ROM Resources

Very difficult to use

Difficult to use

Easy to use

Very easy to use

No response

All things considered, how would you rate the ease of use of the resources available on CD- ROM for teachers?

0 5 14 9 24

Results suggest that the majority of the teachers who used the CD-ROM resources found them at least moderately helpful, although low response rates to these items make drawing even tentative conclusions somewhat difficult.

Math Connects Elementary Teachers’ Assessments of CD- ROM Resources

Not helpful at all

Only slightly helpful

Mostly helpful

Extremely helpful

No response

How helpful was the TeacherWorksTM CD- ROM? 2 6 8 4 32 How helpful was the Interactive Classroom CD- ROM? 2 6 1 3 40 How helpful was the Exam View® Assessment Suite CD- ROM? 1 5 9 2 35 How helpful was the Math Songs CD- ROM? 1 10 11 4 26 How helpful was the StudentWorksTM CD- ROM for your students? 1 6 4 4 37 How helpful was the StudentWorksTM CD- ROM for you? 1 6 7 4 34 How helpful was the My Math Zone CD- ROM for your students? 3 7 3 3 36 How helpful was the My Math Zone CD- ROM for you? 3 6 2 4 37 How helpful was the Math Adventures with Dot and Ray CD- ROM for you? 2 5 3 0 42 How helpful was the Math Adventures with Dot and Ray CD- ROM for your students? 3 3 4 1 41 All things considered, how helpful were the resources available on CD- ROM for teachers? 2 6 9 8 27


The Math Connects Teachers were asked to identify the specific feature of the TeacherWorksTM CD-ROM that they found most helpful. Eleven of the 27 teachers responding indicated that the Chapter Resource Masters/Teacher Resources were most helpful. Eight indicated the Lesson Planning features were most helpful and six identified the Teacher Edition (on the CD-ROM) as most helpful. The two remaining teachers identified the Personal Assistant Links to the Internet as most helpful.

Math Connects teachers also identified how they used the Exam View® Test Generator. Nine of these teachers indicated that they used the question bank “as is.” Another six indicated they used the formatting feature and one teacher indicated they added their own questions to create their own assessments.

Teachers were also invited to comment on the general usefulness of the CD-ROM resources. Representative comments are displayed in the Figure below. The full text of all comments is available in an appendix.

Math Connects Elementary Teachers’ Comments on CD- ROM Resources

Please make a brief statement that summarizes your opinion of the CD- ROM resources provided to you. You may include both positive and negative comments.

“I think they are a must. Time makes it hard to use them all, but in general when I needed them they were available and valuable.” “At the beginning of the year, I was overwhelmed with all of the materials. It was only later in the year that I began exploring some of the CD-ROM resources…They really enjoyed playing the games to review and practice. I believe that if I were to use these materials for another year, I would make more use of the CD-ROM materials. I liked what I did begin to explore.” “There was just so much! There are great resources. I am still exploring!” “It was nice that I could have it read the test for the students. I liked the procedural lessons…StudentWorks™ was great! :)” “I would have liked to use them more, but I was uncomfortable with how to use them. Our training was brief on the use of the technology in the program.” “We had brand new computers this year for teachers, but we were blocked from downloading the programs. Our student computers are ancient Macs that couldn't handle the programs. There didn't seem to be much point in using them at home because there were so many other useful resources.” “It was great to use at home on my own time when I did not carry my Teacher Edition with me. I had difficulty making exams from the exam-making CD-ROM.” “I did not use them. I look forward to exploring them over summer break to implement next year.”


Online digital resources. Digital resources are often available to both teachers and students via the Internet. In Math Connects students had a variety of practice activities and self assessment options. The data regarding the online resources from the Math Connects teachers is reported in the Tables below. Responses from all elementary grade teachers (grades two and four) are once again aggregated because the response rates to these questions were also low.

Math Connects Elementary Teachers’ Assessments of Online Resources

Not helpful at all


Mostly helpful

Extremely helpful

No response

How helpful was the online version of StudentWorksTM for your students? 1 3 4 9 35 How helpful was the online version of StudentWorksTM for you? 0 4 5 7 36 How helpful were the online activities for your students? 0 2 6 11 33 How helpful were the online activities for you? 0 3 6 9 34 How helpful were the online self assessment resources for your students? 0 5 6 7 34 How helpful were the online self assessment resources for you? 0 3 8 6 35

Math Connects teachers who used the online resources seemed to think that these resources were easy to use. The Table below displays the frequency data on that issue.

Math Connects Elementary Teachers’ Assessments of Ease of Use for Online Resources


Difficult to use

Easy to use

Very easy to use

No response

All things considered, how would you rate the ease of use of the online resources available for you?

0 3 8 13 38

General assessment of digital resources. One central interest in this study was to compare the attitudes of teachers using Math Connects with those of teachers using other programs regarding the general effectiveness of the digital resources available to them. All teachers (Math Connects and comparison group) responded to two closed-ended items asking them to provide a general assessment of the digital resources that were a part of their curricular


materials. The Figures below display the comparison of the average responses for the Math Connects teachers and the teachers who used other programs.

The elementary school teachers using Math Connects were significantly more positive about the digital resources for teachers than teachers who used other programs (see Figure below). (For grade 2, Math Connects n = 20, comparison group n = 16; t = 4.7, p < .00; for grade 4, Math Connects n = 16, comparison group n = 7; t = 4.0, p < .00). The teachers using Math Connects were also significantly more positive about the digital resources for students than teachers who used other programs (see Figure below). (For grade 2, Math Connects n = 20, comparison group n = 18; t = 4.0, p < .00; for grade 4, Math Connects n = 14, comparison group n = 8; t = 3.6, p < .00).

3.8

3.4

2.6

1.9

1

1.5

2

2.5

3

3.5

4

Grade 2 Grade 4

Avera

ge R

esp

on

ses

(1=

Str

on

gly

Dis

ag

ree...

4=

Str

on

gly

Ag

ree)

Elementary Grade Teachers: "The program I used to teach math this year had very effective technology resources for teachers."


3.63.3

2.7

2

1

1.5

2

2.5

3

3.5

4

Grade 2 Grade 4

Avera

ge R

esp

on

ses

(1=

Str

on

gly

Dis

ag

ree...

4=

Str

on

gly

Ag

ree)

Elementary Grade Teachers: "The program I used to teach math this year had very effective technology resources for students."



• While many of the elementary grade teachers using Math Connects were not able to use many of the digital resources available to them, those who did use them were very positive about their general effectiveness. The Math Connects teachers who used their digital resources were significantly more positive about them than teachers who used other programs. The teachers who used Math Connects were also significantly more positive than teachers who used other programs about the technology resources available for students.

In sum, it appears as though teachers using the Math Connects program had positive impressions of the digital resources that are a part of the materials, but they struggled to find time to integrate these resources into their daily practices at this point. This impression is validated by the Math Connects teachers’ expressions of their future intentions. They were asked, “If you were to use Math Connects next year, would you use the technology resources more often, less often or about the same as this year?” All together, 43 of the 51 (84%) teachers who responded indicated they would use the digital resources more often. Another 7 of 51 (14%) indicated they would use those resources with about the same frequency and only 1 teacher indicated they would use them less.

Assessment and Intervention. Because of the continuous emphasis on accountability and assessment in mathematics instruction, one central focus of this evaluation was on the teachers’ perceptions of the assessment materials provided within the programs they were using. As Wilson and Kenny (2003) state directly, valid assessment is essential to effective mathematics instruction. The Teacher Reference Handbook of Math Connects asserts that every effort has been made with that program to integrate assessment activities into instruction. This notion seems to be directly consonant with the framework suggested by Lesh and Lamon (1992) when they advocate for the seamless integration of instruction and assessment.

Many elementary materials include initial materials designed to provide a review of preceding materials that allow teachers to both reinforce concepts and begin to diagnose any needs for intervention with students who may need further instruction (this feature is labeled the “Start Smart” section within the Math Connects materials). Teachers were asked directly about this dimension of their assessment resources.

The Figure below displays the data relevant to the review and reinforcement materials for both the Math Connects and the comparison group teachers. While both the second and fourth grade teachers who used Math Connects provided higher average ratings of helpfulness than teachers using other programs, the difference in the fourth grade did not meet standard levels of statistical significance. For the second grade, Math Connects n = 23, comparison group n = 23, t = 2.9, p < .00. For the fourth grade teachers, Math Connects n = 19, comparison group n = 18, t = 1.0, NS.


Many programs also provide assessment resources that allow teachers to conduct initial assessments at the beginning of the school year and also at the beginning of every chapter to diagnose any needs for intervention. The Figure below displays the response data on the direct question about these materials for all teachers by grade and by condition.

Once again the results indicate that the Math Connects teachers at the second grade level were significantly more positive about their materials than teachers who used other programs (Math Connects n = 23, comparison group n = 21, t = 4.6, p < .00). The difference at the fourth grade level was not statistically significant (Math Connects n = 21, comparison group n = 16, t = 1.8, NS).

Most mathematics curricular materials also provide formative assessment resources that allow teachers to quickly gauge their students’ progress. These resources typically take the form of mid-chapter quizzes or other forms of short term testing (within Math Connects these are provided in resources such as the Mid-Chapter Check, Quick Check, or Check What You Know). All teachers in the study were asked for their direct assessment of the formative assessment resources available in their programs (see Figure below). The data displayed below indicates that the teachers using Math Connects were significantly more positive about their formative

32.7

2.4 2.4

1

1.5

2

2.5

3

3.5

4

Grade 2 Grade 4

Avera

ge R

esp

on

ses

(1=

No

t at

all

Help

ful..

.4=

Extr

em

ely

Help

ful)

Elementary Grade Teachers: "In the program you used to teach math this year, how helpful was the review and reinforcement of learning from last

year?"


3.33

2.3 2.4

1

1.5

2

2.5

3

3.5

4

Grade 2 Grade 4

Avera

ge R

esp

on

ses

(1=

No

t at

all

Help

ful..

.4=

Extr

em

ely

Help

ful)

Elementary Grade Teachers: "How helpful were the initial assessment tools in the program you used to teach math this year?"



assessment resources than teachers who used other programs. For the second grade, Math Connects n = 25, comparison group n = 23, t = 4.2, p < .00; for the fourth grade, Math Connects n = 26, comparison group n = 18, t = 3.0, p < .00.

Most programs also provide summative assessment resources in the forms of chapter tests or practice for standardized tests. The Figure below displays teachers’ responses to a question asking them directly about the helpfulness of the summative assessment resources in their programs. Teachers using Math Connects, at both the second grade and fourth grade levels, were significantly more positive about their summative assessment resources than teachers using other programs. For the second grade, Math Connects n = 25, comparison group n = 27, t = 3.5, p < .00; for the fourth grade, Math Connects n = 26, comparison group n = 24, t = 3.1, p < .00.

Finally, it is reasonable to assume that teachers’ use of available assessment resources would be affected by the amount of effort that is required to administer and interpret the assessment. That is, no matter how helpful a resource might be, if it cannot be used efficiently it will be of limited utility to the teacher. So, all teachers in the study were asked to assess how easily their assessment resources were to use. The Figure below displays the response data for this item by grade and by condition. Results indicate that the Math Connects teachers were very

3.63.4

2.82.7

1

1.5

2

2.5

3

3.5

4

Grade 2 Grade 4

Avera

ge R

esp

on

ses

(1=

No

t at

all

Help

ful…

4=

Extr

em

ely

Help

ful)

Elementary Grade Teachers: "Overall, in the program you used to teach math this year, how helpful were the formative assessment options?"


3.43.6

2.7 2.9

1

1.5

2

2.5

3

3.5

4

Grade 2 Grade 4

Avera

ge R

esp

on

ses

(1=

No

t H

elp

ful

at

all

…4

=Extr

em

ely

Help

ful)

Elementary Grade Teachers: "Overall, in the program you used to teach math this year, how helpful were the summative assessment options?"



positive about the ease of use of their assessment resources. The average rating for the Math Connects teachers in both second and fourth grade was 3.6 on a 1-4 scale. For the second grade, Math Connects n = 24, comparison group n = 29, t = 2.9, p < .00; for the fourth grade, Math Connects n = 26, comparison group n = 24, t = 3.7, p < .00.

• The elementary grade teachers who used Math Connects were very positive about all forms of assessment resources in Math Connects. Both second and fourth grade teachers who used Math Connects were very positive about its initial, diagnostic assessment resources. Teachers at both grade levels who used Math Connects were significantly more positive about its formative and summative assessment resources than teachers who used other programs. Both second and fourth grade teachers were very positive about the ease with which the Math Connects assessment resources could be used. The Math Connects teachers were significantly more positive about this than teachers who used other programs.

Mathematical Literacy. Especially at the elementary level, the learning of mathematics interacts with the development of reading competencies (Adams & McKoy, 2007; van Garderen, 2004; Walker, Zhang and Surber, 2008). Students must be able to comprehend math problems expressed in language. Also, a central part of learning mathematics is the development of mathematical literacy. Put simply, mathematical literacy occurs as students develop a sound understanding of the vocabulary of mathematics (see Jablonka, 2003). Thompson and Chappell (2007) explain how the NCTM standards emphasize that learning effective communication and representation of mathematical concepts is absolutely fundamental to learning mathematics. So, one objective of this evaluation was to explore how the Math Connects program facilitates mathematical literacy and discern the appropriateness of the vocabulary used within Math Connects.

The elementary school teachers using the Math Connects program indicated that their materials were effective at helping their students develop mathematical literacy (see Figure below). Teachers using Math Connects in both grades two and four were significantly more positive about their materials in this regard than teachers using other materials. (For grade 2,

3.6 3.6

3.13

1

1.5

2

2.5

3

3.5

4

Grade 2 Grade 4

Avera

ge R

esp

on

ses

(1=

Very

Dif

ficu

lt…

4=

Very

Easy

)Elementary Grade Teachers: "Overall, how would you rate the ease of use for the assessment resources in the program you used to teach math this year?"



Math Connects n = 25, comparison group n = 31; t = 2.1, p < .00; for grade 4, Math Connects n = 26, comparison group n = 29; t = 3.3, p < .00).

All teachers in the study also provided open-ended comments about the effectiveness of their materials in helping students develop mathematical literacy. Teachers’ open-ended comments about this dimension of Math Connects indicated that they felt the resources for teaching mathematical literacy were some of the strongest and most effective resources in the program. The Figures below present representative comments from the teachers using Math Connects. The full text of all comments from all teachers is available in an appendix. Generally, teachers were very positive about the presentation of mathematical vocabulary with Math Connects, particularly the way in which vocabulary was integrated into the lessons with multiple resources.

3.2 3.2

2.72.5

1

1.5

2

2.5

3

3.5

4

Grade 2 Grade 4

Avera

ge R

esp

on

ses

(1=

Str

on

gly

Dis

ag

ree...

4=

Str

on

gly

Ag

ree)

Elementary Grade Teachers: "The program I used to teach math this year was excellent at helping students develop mathematical literacy."



Math Connects Second Grade Teachers’ Comments on Mathematical Literacy

Please comment on the effectiveness of the program you used to teach math this year in helping students develop mathematical literacy. You may include both positive and negative comments. Please be as specific as you can.

“I loved this piece of the program! My students are much more math literate and know language that will carry with them into 3rd grade. I used the vocabulary cards very frequently. I also provided direct instruction in vocabulary in each chapter.” “Vocabulary is a very important part of the state testing. They were used a lot.” “I think it is critical to teach math vocabulary. I utilized the vocabulary cards and repeated daily practice of vocabulary. My ELL students had a hard time with math vocabulary. They may know the concept, but would get mixed up with terms. I strongly feel that they need to know the math vocabulary to be successful on standardized tests.” “The mini books provided a great source of information, both in math and in the content areas they focused on...The children still gravitate toward those books and each has their own favorites. The books focusing on science, particularly animals, really provided a great deal of mileage for my class.” “This was very effective for my students because it exposed them to vocabulary that they wouldn't get in the other disciplines. My ELLs especially benefited because it increased their vocabulary in math. Thanks for adding this very important component.” “The vocabulary guide for each chapter was helpful in reinforcing what was taught and provided students with a math dictionary. The activities and lessons provided supported mathematical literacy.” “I loved those! I have a high ELL population, and we really put the vocabulary cards and words in the lessons to work!” “We read all of the resource books in the kit. I thought that they were good, and they fit our social studies and reading program very well… I did not have access to many of the suggested read-alouds.” “One of the first things I noticed about the program was the highlighted key vocabulary. It was very helpful, and as a class we would look back and read the key vocabulary words as we worked through the lessons, especially when the students were explaining their thinking. They needed to use the key vocabulary, and it was easy for them to use when they could see the words. A pocket chart for the math vocabulary words helped keep the math vocabulary visible during each chapter lesson. I know we used the proper vocabulary more this year because it was introduced and used throughout the workbooks. Without the vocabulary, it is very difficult for the students to explain their mathematical thinking, which is on the standardized tests.”


Math Connects Fourth Grade Teachers’ Comments on Mathematical Literacy


“I was blown away by the vocabulary my class developed thanks to Math Connects this year. Not only did their math vocabulary increase, but their awareness of what they were discussing was impressive for a 4th grade level. I said it many times this year, and stand by the fact that Math Connects taught my 4th graders so much more than Everyday Math taught my 5th graders in the past. I feel like I am sending my kids to 5th grade with the skills they really need to know.” “The students are exposed to a lot of vocabulary, and I often heard them using the vocabulary words throughout the year relating to other concepts in math.” “My students loved the Visual Vocabulary Cards. Their math vocabulary really improved.” “This was a strong part of the program. The kids became very comfortable with the math terms because they saw them often throughout the text. After talking with a 2nd grade teacher, I noted that the vocabulary was the same, and we were ecstatic to see how it built on a year-to-year spiral.” “This is one of my favorite aspects of the program. We have been weak in this area until now. The vocabulary is reinforced in different ways and highlighted in the textbook and glossary. I also liked the flash cards and the student-built glossary activity. I really used these things a lot and found them very helpful!” “The glossaries were great. We used them in almost every chapter. When I gave written assessments, many times I would make a copy of vocabulary part of the test and have students complete it as part of their test. I really think doing the vocabulary first, before delving into the chapter number concepts, is critically important. I have not seen any other math textbook that offers this.” “I noticed a positive difference this year in my students' mathematical vocabulary. Obviously, it must start with the teacher, but I think there is a marked difference from previous years of teaching math. I also made a point to create math words and definitions to post on the wall throughout the year for most of the units. Again, time is the only limitation in reinforcing the vocabulary.” “It is very important for the students to learn their math vocabulary. The Building Math Vocabulary section really helped the students understand the various terms and concepts.” “I think this series does a wonderful job with mathematical vocabulary. I especially liked the already made cards; we used them to create a 3-D wall of mathematical concepts. The students loved this. This really helped to reinforce the concepts each day and build upon each other.”


Teacher Professional Development. The Math Connects program includes a number of resources for professional development. For example, the Teacher Reference Handbook includes sections with articles on a number of topics (e.g. intervention strategies, data-driven decision making). There are also resources within the Teacher Edition and on the Internet. Many teachers’ comments were very positive (see Figure below) but some indicated they did not have time to use these resources or they did not feel a need for them. Teachers provided a very limited number of comments, so the comments from both the second and fourth grade teachers have been combined into a single Figure.

Math Connects Elementary Grade Teachers’ Comments on Professional Development

Please make a brief statement that summarizes your opinion of the professional development resources available to you through the program you used to teach mathematics this year. You may include both positive and negative comments.

“The professional resources allowed me to think out of the box when it came to planning lessons. Even after 30 years of teaching, I still try to be creative and innovative in my daily teaching. I am constantly searching for new methods and strategies to use with my students, as well as new materials to present to them. I think the program is extremely comprehensive and provides a more than adequate amount of material for the teacher. It goes above and beyond any other program that I have worked with.” “It was very easy to use and effective.” “I liked reading about the research behind the development of the program. I also found this information helpful as I familiarized myself with the program and as I started planning for lessons.” “I am glad it was offered and available when needed. Thank you.” “It was nice to know that they were available should I have needed them.” “I think that this is one of the best features of the Math Connects series. It's short, sweet and not overwhelming at all.” “In this world of differentiation-, data- and assessment-driven teaching, we need all the help we can get. Any resources that can bolster our educational practices / strategies in the classroom are extremely useful.” “I didn't use any of the online materials, but the intervention section had some good ideas, as did the data-driven part. We do a lot of this type of thing at my school, so I was pretty familiar with the ideas and strategies presented. Still, it was convenient to have a reminder and resources.” “I've taught 4th grade math enough years that I really didn't refer to the professional development resources very often, so I can't say whether they would be helpful or not to a new teacher.”


When asked directly about the professional development resources available to them, the Math Connects teachers were very positive. The Table immediately below displays the response frequencies for a question which asked teachers about the effectiveness of these resources. The vast majority of teachers who responded indicated the resources were “mostly effective” or “extremely effective.”

Math Connects Elementary Grade Teachers’ Ratings of Professional Development Resources

Overall, how would you rate the professional development resources available to you with Math

Connects?

Not effective at all

Only slightly

effective

Mostly

effective

Extremely effective

Did not use Grade 2

0

2

12

5

6

Grade 4

0

2

8

5

12

In sum, it appears as though the teachers had positive perceptions of the professional development resources in Math Connects but did not utilize these resources to any great extent. Some of their comments implied that they were glad to know that the resources were available to them if needed, but they had not yet had any reason to access them with any frequency.

Blended Instruction. Mathematics education in recent years has been marked by

a call for a move from “traditional” methods grounded in memorization and reproducing existing solutions to “reform-based” methods grounded in hands-on activities and active investigations designed to help students come to more personally relevant understandings of mathematical concepts (see Findell, 1996; Fraivillig, Murphy & Fuson, 1999; Greenes, 1996; Sood & Jitendra, 2007). The Math Connects program includes a supplemental program (IMPACT Mathematics) which provides reform-based lessons for teachers to use in coordination with the basal curriculum materials. All teachers in this study were asked directly about their use of IMPACT Mathematics or other reform-based materials available to them.

Results indicate that the Math Connects teachers, in their first year of program implementation, did not access the IMPACT Mathematics materials very frequently. The Table below displays the response data for the elementary level Math Connects teachers. The majority of teachers indicated they never used the alternate lessons; few of the teachers used them with any frequency.


Math Connects Elementary Grade Teachers’ General Use of IMPACT Mathematics

How often did you use the resources from IMPACT Mathematics?

Never

Occasionally

Frequently

Very frequently Grade 2

12

10

3

0

Grade 4

15

10

1

1

The teachers who did use the IMPACT Mathematics materials were asked to provide more detailed information about how they used these materials. The data displayed in the Table below indicates that teachers used these lessons in both a small group format and as general class lessons. In follow up questions, the teachers indicated that they seldom used the IMPACT Mathematics lessons to replace the lessons in the basal Math Connects text. All 13 of 13 second grade teachers who responded indicated they did this “occasionally” or “never,” and all 12 of 12 fourth grade teachers responded the same way. Instead, it appears as though teachers who did use these lessons integrated them into the Math Connects lessons.

Math Connects Elementary Grade Teachers’ Use of IMPACT Mathematics

When you used IMPACT Mathematics, did you use it with all, most, some, or only a few of your students?

Only a few of my students

Some of my

students

Most of my

students

All of my students Grade 2

0

5

3

4

Grade 4

5

2

3

2

While teachers were provided with a Blended Usage Guide designed to help them combine their IMPACT Mathematics materials with their basal Math Connects materials, data indicates that this guide was seldom used. Seven of the 12 second grade teachers responding to a question about the Blended Usage Guide indicated they never used the guide and another five of the 12 indicated only occasional use (1 indicated frequent use). Six of the 12 fourth grade teachers who responded indicated they never used the guide and another five indicated occasional use (1 indicated frequent use). Even without the use of the guide, teachers were very positive about the ease of which the IMPACT Mathematics lessons could be combined with their Math Connects lessons (see Table below). The vast majority of the teachers felt it was “easy” or “very easy” to combine these lessons.


Math Connects Elementary Grade Teachers’ Ease of Combining IMPACT Mathematics with Math Connects

Overall, how easy was it for you to combine your use of IMPACT Mathematics with your use of the basal

Math Connects lessons?

Very difficult

Difficult

Easy

Very easy Grade 2

0

3

8

1

Grade 4

0

2

9

1

Finally, one common reason that teachers cite for not accessing alternate reform-based lessons for their classes is the fact that sometimes the mathematics vocabulary in the alternate lessons will differ from the vocabulary in their basal materials. This inconsistency can confuse students. The teachers using Math Connects and IMPACT Mathematics were generally very positive about the consistency in the vocabulary between these two sets of materials (see Table below).

Math Connects Elementary Grade Teachers’ Perceptions of Vocabulary in IMPACT Mathematics

Was the math vocabulary in IMPACT Mathematics consistent with the vocabulary in the basal Math

Connects text?

No, it was not at all

consistent

It was somewhat consistent

It was mostly consistent

Yes, it was completely consistent

Grade 2

1

2

7

3

Grade 4

0

2

9

1

So, most generally, it appears as though teachers had initially positive orientations toward the IMPACT Mathematics materials, but had not fully implemented them yet. Some comments implied that teachers perceived these resources as “extras” that might be used only in special circumstances or to solve a special problem with underachieving students.


Results: Middle School Teachers’ Evaluations of Curriculum Materials

While the valid assessment of student learning outcomes is certainly an essential component of any evaluation of educational materials, it may not be reasonable to expect any program to yield robust changes in student learning scores after only its first year of implementation. Common sense suggests that teachers need time to learn how to use any program, especially a genuinely innovative program, to its fullest advantage. Thus, a second major dimension of this national field study was the direct assessment of teachers’ evaluations of program materials. Our assumption is that the teachers who use the materials are expert judges of the programs’ effectiveness. Their opinions should be considered along with the standardized testing (see U.S. Department of Education, 2008; The Final Report of the National Mathematics Advisory Panel). Existing research suggests that effective curriculum materials should have several key characteristics. Teachers were asked to provide their general assessments of their materials and also asked to assess the global characteristics of their curriculum materials including:

• The extent to which the materials addressed essential curriculum focal points.

• The degree to which the materials are vertically aligned.

• The effectiveness of the materials in facilitating balanced instruction.

o This would include effectiveness in teaching applied problem solving.

o This would also include materials to facilitate cross-curricular connections.

• The extent to which the materials supported differentiated instruction.

• The general effectiveness of the digital resources available with the materials.

• The effectiveness of the diagnostic assessment and intervention resources.

• The effectiveness of the formative and summative assessment resources.

• The effectiveness of the materials in helping students develop mathematical literacy.

• The value of the professional development resources provided with the materials.

• The effectiveness of the materials in facilitating blended instruction.

The results reported in this major section are for the seventh grade teachers and students only. Results for elementary teachers and students are reported in an earlier section.


Seventh Grade Teachers’ General Assessments

All teachers were asked to provide their overall impressions of the effectiveness of the materials they used over the course of the school year. Teachers provided both open-ended qualitative data and closed-ended quantitative data. These general assessments are presented first, followed by more focused assessments of specific functions and components of their curriculum materials.

Qualitative Comments about Program Strengths. Teachers’ assessments of the Math Connects materials began with a general open-ended question asking them to identify their most positive experiences over the course of the year. To avoid priming the teachers to comment on any specific components, this question was asked before any other specific questions were presented. The intention was to discern the teachers’ most salient impressions about the strengths of the Math Connects program. The Figures below present representative samplings of the seventh grade teachers’ most positive experiences with Math Connects. The full text of all comments from both the Math Connects teachers and the comparison group teachers is included in an appendix. Teachers were also asked to identify their most negative experiences with their materials. The full text of all of these comments is available in an appendix.

Teachers’ positive comments covered a variety of topics. But, many were very positive about the variety of resources provided to them by Math Connects. Many also mentioned the assessment options and the manipulatives as very helpful. Negative comments most frequently mentioned typographical errors within these new materials or complained about the lack of time to use some resources.


Selected Math Connects Seventh Grade Teachers’ Positive Experiences


“I appreciated the amount of resources given to us. The section at the beginning of each chapter section was always great because it was a lead into what we would be learning and usually had some application to it. The little investigations allowed students to figure some things out on their own while still being guided through the process. They then could apply that to the overall lesson.”

“The layout of each lesson is well done. I loved the Foldables™, and so did my students. ExamView® and the resources CD-ROM were great because of the multiple choice and short answer questions offered.”

“I liked the setup of the book. The order of the concepts was very good.”

“The most positive experience that I had with the Math Connects Teacher Edition is the way it related the material to real-life learning. I had positive experiences with the manipulatives, the connections to other subjects and the connection with different books such as IMPACT Mathematics and MathScape™. I also liked ExamView® because I was able to create tests that fit exactly what I emphasized in class. MathScape™ helped to enhance some of the topics as it allowed students an opportunity to get their hands on the concepts. I also liked that students had access to manipulatives without having to find more materials and then possibly having to adapt the activity to fit with what I could afford for the class…”

“At first, the entire package was a bit overwhelming. I did learn to enjoy the CD-ROM and the manner in which the entire unit could be planned out…With the Math Connects materials, I was pleased with the straightforwardness of the text. I also found myself extending my teaching to meet the needs of all students…It was typically easy to find something from the materials that would fit each student’s level of learning.”

“I liked the book! I also liked the DVDs with the extra materials.”

“…My students loved the MindJogger quiz DVD. I did not discover this until the end of the school year. They were extremely engaged and willing to solve every problem during the quiz. It surprised me, and I will start the school year off using this.”

“…The PowerPoint presentations that came with each section were extremely helpful to me. I did edit them to best fit my students’ needs, but they saved me a tremendous amount of time. I loved the cartoon book that came with this book. I wish there had been more cartoons to read and have my students complete the story…”


NCTM Standards. The National Council of Teachers of Mathematics (NCTM) has identified curriculum focal points for every grade level (NCTM, 2006). That is, they have identified three major objectives that mathematics curricula should address at each level. All teachers in this study were asked to evaluate the extent to which the materials they used addressed the NCTM focal points for their grade level. Specifically all teachers were asked:

“These questions ask you about the general content focus of the curricular materials that you used this year. Please take a moment and reflect on those materials overall and then respond to the items below. Remember, we are asking you about the materials themselves and how they facilitated your teaching in these different areas.”

Then the teachers responded to statements which asked directly about the curricular focal points for their grade level (verbatim items are displayed in Tables below). Teachers made a judgment on a scale of 1-4 (1=not at all effective, 4=very effective) of the extent to which their materials helped students achieve each curriculum objective.

Teachers’ responses indicate that, in their opinions, the Math Connects materials are effective at addressing the NCTM curricular focal points. The Tables below display the frequency of responses for all seventh grade teachers in both the experimental and comparison conditions. While the relatively small sample sizes make significance testing problematic, the response patterns for the first two focal points suggest that the Math Connects teachers may be more positive about their materials than teachers using other programs. For the first two curriculum focal points more than 70% of the Math Connects teachers responded that Math Connects was either “mostly effective” or “very effective” at addressing those focal points. Results pertaining to the third focal point (regarding rational numbers and linear equations) are generally positive for both groups of teachers.

Seventh Grade Teachers’ Assessments Regarding NCTM Focal Point 1

How effective were your materials in helping students develop an understanding of and applying proportionality, including similarity?


Somewhat effective

Mostly effective

Very effective

Math Connects Teachers (n=16) 0% 19% 69% 13% Teachers using other programs (n=9)

0%

56%

44%

0%



How effective were your materials in helping students develop an understanding of and using formulas to determine surface area and volumes of three dimensional shapes?


Somewhat effective

Mostly effective

Very effective



22%

44%

33%

0%


How effective were your materials in helping students develop an understanding of operations on all rational numbers and solving linear equations?


Somewhat effective

Mostly effective

Very effective

Math Connects teachers (n=16) 0% 38% 50% 13%


0%

44%

56%

0%

Middle school teachers using Math Connects clearly felt that their materials were effective at addressing the curriculum focal points specified by the National Council of Teachers of Mathematics. For each of the three seventh grade curriculum focal points, the majority of the teachers indicated that Math Connects was either “very effective” or “mostly effective.” The teachers using Math Connects were consistently more positive about the effectiveness of their materials in addressing the NCTM focal points than teachers using other programs.


Focused Assessments of Basic Characteristics. A substantial part of this evaluation addressed some of the most basic characteristics of any set of curriculum materials. First, all teachers in the study completed an initial series of closed-ended items focused on general characteristics of curriculum materials that are known to be of central interest to teachers. Specifically, these items asked about pragmatic factors that have direct implications for teachers’ daily practices (ease of implementation, program pacing, assistance in lesson planning and the adequacy of practice materials). Finally, teachers were asked to report on the extent to which their materials seemed to be engaging their students in the study of mathematics.

Results indicate that, regarding the most practical issues for teachers, the teachers who used Math Connects were generally very positive about their materials (see Figures below). First, the teachers who used Math Connects were significantly more positive about ease of implementation than teachers who used other programs (Math Connects n = 16, comparison group n = 9, t = 2.1, p < .05). The Math Connects teachers were also significantly more positive about the helpfulness of the suggestions and guides for planning lessons within their program materials (see Figure below). (Math Connects n = 16, comparison group n = 9, t = 1.1, p < .05.) Teachers also rated their programs regarding its general pacing. Data also indicates that the Math Connects teachers were very positive about this dimension of the program, but the difference in Mean responses between the Math Connects teachers and the comparison group teachers did not meet conventional levels of statistical significance (Math Connects n = 16, comparison group n = 9, t = 1.3, NS).

3.3

2.8

1

1.5

2

2.5

3

3.5

4

Avera

ge R

esp

on

ses

(1=

Str

on

gly

Dis

ag

ree…

4=

Str

on

gly

Ag

ree)

Seventh Grade Teachers: "The program I used to teach math this year was very easy to implement."


3.3

2.7

1

1.5

2

2.5

3

3.5

4

Avera

ge R

esp

on

ses

(1=

No

t H

elp

ful…

4=

Extr

em

ely

Help

ful)

Seventh Grade Teachers: "The program I used to teach math this year had very helpful suggestions

and guides for planning my lessons."



Teachers responded to two items that were more content-oriented in nature (see Figure below). First, teachers were asked to evaluate the appropriateness of the general reading level of the materials they used. The Figure below displays the response data for this item on reading level. Teachers in both the experimental and comparison conditions were reasonably positive about the reading level of their materials (Math Connects n = 16, comparison group n = 9, t = .71, NS).

An additional content-centered item asked teachers about the adequacy of the number of practice problems provided by their programs (see Figure below). The Math Connects teachers were generally very positive in this regard, but the test of Means across conditions did not meet conventional levels of statistical significance (Math Connects n = 16, comparison group n = 9, t = 1.8, NS).

Middle school teachers’ opinions of Math Connects were generally positive on several important dimensions. Teachers using Math Connects were significantly more positive about ease of implementation and the assistance provided to them in planning lessons than teachers using other programs. The Math Connects teachers were also positive about the pacing of the program and the amount of practice problems provided for their students.

3.4

3

1

1.5

2

2.5

3

3.5

4

Avera

ge R

esp

on

ses

(1=

Str

on

gly

Dis

ag

ree…

4=

Str

on

gly

Ag

ree)

Seventh Grade Teachers: "The program I used to teach math this

year had a sufficient amount ofpractice problems for all students."


3.12.9

1

1.5

2

2.5

3

3.5

4

Avera

ge R

esp

on

ses

(1=

Str

on

gly

Dis

ag

ree…

4=

Str

on

gly

Ag

ree)

Seventh Grade Teachers: "The program I used to teach math this year was written at the appropriate

reading level for my students."



Teachers were also asked to report on their perceptions of their students’ affective reactions to studying mathematics during the school year. Specifically, one component of their biweekly log entries asked them to report on their perceptions of their students’ level of engagement in mathematics. Every two weeks the teachers responded on a 1-4 scale (strongly disagree to strongly agree) to a series of statements designed to assess how their students were reacting to the study of math. These biweekly scores were then averaged over the course of the year to derive a general score for each of the four items. Results for the seventh grade teachers in both the experimental and comparison groups are displayed in the Figures below.

For the item asking about students’ enjoyment of math, the average response for the Math Connects teachers was significantly higher than the average response of the comparison group teachers (Math Connects n = 17, comparison group n = 10, t = 2.2, p < .04). The difference between the Mean on the item asking about students’ general interest in math was not statistically significant (Math Connects n = 17, comparison group n = 10, t = 1.3, NS).

The Math Connects teachers were also positive about their students’ level of engagement and their perceptions of how their students’ attitudes about math were evolving (see Figures above). However, the difference in the average scores on the engagement item was not statistically significant (Math Connects n = 17, comparison group n = 10, t = 1.2, NS). The test of Means for the item about students becoming more positive about math fell just short of conventional levels of statistical significance (Math Connects n = 17, comparison group n = 10, t = 2.0, p < .06).

3.4

3

1

1.5

2

2.5

3

3.5

4

Avera

ge R

esp

on

ses

(1=

Str

on

gly

Dis

ag

ree…

4=

Str

on

gly

Ag

ree)

Seventh Grade Teachers: "My students seem very interested in

learning math."


3.2

2.8

1

1.5

2

2.5

3

3.5

4

Avera

ge R

esp

on

ses

(1=

Str

on

gly

Dis

ag

ree…

4=

Str

on

gly

Ag

ree)

Seventh Grade Teachers: "My students are enjoying their math lessons."



3.5

2.7

1

1.5

2

2.5

3

3.5

4

Avera

ge R

esp

on

ses

(1=

No

t H

elp

ful…

4=

Extr

em

ely

Help

ful)

Seventh Grade Teachers: "All things considered, how helpful is the

Teacher Edition (of the program you used to teach math this year)?"


3.6

3

1

1.5

2

2.5

3

3.5

4

Avera

ge R

esp

on

ses

(1=

Very

Dif

ficu

lt…

4=

Very

Easy

)

Seventh Grade Teachers: "All things considered, how would you rate the

ease of use of the Teacher Edition (of the program you used to teach math

this year)?"


Assessments of Primary Resources. The basic assessment of curriculum materials included a straightforward set of questions centered on teachers’ general perceptions of the quality of their Teacher Edition of their basal program and also their perceptions of the effectiveness of the Student Edition of their basal programs. All teachers in the study responded to direct questions about the effectiveness of the Teacher Edition and the Student Edition of the program they used. The Figures below displays the response data for the middle school teachers’ perceptions of their Teacher Editions.

3.33

1

1.5

2

2.5

3

3.5

4

Avera

ge R

esp

on

ses

(1=

Str

on

gly

Dis

ag

ree…

4=

Str

on

gly

Ag

ree)

Seventh Grade Teachers: "My students are actively engaged in their

math lessons."


3.2

2.8

1

1.5

2

2.5

3

3.5

4

Avera

ge R

esp

on

ses

(1=

Str

on

gly

Dis

ag

ree…

4=

Str

on

gly

Ag

ree)

Seventh Grade Teachers: "My students are becoming more

positive about learning math."



The Math Connects teachers were very positive about their Teacher Edition. Teachers’ ratings of helpfulness were higher than the helpfulness ratings of teachers using other programs although the difference fell just short of conventional levels of statistical significance (Math Connects n = 16, comparison group n = 9, t = 2.0, p = .06). Teachers were also asked to rate how easy, or difficult, it was to use their Teacher Edition. The data displayed in the Figure above indicates that the teachers using Math Connects were significantly more positive about the ease of use of their Teacher Edition than teachers using other programs (Math Connects n = 16, comparison group n = 9, t = 2.3, p < .03.)

Teachers using Math Connects were also positive about the Student Edition of their program (see Figures below). They rated the helpfulness of their Student Edition significantly higher than teachers using other programs (Math Connects n = 16, comparison group n = 9, t = 2.0, p < .05). They also rated the ease of use for their Student Edition more positively than teachers using other programs (Math Connects n = 16, comparison group n = 9, t = 2.9, p < .00).

Seventh grade teachers using Math Connects were positive about both the Teacher Edition and the Student Edition of the Math Connects program. They were very positive about the helpfulness of the Teacher Edition. Teachers using Math Connects rated their Teacher Edition as easier to use than teachers using other programs. Teachers using Math Connects were very strongly positive about the helpfulness of the Student Edition in Math Connects. Their ratings of the helpfulness of this text were significantly higher than the ratings provided by teachers using other programs.

3.3

2.7

1

1.5

2

2.5

3

3.5

4

Avera

ge R

esp

on

ses

(1=

No

t H

elp

ful…

4=

Extr

em

ely

Help

ful)

Seventh Grade Teachers: "All things considered, how helpful is the

Student Edition of the program you used to teach math this year?"


3.4

2.7

1

1.5

2

2.5

3

3.5

4

Avera

ge R

esp

on

ses

(1=

Very

Dif

ficu

lt…

4=

Very

Easy

)

Seventh Grade Teachers: "All things considered, how would you rate the ease of use of the Student Edition of the program you used to teach math

this year?"



Focused Assessments of Curriculum Materials

Vertical Alignment. One of the most primary characteristics of any curriculum is its vertical alignment. In their articulation of principles for effective mathematics education, The National Council of Teachers of Mathematics (NCTM) asserts that “(a) curriculum is more than a collection of activities; it must be coherent, focused on important mathematics, and well articulated across grades.” (NCTM Principles and Standards; see www.nctm.org). They go on to assert that “(i)n a coherent curriculum, mathematical ideas are linked to build upon one

another so that students’ understanding and knowledge deepen and their ability to apply mathematics expands.” The Teacher Reference Handbook of the Math Connects program states that Math Connects is vertically aligned into an articulated, coherent sequence of content. One important objective of this study was to test that assertion directly by asking all teachers in the study to evaluate the vertical alignment of the mathematics curriculum they used during the school year. They were asked their assessment directly. Results are displayed in the accompanying Figure.

Results indicate that both the teachers using Math Connects and the teacher using other programs had moderately positive evaluations of their programs’ vertical alignment. The Mean for the Math Connects teachers (n = 16) was 2.8 and the Mean for teachers using other programs (n = 9) was 2.4 (t

= 1.4, NS).

Balanced Instruction. Effective curricular materials enable teachers to provide balanced instruction. That is, to speak very generally, effective instruction requires that teachers provide a coherent blend of holistic learning activities along with direct instruction and other applications (see, for example Stanulis & Floden, 2009). The pre-development research that provided the foundation for Math Connects indicates that the program was intended to help teachers in their efforts to provide such balance (Edwards, 2009). The Teacher Reference Handbook of the Math Connects asserts that the program facilitates balanced instruction by providing resources for teaching conceptual understanding, computational and procedural skills and real-world problem solving within every unit. More generally, balanced instruction can be facilitated by integrating cross-curricular connections that link mathematical concepts and practices to other subjects of interest to students.

2.8

2.4

1

1.5

2

2.5

3

3.5

4

Avera

ge R

esp

on

ses

(1=

Str

on

gly

Dis

ag

ree…

4=

Str

on

gly

Ag

ree)

Seventh Grade Teachers: "The program I used to teach math this year is vertically aligned in a very

effective way."



Teachers’ perceptions relevant to balanced instruction were assessed in a number of ways. First, three closed-ended items on their posttest questionnaire asked them to assess the extent to which their materials provided resources for teaching conceptual understanding, computational skills and applied problem solving. The assumption is that a balanced program would be evaluated as equally effective on all these dimensions. The Figures below present the teachers’ responses by condition.

Perhaps the primary characteristic of balanced materials in mathematics is that they facilitate both the teaching of conceptual understanding and computational procedures. Teachers using Math Connects rated their materials highly on both dimensions. Thus the seventh grade teachers who used Math Connects indicated it was effective at teaching both conceptual understanding and computational procedures. Also, the middle school teachers using Math Connects rated their materials significantly higher than teachers using other programs on both dimensions. These differences are substantial and statistically significant. For the item on computation, Math Connects n = 16, comparison group n = 9, t = 2.4, p < .03. For the item on conceptual understanding, Math Connects n = 16, comparison group n = 9, t = 2.8, p < .01.

Teachers using Math Connects were also very positive about the problem-solving resources in Math Connects (see Figures below). The Math Connects teachers were significantly more positive about the applied, “real-world” problem solving resources in their program than teachers using other programs (Math Connects n = 15, comparison group n = 9, t = 2.4, p < .03). The Math Connects teachers were also significantly more positive about the “hands-on” activities in their materials than teachers who were not using Math Connects (Math Connects n = 16, comparison group n = 9, t = 4.5, p < .00).

3.1

2.4

1

1.5

2

2.5

3

3.5

4

Avera

ge R

esp

on

ses

(1=

Stro

ng

ly D

isag

ree…

4=

Stro

ng

ly A

gre

e)

Seventh Grade Teachers: "The program I used to teach math this year provides excellent resources for teaching computational and

procedural skills."


3.2

2.6

1

1.5

2

2.5

3

3.5

4

Avera

ge R

esp

on

ses

(1=

Str

on

gly

Dis

ag

ree…

4=

Str

on

gly

Ag

ree)


year provides excellent resources for teaching conceptual understanding."



• Seventh grade teachers using Math Connects were very positive about the problem solving resources within the Math Connects program. These teachers indicated that Math Connects is effective at helping them teach both conceptual understanding and computational procedures. The seventh grade teachers who used Math Connects also indicated that Math Connects provides effective resources for teaching practical, real- world problem solving by applying mathematical concepts. Teachers who used Math Connects provided significantly higher assessments of the Math Connects materials than teachers who used other programs.

The quantitative data reported above was supplemented by asking teachers to comment in an open-ended fashion about the effectiveness of the materials they had used in helping students develop problem solving skills. Fifteen of the 16 seventh grade teachers who used Math Connects did so. All responses were generally categorized as “positive,” “negative,” or “irrelevant” (teachers would sometimes comment in ways that did not address the question) by two independent coders. Disagreements were resolved by discussion. Representative comments are displayed in the Figure below. Positive and negative comments are included in proportion to their general frequency.

3.4

2.7

1

1.5

2

2.5

3

3.5

4

Avera

ge R

esp

on

ses

(1=

Str

on

gly

Dis

ag

ree…

4=

Str

on

gly

Ag

ree)

Seventh Grade Teachers: "The program I used to teach math this year contains very effective 'real-

world' problems that help students learn how to apply mathematical

knowledge."


3.1

2.1

1

1.5

2

2.5

3

3.5

4

Avera

ge R

esp

on

ses

(1=

Str

on

gly

Dis

ag

ree…

4=

Str

on

gly

Ag

ree)


year contains very effective hands- on activities that really help the students

learn math."



Math Connects Seventh Grade Teachers’ Comments on Problem Solving


“The Problem-Solving section at the end of each section was excellent. I usually assigned those problems for my students.” “Math Connects provided numerous practice problems for the students to receive adequate practice. I really like to use the H.O.T. problems to challenge the students. Of all the problem-solving materials, I found the graphic novels most useful for my classroom as they helped me set the tone for the day at the start of class.” “These were fun for small group activities.” “I made a lot of use out of the H.O.T. problems for each lesson. Having the abundance of problems in the blackline master resource book was helpful.” “These resources were helpful in the differentiated instruction of my classes. They were quick and easy to have the students pick up and do all or part of and a way for me to check for understanding of skills without giving a test.” “The amount of practice, the reading level and the interest level of the problems truly helped my students become more comfortable and more proficient with problem solving.” “I really liked the real-world examples. They were up-to-date for students to relate to and understand. I liked how the subjects were relevant to students' lives, such as cell phone usage or graphs that had data that ended in the early to mid-2000s. I appreciated having a book with these types of examples…” “My students became proficient at solving problems by using the plan of problem solving. Students also benefited from the problem-solving workbook.” “The H.O.T. problems were very challenging problems for some of my students, and I thought they were good for them to do. It pushed their level of thinking another step. The students liked the graphic novels.” “My students really liked the graphic novels and were excited when we did one. We did not use the H.O.T. problems at all. My success with word problems came through taking baby steps with my students to build confidence, so that by the end of the year they were confident and did not groan and complain when faced with those.” “The graphic novels are a great way to get students to problem solve; many times they did not know they were problem solving. The H.O.T. problems really challenge students, and those who struggle will often feel that they are incompetent at math. The struggling students like to try one of the H.O.T. problems when others are given, but they lack the skills to think outside the box and therefore do not really challenge themselves in the way that those problems are meant. The word problem practice was nice in that there were only a few problems with plenty of space to work out the problems.”


As a direct assessment of the final component of balanced instruction, teachers were also asked to assess the effectiveness of their curricular materials in facilitating cross-curricular connections. First, they were asked to evaluate it in a very direct manner and then they were invited to comment and explain their assessment in an open-ended manner. The Figure below displays the quantitative data teachers provided regarding the effectiveness of their materials in this regard. The seventh grade teachers using Math Connects were less positive about this aspect of the materials than they had been about other dimensions. Their ratings of the materials were functionally equivalent to the ratings of teachers using other programs

(Math Connects n = 16, comparison group n = 9, t = .7, NS).

Only 9 teachers provided open-ended comments regarding cross-curricular connections. Their responses were generally categorized as “positive,” “negative,” or “irrelevant” (teachers would sometimes comment in ways that did not address the question) by two independent coders. Disagreements were resolved by discussion. The Figure below displays representative comments.

In sum, there was substantial evidence suggesting that Math Connects is effective at providing balanced instruction. Teachers rated it very positively in its ability to facilitate both abstract, conceptual learning and more concrete computational procedures. They

were also positive about the effectiveness of the resources available to teach applied, real-world problem solving. Teachers were less positive about making cross-curricular connections with Math Connects. Some of their open-ended comments suggest that they did not have the time in this first year of implementation to fully explore how to make connections to other subjects.

2.82.6

1

1.5

2

2.5

3

3.5

4

Avera

ge R

esp

on

ses

(1=

Str

on

gly

Dis

ag

ree…

4=

Str

on

gly

Ag

ree)


year was very effective at helping me make cross- curricular connections."



Math Connects Seventh Grade Teachers’ Comments on Cross-Curricular Connections


“I used the Real-World Projects presented in a separate booklet rather than the ones at the beginning of the chapters. For instance, we did the Nutrition Mission and the one about planning a vacation. The students loved doing these as it gave them an opportunity to be creative and to present their project in a variety of ways.” “Cross-curricular and career connections have been a part of my teaching for years. However, I enjoyed having it right in the materials rather than coming from other sources or just my own knowledge.” “It was great to have several ideas per chapter on how to implement cross-curricular activities in my math class. Now, if I can just convince the other teachers on my team to do it as well I'll be all set.” “All of the mentioned resources were great, but I found that I did not use them as much as I think I would like to in the future. At the time, I was concerned with running behind and not getting to the nuts-and-bolts of the units. At times, I used the real-world lessons toward the end of the unit if we had time. In hindsight, I think it would be a nice way to start a unit to get the students involved and thinking a little bit about how the math can be used in real life.” “I did not have a lot of time to do as many of these as I would have liked. It is always useful to connect math to its real-world application to help the kids understand it better. I loved the projects, and it is a goal of mine to do several of them next year.” “While my classes did not do any of these projects, I'm sure they would have been fun. I didn’t use them because I found it hard to place them into the curriculum given the timeline we were on, and I didn’t think the activities selected had a deep, lasting meaning for the students. Overall, the book lacked connections to other subjects in terms of what students are learning in other 7th grade classes.” “I did not use any of these projects because the curriculum was so vigorous and there was not enough time to complete a project that takes a week. If I were to teach next year, I would probably try to incorporate some of these into my daily teaching. There were just so many resources to use this year, it was impossible to implement everything in year one.” “The projects were not as well researched or as detailed as I would have liked. I spent considerable time adjusting them. I enjoyed assigning them and feel they are a great way to include cross-curricular connections for the students.”


Differentiated Instruction. In the background review of research for Math Connects, Papa and Brown (www.glencoe.com/glencoe_research/research_maths.html) summarize the voluminous research on differentiated instruction. It is widely recognized that effective instruction requires adapting both instructional content and instructional practices to meet the students at their individual levels of readiness. This can be a major challenge for most teachers as they encounter classrooms that have, at the same time, students who are above grade level in ability and ready for enrichment activities, students who are below grade level for any of a variety of reasons and students for whom English is not their first language. According to the Teacher Reference Handbook of Math Connects, one major component of the Math Connects program is the provision of multiple options for providing differentiated instruction.

Results suggest that the seventh grade teachers believe the Math Connects materials are effective at helping to provide differentiated instruction. First, the adjacent Figure displays the

response data for a general question asking all teachers (Math Connects and comparison group) about the effectiveness of their programs in the area of differentiated instruction. The teachers using Math Connects were very positive about this dimension of the materials. The Math Connects teachers were significantly more positive about their materials than teachers using other programs (Math Connects n = 16, comparison group n = 9, t = 2.8, p < .01).

Specific challenges to teachers regarding differentiating instruction were also explored directly. The Figures below display the data reflecting the teachers’ perceptions of their programs’ effectiveness at helping students who were below grade level in mathematics and their programs’ effectiveness with students above grade level.

First, the teachers using Math Connects were significantly more positive about the effectiveness of their materials with students below grade level than the teachers who used other programs (Math Connects n = 16, comparison group n = 9, t = 2.8, p < .01). The Math Connects teachers also indicated that their materials were effective in teaching students who were above grade level in mathematics. However, the test of Means for differences between conditions did not meet conventional tests of statistical significance (Math Connects n = 14, comparison group n = 9, t = 1.8, NS).

3.2

2.6

1

1.5

2

2.5

3

3.5

4

Avera

ge R

esp

on

ses

(1=

Str

on

gly

Dis

ag

ree…

4=

Str

on

gly

Ag

ree)


year was very effective in helping me to provide differentiated instruction."



Finally, teachers were asked about the effectiveness of their materials in teaching students who were English Language Learners. The Figure below displays the average responses for teachers in both conditions to an item asking about this dimension of differentiating instruction. While the teachers using Math Connects were positive about this aspect of their materials, the test of Means between conditions did not meet conventional tests of statistical significance (Math Connects n = 14, comparison group n = 9, t = 1.9, NS).

• The seventh grade teachers who used Math Connects were positive about its effectiveness in providing differentiated instruction. In reference to students below grade level, the Math Connects teachers were significantly more positive than teachers using other programs. The Math Connects teachers were also positive about the effectiveness of Math Connects in working with students who were above grade level and for students for whom English is not their native language.

2.9

2.1

1

1.5

2

2.5

3

3.5

4

Avera

ge R

esp

on

ses

(1=

Str

on

gly

Dis

ag

ree…

4=

Str

on

gly

Ag

ree)


year was very effective for teaching students who were below grade level

in mathematics."


3.1

2.6

1

1.5

2

2.5

3

3.5

4

Avera

ge R

esp

on

ses

(1=

Str

on

gly

Dis

ag

ree…

4=

Str

on

gly

Ag

ree)


year was very effective for providing enrichment for students who were above grade level in mathematics."


2.8

2.1

1

1.5

2

2.5

3

3.5

4

Avera

ge R

esp

on

ses

(1=

Str

on

gly

Dis

ag

ree…

4=

Str

on

gly

Ag

ree)


year was very effective for teaching students who were English Language

Learners."



Teachers also provided open-ended comments explaining their assessments. Their comments support the positive closed-ended data. The Figure below displays selected comments from both the seventh grade teachers who used Math Connects during the year. The complete text of all comments from all teachers is available in an appendix.

Seventh Grade Math Connects Teachers’ Comments on Resources for Differentiating Instruction


“Differentiated instruction was the one thing that I really wanted to do well this year, and the aforementioned resources were so very helpful in working toward that goal. I typically teach to the struggling learner, and I found so many resources to help students gain confidence, but more so, I was able to help students challenge themselves and move beyond what was being taught in the classroom. I had a number of parents, this year in particular, who were pleased with the extra activities provided to challenge their child.” “I used these types of activities while developing a more differentiated approach in my classroom. I appreciated the variety of ideas the Math Connects series incorporates into the text. It is so difficult to come up with a variety of new activities, and this text helped make this easier.” “Math Connects was very helpful with the differentiated instruction in my classroom. From the abundance of enrichment activities to the guided notes workbook, it made it easy for me to give instruction and work for students at their appropriate level. The students also enjoyed the graphic novels which I used often to start the class.” “The variety of resources available made it easier to differentiate instruction for my students, especially my lower-level students. The homework options saved me some planning time.” “I found all of the resources helpful; though some of the projects were too involved for seventh grade…I was also disappointed not to have examples of the projects.” “Math Connects provides enough materials to reach every student. For those students who struggle with math, the online materials were definitely used at home to help with their understanding of concepts and homework. The personal tutor was a great resource for some students. At the same time, this was difficult because most students who are struggling are not going to take the time to figure it out for themselves. The book definitely expected students to grasp a concept the first time they learned it because it was then used later to emphasize new concepts.” “There are so many available resources it was hard to determine what to use when. Sometimes less is more.”


Digital Resources. The NCTM Principles for Teaching Mathematics state directly that technology is essential in the teaching and learning of mathematics. Advances in technologies of many types have had a great impact on the teaching and learning of mathematics in recent years (see Wong, 2003) and educators are being encouraged to continue to explore the potential of new digital resources (Martin & Spear, 2009). One major objective of this evaluation was to gather information from teachers regarding the effectiveness of the wide range of digital resources available to them.

All teachers in this study (experimental and comparison) were asked to evaluate the digital resources that are a part of their mathematics curricular materials, and the Math Connects teachers were asked direct questions about specific components of the Math Connects materials. These digital resources come in two basic forms. First, there are materials on a CD-ROM for both teachers and students. Also, most programs provide online resources available via the Internet. All teachers were asked directly about the effectiveness of these components and comparisons between conditions were made wherever appropriate. Additionally, at the seventh grade level all students in the study were asked a short series of questions asking them to report on the digital resources available in their mathematics classes.

While the response rates for the seventh grade teachers were higher than those of the elementary grade teachers, this evaluation data is still marked by a relatively high percentage of missing data. It is reasonable to expect that teachers may have struggled to implement the digital resources that came with Math Connects (see Kysilka, Geary & Schepise, 2002).

CD-ROM resources. The data regarding CD-ROM resources from the seventh grade Math Connects teachers is reported in the Tables below. The response rate from the seventh grade teachers was proportionately much higher than that of the elementary grade teachers. This may imply that middle school teachers are more comfortable with digital resources or that their classrooms are more likely to include necessary hardware. The objective data from the middle school teachers indicated that the CD-ROM resources were generally easy for them to use. When asked directly, the majority of the Math Connects teachers indicated that the CD-ROM resources included in Math Connects were either “easy” or “very easy” to use. The Table below displays the frequency data for teachers who responded to this item. .

Math Connects Seventh Grade Teachers’ Assessments of Ease of Use for CD- ROM Resources


Difficult to use

Easy to use

Very easy to use

No response

All things considered, how would you rate the ease of use of the resources available on CD- ROM for teachers?

0 1 6 5 4

The teachers were asked directly about individual CD-ROM resources available in Math Connects. The Table below displays their assessments of each component. Responses suggest that teachers found the TeacherWorksTM CD-ROM helpful and they felt that the StudentWorksTM CD-ROM was helpful to their students. The teachers were also generally positive in their overall rating of the CD-ROM resources.


Math Connects Seventh Grade Teachers’ Assessments of CD- ROM Resources

Not

helpful at all


Mostly helpful

Extremely helpful

No response

How helpful was the TeacherWorksTM CD- ROM? 0 2 5 3 6 How helpful was the Interactive Classroom CD-ROM? 1 3 4 3 5 How helpful was the Exam View® Assessment Suite CD- ROM? 0 4 4 0 9 How helpful was the StudentWorksTM CD- ROM for your students? 1 1 6 1 7 How helpful was the StudentWorksTM CD- ROM for you? 0 2 4 2 8 All things considered, how helpful were the resources available on CD- ROM for teachers? 0 2 5 6 3

The Math Connects teachers were asked to identify the specific feature of the TeacherWorksTM CD-ROM that they found most helpful. Nine of the 13 teachers responding indicated that the Chapter Resource Masters/Teacher Resources were most helpful. Math Connects teachers also identified how they used the Exam View® Test Generator. Five of the

nine teachers responding indicated that they used the formatting feature of that resource

The accompanying Figure displays the response data from the seventh grade students regarding any CD-ROM resources that may have been a part of their math classes. Students using Math Connects were significantly more positive about the CD-ROM resources than students using other programs (Math Connects n = 587, comparison group n = 210, t = 6.3, p < .00). However, it should be noted that, overall the students’ ratings were only moderately positive (less than 3 on a 4-point scale).

Finally, teachers were also invited to comment on the general usefulness of the CD-ROM resources. Representative comments are

displayed in the Figure below. The full text of all comments is available in an appendix.

2.8

2.4

1

1.5

2

2.5

3

3.5

4

Avera

ge R

esp

on

ses

(1=

No

t H

elp

ful..

.4=

Extr

em

ely

Help

ful)

Seventh Grade Students: "How helpful to you was the CD- ROM that came

with your math book?"



Math Connects Seventh Grade Teachers’ Comments on CD- ROM Resources

Please make a brief statement that summarizes your opinion of the CD- ROM resources provided to you. You may include both positive and negative comments.

“I was grateful to have the technology materials and to be able to quickly use them with students!” “The CD-ROM resources were well designed and very easy to use. It was a treat to have anything I was looking for at my fingertips. Great job.” “It was nice to have the Super DVD where a bunch of material was in one place.” “I looked through the CD-ROM resources, but did not really use them. I think they would have helped me with making the assessments I needed, but I didn't get a chance to try it.”

Online digital resources. Digital resources are often available to both teachers and students via the Internet. In Math Connects students had a variety of practice activities and self assessment options that they could access online. The ratings of the online resources for the seventh grade Math Connects teachers are reported in the Tables below.

Math Connects Seventh Grade Teachers’ Assessments of Online Resources

Not

helpful at all


Mostly helpful

Extremely helpful

No response

How helpful was the online version of StudentWorksTM PLUS for your students? 0 3 3 2 8 How helpful was the online version of StudentWorksTM PLUS for you? 0 3 3 2 8 How helpful were the online resources for your students? 0 3 4 3 6 How helpful were the online resources for you? 0 4 5 2 5 How helpful were the online self assessment resources for your students? 0 4 5 2 5 How helpful were the online self assessment resources for you? 0 1 5 2 8

Math Connects teachers were also asked to assess how difficult, or easy, it was to use the online resources available with Math Connects. Their responses (see Table below) indicated that most found both the resources for teachers and the resources for students at least moderately easy to use.


Math Connects Seventh Grade Teachers’ Assessments of Ease of Use for Online Resources


Difficult to use

Easy to use

Very easy to

use No

response All things considered, how would you rate the ease of use of the online resources available for you?

0 1 7 4 4

All things considered, how would you rate the ease of use of the online resources available for your students? 1 3 6 2 4

The seventh grade students also provided a very general assessment of the online resources available to them. The Figure below presents a comparison of ratings by students who used Math Connects and students using other programs. Students using Math Connects were significantly more positive about their online resources than students using other programs (Math Connects n = 604, comparison group n = 171, t = 4.6, p < .00).

General assessment of digital resources. One central interest in this study was to compare the attitudes of teachers using Math Connects with those of teachers using other programs regarding the general effectiveness of the digital resources available to them. All teachers (Math Connects and comparison group) responded to two closed-ended items asking them to provide a general assessment of the digital resources that were a part of their curricular materials. The Figures below display the comparison of the average responses for the teachers using Math Connects and teachers using other programs. The seventh grade teachers using Math Connects were significantly more positive about the digital resources for students than teachers who used other programs (Math Connects n = 15, comparison group n = 6; t = 2.5, p < .02). The difference between the average responses on the

item referring to digital resources for teachers was not statistically significant (Math Connects n = 15, comparison group n = 6; t = 1.6, NS).

2.8

2.4

1

1.5

2

2.5

3

3.5

4

Avera

ge R

esp

on

ses

(1=

No

t H

elp

ful…

4=

Very

Help

ful)

Seventh Grade Students: "Was your math program's Web site helpful to

you?"



• Middle school teachers who used Math Connects were positive about the general effectiveness of the technology resources in the program. They were significantly more positive about the resources available for students than teachers who used other programs.

In sum, it appears as though teachers using the Math Connects program had positive impressions of the digital resources that are a part of the materials, but they were not able to integrate these resources as much as they might have liked. This impression is validated by the Math Connects teachers’ expressions of their future intentions. They were asked, “If you were to use Math Connects next year, would you use the technology resources more often, less often or about the same as this year?” Twelve of the 15 respondents (80%) indicated that they would use those resources more often.

Assessment and Intervention. Because of the continuous emphasis on accountability and assessment in mathematics instruction, one central focus of this evaluation was on the teachers’ perceptions of the assessment materials provided within the programs they were using. As Wilson and Kenny (2003) state directly, valid assessment is essential to effective mathematics instruction. The Teacher Reference Handbook of Math Connects asserts that every effort has been made with that program to integrate assessment activities into instruction. This notion seems to be directly consonant with the framework suggested by Lesh and Lamon (1992) when they advocate for the seamless integration of instruction and assessment.

Many middle school mathematics materials include initial materials designed to provide a review of materials from preceding years that allow teachers to both reinforce concepts and begin to diagnose any needs for intervention with students who may need further instruction (this feature is labeled the “Start Smart” section within the Math Connects materials). Teachers were asked directly about this dimension of their assessment resources.

3.4

2.7

1

1.5

2

2.5

3

3.5

4

Avera

ge R

esp

on

ses

(1=

Str

on

gly

Dis

ag

ree…

4=

Str

on

gly

Ag

ree)

Seventh Grade Teachers: "The program I used to teach math this year has very effective technology

resources for students."


3.4

2.7

1

1.5

2

2.5

3

3.5

4

Avera

ge R

esp

on

ses

(1=

Str

on

gly

Dis

ag

ree…

4=

Str

on

gly

Ag

ree)

Seventh Grade Teachers: "The program I used to teach math this year has very

effective technology resources for teachers."



The accompanying Figure displays the data relevant to the review and reinforcement materials for both the Math Connects and the comparison group teachers. Results indicate that the teachers who used Math Connects and teachers who used other programs were nearly equivalent in their assessments of their programs’ review and reinforcement functions. Both were moderately positive about their materials.

Many programs also provide assessment resources that allow teachers to conduct initial assessments at the beginning of the school year and also at the beginning of every chapter to diagnose any needs for intervention. The Figure below presents a summary of the teachers’ responses to a question asking about the helpfulness of their programs’ initial assessment resources. While these results indicate that the Math Connects teachers were positive about their materials, the testing of differences between groups was hampered by the fact that only 4 teachers in the comparison group responded to this item.

Most mathematics curricular materials also provide formative assessment resources that allow teachers to quickly gauge their students’ progress as they move through the materials. These resources typically take the form of mid-chapter quizzes or other forms of short-term testing (within Math Connects these are provided in resources such as the Mid-Chapter Check, Quick Check, or Check What You Know). All teachers in the study were asked for their direct assessment of the formative assessment resources available in their programs (see

Figure below). The data displayed there indicates that the teachers using Math Connects were significantly more positive about their formative assessment resources than teachers who used other programs (Math Connects n = 15, comparison group n = 6, t = 3.3, p < .00).

2.4 2.3

1

1.5

2

2.5

3

3.5

4

Avera

ge R

esp

on

ses

(1=

Str

on

gly

Dis

ag

ree…

4=

Str

on

gly

Ag

ree)

Seventh Grade Teachers: "In the program you used to teach math this year, how helpful was the review and reinforcement of learning from last

year?"


3.1

2.3

1

1.5

2

2.5

3

3.5

4

Avera

ge R

esp

on

ses

(1=

No

t H

elp

ful

at

all

…4

=V

ery

Help

ful)

Seventh Grade Teachers: "In the program you used to teach math this

year, how helpful were the initial assessment resources?"



Most programs also provide summative assessment resources in the forms of chapter tests or practice for standardized tests. The accompanying Figure displays teachers’ responses to a question asking them directly about the helpfulness of the summative assessment resources in their programs. Teachers using Math Connects were significantly more positive about their summative assessment resources than teachers using other programs (Math Connects n = 15, comparison group n = 7, t = 4.4, p < .00).

Finally, it is reasonable to assume that teachers’ use of available assessment resources would be affected by the amount of effort that is required to administer and interpret the assessment. That is, no matter how helpful a resource might be, if it cannot be used efficiently it will be of limited utility to the teacher. So, all teachers in the study were asked to assess how easily their assessment resources were to use. The accompanying Figure displays the response data for this item by condition. Results indicate that the Math Connects teachers were very positive about the ease of use of their assessment resources. The average rating for the Math Connects teachers was 3.5 on a 1-4 scale. This average rating for the teachers using Math Connects was significantly higher than the average rating provided by teachers using other programs

(Math Connects n = 16, comparison group n = 9, t = 2.7, p < .01).

3.6

2.5

1

1.5

2

2.5

3

3.5

4

Avera

ge R

esp

on

ses

(1=

No

t H

elp

ful

at

all

…4

=V

ery

Help

ful)

Seventh Grade Teachers: "In the program you used to teach math this year, how helpful were the formative

assessment resources?"


3.5

2.4

1

1.5

2

2.5

3

3.5

4

Avera

ge R

esp

on

ses

(1=

No

t H

elp

ful

at

all

…4

=V

ery

Help

ful)

Seventh Grade Teachers: "In the program you used to teach math this

year, how helpful were the summative assessment resources?"


3.5

2.9

1

1.5

2

2.5

3

3.5

4

Avera

ge R

esp

on

ses

(1=

Very

Dif

ficu

lt…

4=

Very

Easy

)

Seventh Grade Teachers: "Overall, how would you rate the ease of use for the assessment resources in the program you used to teach math this

year?"



• The seventh grade teachers who used Math Connects were very positive about the assessment resources in Math Connects. They were significantly more positive about both their formative and summative resources than teachers who used other programs. The teachers who used Math Connects were also more positive about the ease of use of their assessment resources than teachers who used other programs.

Blended Instruction. Mathematics education in recent years has been marked by a call for a move from “traditional” methods grounded in memorization and reproducing existing solutions to “reform-based” methods grounded in hands-on activities and active investigations designed to help students come to more personally relevant understandings of mathematical concepts (see Findell, 1996; Fraivillig, Murphy & Fuson, 1999; Greenes, 1996; Sood & Jitendra, 2007). The Math Connects program includes a supplemental program (IMPACT Mathematics) which provides reform-based lessons for teachers to use in coordination with the basal curriculum materials. All teachers in this study were asked directly about their use of IMPACT Mathematics or other reform-based materials available to them.

Results indicate that the Math Connects teachers, in their first year of program implementation, did not access the IMPACT Mathematics materials very frequently. When asked directly about usage, a total of 8 of the 16 teachers responding (50%) indicated that they never used IMPACT Mathematics (6 others indicated they used it “occasionally” and only 3 indicated they used it “frequently.”) Some teachers’ open-ended comments mentioned that they may not have had enough time to implement materials from IMPACT Mathematics in their first year of using Math Connects. These comments were typical:

“(IMPACT Mathematics) should be used as an extra resource for extension activities. Time is always an issue. I feel like I barely have enough time with the kids to teach them the basics. I would love to be able to use all of these great activities.”

“IMPACT Mathematics can be used both before and after introducing a concept. I sadly did not use this and hope to incorporate it into new material next year.”

Of the teachers who did use IMPACT Mathematics, most found it easy to integrate those lessons into the materials from the basal Math Connects text. When asked directly, 7 of 8 respondents (88%) indicated that the IMPACT Mathematics lessons were “easy” or “very easy” to integrate into their basal lessons.

Mathematical Literacy. The learning of mathematics interacts with the development of reading competencies (Adams & McKoy, 2007; van Garderen, 2004; Walker, Zhang & Surber, 2008). Students must be able to comprehend math problems expressed in language. Also, a central part of learning mathematics is the development of mathematical literacy. Put simply, mathematical literacy occurs as students develop a sound understanding of the vocabulary of mathematics (see Jablonka, 2003). Thompson and Chappell (2007) explain how the NCTM standards emphasize that learning effective communication and representation of mathematical concepts is absolutely fundamental to learning mathematics. So, one objective of this evaluation was to explore how the Math Connects program facilitates


mathematical literacy and discern the appropriateness of the vocabulary used within Math Connects.

The seventh grade teachers using the Math Connects program were very positive about the effectiveness of Math Connects in helping their students develop mathematical literacy (see accompanying Figure). The Mean response score for the teachers using Math Connects was significantly higher than the Mean score for teachers using other programs (Math Connects n = 16, comparison group n = 9, t = 2.9, p. < .00).

All teachers in the study also provided open-ended comments about the effectiveness of their materials in helping students develop mathematical literacy. The Figure below presents representative comments from the

teachers using Math Connects. The full text of all comments from all teachers is available in an appendix.

3.1

2.3

1

1.5

2

2.5

3

3.5

4

Avera

ge R

esp

on

ses

(1=

Str

on

gly

Dis

ag

ree…

4=

Str

on

gly

Ag

ree)


year is excellent at helping students develop mathematical literacy."



Math Connects Seventh Grade Teachers’ Comments on Mathematical Literacy


“My students’ math vocabulary improved and so did their writing skills. In general, their writing and spelling was poor at the beginning, but each student showed improvement. I am always looking for literature to connect with lessons.” “The Reading to Solve Problems was very effective. I also liked the vocabulary check at the end of the chapters and how the words were highlighted in the chapters.” “We did not read the chapter as a class. The other materials that helped you relate reading and writing in math to interesting topics were extremely useful. Any time the book had the students write their own word problem or explanation, my students definitely grew in their depth of understanding. Also, how to break apart and read word problems was very valuable.” “I was pleased with the way the notes in the margins help the students read and understand the material.” “We studied math vocabulary, and I required that the students use the correct terminology so we sounded like mathematicians. They were very good at that. We did extensive writing in math activities to help prepare them for the open-ended questions they are required to answer both in class and as part of their state testing.” “While outside reading was never assigned, I could see student vocabulary growing as the year progressed. Students occasionally had to do some written responses about their understanding of a concept, and it was there that it became evident that they had progressed.” “I liked how the text presented vocabulary. The illustrations and examples that went along with each section were easy for students to grasp. I also liked the use of the Noteables™ and Foldables™ for students to use as a tool to navigate through the text.” “The development of mathematical literacy was not an effective component of this book. It used the words in a sentence, but did not break it down so that students could create a definition that would help them to understand what it meant. This part of the lesson would often take more time from the lesson to make sure that students were able to grasp meaning and not just memorize a definition that they may not ever use. I feel that students need to write the definition in their own words so that they make connections and understand what they are reading. I liked the techniques that were used in the Reading and Writing sections. Students like to solve problems and often do not understand what the question is asking. The students that were willing to try these techniques did very well. The Writing in Math questions had the students thinking, but with limited experience they would often just restate the question.”


Teacher Professional Development. The Math Connects program includes a number of resources for professional development. For example, the Teacher Reference Handbook includes sections with articles on a number of topics (e.g., intervention strategies, data-driven decision making). There are also resources within the Teacher Edition and on the Internet. Many teachers’ comments were very positive (see Figure below), but some indicated they did not have time to use these resources or they did not feel a need for them.

Math Connects Seventh Grade Teachers’ Comments on Professional Development

Please make a brief statement that summarizes your opinion of the professional development resources available to you through the program you used to teach mathematics this year. You may include both positive and negative comments.

“The professional development resources were of the highest level of development. I was very pleased with every aspect of the resources available with the Math Connects series.” “It was great to have extra tips and reminders throughout the chapters. Everything was super-friendly and easy to use.” “I feel the resources are very well planned and presented. I used the workbooks for differentiating and for ESL far more than I did the reference handbook.” “The potential to be a fantastic teacher is there. You just need to take the time to unlock the resources that are available.” “I did not use it this past year, but I will spend more time looking at the material next year.” “I did not look to Math Connects for professional development given that I had several things already lined up.” “It is a lovely idea, but can you also provide about six more hours a day for me to be able to utilize it?” “We have professional development offered through our district, so this really does not interest me.”


When asked directly about the professional development resources available to them, the Math Connects teachers were generally positive. The Table immediately below displays the response frequencies for a question which asked teachers about the effectiveness of these resources. A majority of teachers who responded indicated the resources were “mostly effective” or “extremely effective.”

Math Connects Seventh Grade Teachers’ Ratings of Professional Development Resources

Not

effective at all

Only

slightly effective

Mostly effective

Extremely effective

Did not use

Overall, how would you rate the professional development resources available to you with Math Connects?

2

1

4

2

7


General Conclusions

As noted initially, five specific evaluation questions guided this assessment. The first two focused on issues of student learning while the last three were aimed at assessing teachers’ and students’ attitudes about the Math Connects curriculum materials. Our general conclusions are organized around these two basic concerns.

Student Learning

The most fundamental issue, of course, centers on the basic effectiveness of Math Connects in helping students learn mathematics. The major portion of our evaluation efforts centered on investigating this issue. The student testing, performed with one of the most widely accepted and trusted tests of student abilities, the TerraNova, validates that Math Connects can be used to effectively improve students’ learning of mathematics. Students at all grade levels showed substantial, statistically significant knowledge gains in mathematics using the Math Connects materials.

Furthermore, the student testing validates that Math Connects is effective with all types of students. Analyses verified that Math Connects works equally well with both male and female students and students of all races. The only exception occurred when middle school Latino students improved more than students of other races. The testing data also validates that Math Connects works equally well with all types of teachers. Generally, student knowledge gains were consistent regardless of teachers’ confidence in teaching math (teacher self efficacy) and regardless of the teachers’ basic orientation to teaching math (their math beliefs).

Of course, one major consideration is how well Math Connects performs in comparison to other programs that might be used in the same schools. Making valid comparisons can be somewhat problematic because in cases such as this, the teachers in the experimental condition (using Math Connects) are in their first year of program implementation. Conventional wisdom would suggest that it may take two or three years for a teacher to be able to use some materials to their fullest potential. Even so, students using the Math Connects materials showed knowledge gains at least equal to the knowledge gains of students using other programs. In fact, second grade students using Math Connects showed significantly larger knowledge gains than students using other programs. There were no cases where students using other programs showed larger knowledge gains than the students using Math Connects. Thus it is fair to say that the Math Connects materials compare very favorably to other programs in terms of student learning.


Teacher and Student Opinions of Math Connects

One major assumption of this evaluation was that teachers’ assessments of materials are as meaningful as the objective student test data and a valid evaluation of any program must include both of those elements. Thus great effort was devoted to gathering detailed teacher perceptions of how the Math Connects materials functioned in their classrooms.

First, it is fair to say that teachers who used Math Connects were very positive about nearly every component of the program. And, teachers who used Math Connects saw it to be very effective in several important domains. Teachers generally viewed Math Connects as easy to implement and they were very positive about basic pragmatic considerations such as the reading level of the materials, the pacing and the amount of practice problems provided for students. They were also very positive about primary resources such as the Teacher Edition and the Student Edition of the program. The Math Connects teachers were generally much more positive about these elements than teachers using other programs.

Teachers also gave Math Connects very positive assessments regarding crucial functions of curriculum materials. The teachers using Math Connects indicated that their materials were very effective at addressing the grade specific curriculum focal points advanced by the NCTM. Regarding balanced instruction, teachers using Math Connects indicated that their materials were effective for teaching both conceptual understanding and computational procedures. They indicated that the program was effective at teaching applied problem solving to their students. Finally, the teachers using Math Connects were very positive about the assessment resources available in the program. They were more positive about these resources than teachers using other programs regarding both initial assessment, formative assessment and summative assessment.

The evaluation of some program components was hindered by the fact that teachers did not make use of some components with any regularity. This was the case with the digital resources. The majority of teachers in the study did not use many of the resources available to them. Those who did use these resources were very positive about them as were the students who used them.


References

1) Abrami, P.C. & Bernard, R.M. (2006). Research on distance education: In defense of field experiments. Distance Education, 27, 5-26.

2) Adams, T.L. & Lowery, R.M. (2007). An analysis of children’s strategies for reading mathematics. Reading and Writing Quarterly, 23, 161-177.

3) Alderman, M. K. Motivation for Achievement: Possibilities for Teaching and Learning (2nd Ed.) Mahwah, N.J. : Lawrence Erlbaum Associates.

4) Babbie, E. (1992). The Practice of Social Research (6th Ed.). Belmont, CA: Wadsworth.

5) Bandura, A. (1986). Social Foundations of Thought and Action: A Social Cognitive Theory. Englewood Cliffs, NJ: Prentice-Hall.

6) Bandura, A. (1997). Self-efficacy: The exercise of control. New York: W.H. Freeman.

7) Capraro, M.M. (205). A more parsimonious mathematics belief scale. Academic Exchange Quarterly, 3, 1096-1453.

8) Carpenter, T.P., Dossey, J.P., & Koehler, J. (2004). Introduction. In T.P. Carpenter, J.P. Doresey & J. Koehler (Eds.), Classics in mathematics education research (pp. 1-6). Reston, VA: The National Council of Teachers of Mathematics.

9) Cook, T.D. & Campbell, D.T. (1979). Quasi-experimentation: Design and analysis issues for field settings. Boston: Houghton-Mifflin.

10) Cook, T.D. (2004). Randomized experiments in education: Why are they so rare? Working paper WP-02-19, Institute for Policy Research, Northwestern University, Evanston, IL.

11) Edwards, L. (2009). Pre-development research: The Research Base for PreK-12 Mathematics. Macmillan/McGraw-Hill and Glencoe.

12) Enochs, L., Smith, P.L. & Huinker, D. (2000). Establishing factorial validity of the Mathematics Teaching Efficacy Belief instrument. School Science and Mathematics, 100 194-202.

13) Fennema, E., Carpenter, T., & Loef, M. (1990). Mathematics beliefs scales. Madison, WI: University of Wisconsin, Madison.

14) Findell, C. (1996). Mathematics education then and now: The need for reform. Journal of Education, 178, 3-13.

15) Fraivillig, J.L., Murphy, L.A. & Fuson, K.C. (1999). Advancing mathematical thinking in Everyday Mathematics classrooms. Journal for Research in Mathematics Education, 30, 148-170.

16) Greenes, C. (1996). Investigations: Vehicles for learning and doing mathematics. Journal of Education, 178, 35-49.


17) Hays, W.L. (1981). Statistics. New York: Holt, Rinehart & Winston.

18) Henson, R.K., Kogan, L.R. & Vacha-Haase, T. (2001). A reliability generalization study of the teacher efficacy scale and related instruments. Educational and Psychological Measurement, 61, 404-420.

19) Hubbard, R. & Armstrong, J.S. (2005). Why we really don’t know what “statistical significance” means: A major educational failure. Available at www. marketing.wharton.upenn.edu.

20) Jablonka, E. (2003). Mathematical literacy. In A.J. Bishop, M.A. Clements, C. Keitel, J. Kilpatrick & F.K.S. Levine (Eds.), Second International Handbook of Mathematics Education (pp. 75-102). Boston: Kluwer Academic.

21) The Joint Committee on Educational Standards for Educational Evaluation, (1994). The program evaluation standards: How to assess evaluations of educational programs (2nd ed.). Thousand Oaks, CA: Sage.

22) Lesh, R. & Lamon, S. (1992). Trends, goals and priorities in mathematics assessment. In R.A. Lesh & S. Lamon (Eds.), Assessment of Authentic Performance in School Mathematics (pp. 3-16). Washington D.C.: AAAS Press.

23) Kysilka, M.L., Geary, M. & Schepise, S. (2002). The complexity of teaching in the information age school. Curriculum and Teaching Dialogue, 4, 59-65.

24) Marks, H.M. (2000). Student engagement in instructional activity: Patterns in the elementary, middle and high school years. American Educational Research Journal, 37, 153-184.

25) Martin, T.S. & Spear, W.R. (2009). Mathematics teaching today. Teaching Children Mathematics, 15, 400-403.

26) McCaffrey, D., Hamilton, L., Stecher, B., Klein, S., Bugliari, D. & Robyn, A. (2001). Interactions among instructional practices, curriculum, and student achievement: The case of standards-based high school mathematics. Journal for Research in Mathematics Education, 32, 493-517.

27) National Council of Teachers of Mathematics (2006). Curriculum Focal Points for Prekindergarten through Grade 8 Mathematics: A Quest for Coherence. Available at www.nctm.org.

28) Nespor, J. (1987). The role of beliefs in the practice of teaching. Journal of Curriculum Studies 19, 317-328.

29) Pajares, F. & Urdan, T. (1996). An exploratory factor analysis of the Mathematics Anxiety Scale. Measurement and Evaluation in Counseling and Development, 29, 35-47.

30) Papa, R. & Brown, R. (undated). The research for Math Connects Grades PreK-8. Available at www.glencoe.com/glencoe . _research/research_maths.html

31) Pedhazur, E.J. & Schmelkin, L.P. (1991). Measurement, Design and Analysis: An Integrated Approach. Hillsdale, NJ: Erlbaum.

http://www.glencoe.com/glencoe_research/research_maths.html�


32) Pearson, A. & Champagne, P.D. (2003). Subject domain: What is being measured? In NAEP Validity Studies: An Agenda for NAEP Validity Studies (pp. 5-12). Washington, DC: National Center for Educational Research.

33) Pintrich, P.R. & DeGroot, E.V. (1990). Motivational and self-regulated learning components of classroom academic performance. Journal of Educational Psychology, 82, 33-40.

34) Schoenfield, A.H. (2006). What doesn’t work: The challenge and failure of the What Works Clearinghouse to conduct meaningful reviews of studies of mathematics curricula. Educational Researcher, 35, 13-31.

35) Shadish, W.R., Cook, T.D. & Campbell, D.T. (2002). Experimental and Quasi-Experimental Designs for Generalized Causal Inferences. Boston: Houghton-Mifflin.

36) Sood, S. & Jitendra, A.K. (2007). A comparative analysis of number sense instruction in reform-based and traditional mathematics textbooks. The Journal of Special Education, 41, 145-157.

37) Stanulis, R.N. & Floden, R.E. (2009). Intensive mentoring as a way to help beginning teachers develop balanced instruction. Journal of Teacher Education, 60, 112-122.

38) Thompson, D.R. & Chappell, M.F. (2007). Communication and representation as elements in mathematical literacy. Reading and Writing Quarterly, 23, 179-196.

39) United State Department of Education, (2008). Foundations for Success: The Final Report of the National Mathematics Advisory Panel.

40) Van Garderen, D. (2004). Reciprocal teaching as a comprehension strategy for understanding mathematical word problems. Reading and Writing Quarterly, 20, 225-229.

41) Walker, C.M., Zhang, B. & Surber, J. (2008). Using a multidimensional differential item functioning framework to determine if reading ability affects student performance in mathematics. Applied Measurement in Education, 21, 162-181.

42) Wilson, L.D. & Kenny, P.A. (2003). Classroom and large scale assessment. In J. Kilpatrick, W.G. Martin & D. Schifter (Eds.), A Research Companion to Principles and Standards for School Mathematics (pp. 53-67). Reston, VA: The National Council of Teachers of Mathematics.

43) Wong, Ngai-Ying (2003). The influence of technology on the mathematics curriculum. In A.J. Bishop, M.A. Clements, C. Keitel, J. Kilpatrick & F.K.S. Levine (Eds.), Second International Handbook of Mathematics Education (pp. 271-321). Boston: Kluwer Academic.

Math Connects National Field Study - Amazon S3 · 2015-09-29 · National Field Study Final Results...

Documents

Transcript of Math Connects National Field Study - Amazon S3 · 2015-09-29 · National Field Study Final Results...