ED 457 261

AUTHORTITLE

INSTITUTIONPUB DATENOTEAVAILABLE FROM

PUB TYPEJOURNAL CITEDRS PRICEDESCRIPTORS

IDENTIFIERS

ABSTRACT

DOCUMENT RESUME

UD 034 400

Mitchell, Ruth; Barth, PatteNot Good Enough: A Content Analysis of Teacher LicensingExaminations. How Teacher Licensing Tests Fall Short.Education Trust, Washington, DC.1999-00-0026p.; Theme issue.Education Trust, 1724 K Street, N.W., Suite 200, Washington,DC 20006. Tel: 202-293-1217; Fax: 202-293-2605; Fax:202-293-0665; Web site: http://www.edtrust.org.Collected Works - Serials (022)Thinking K-16; v3 n1 Spr 1999MF01/PCO2 Plus Postage.Elementary Secondary Education; English Instruction; HigherEducation; Knowledge Base for Teaching; Language Arts;*Licensing Examinations (Professions); Literacy Education;Mathematics Instruction; Multiple Choice Tests; PreserviceTeacher Education; Science Instruction; Scores; StandardizedTests; State Licensing Boards; State Standards; *TeacherCertification; Teacher Competencies; *Teacher CompetencyTesting; Test Content*Subject Content Knowledge; Teacher Knowledge

This publication examines research on the quality of teacherlicensing examinations, explaining that state examinations do not necessarilyguarantee that teachers have the breadth and depth of subject knowledge toteach all students to high standards. Researchers focused on English/languagearts, mathematics, and science, analyzing actual exams published byEducational Testing Service (ETS), which publishes the Praxis series, andNational Evaluation Systems (NES), which designs state-specific examinations.They investigated the approximate grade level of the test, how challengingthe test questions were, and whether the knowledge was relevant to teaching.Results found that the ETS series of essay examinations, which requirescandidates to demonstrate their depth of knowledge, is a good measure ofteachers' skills. However, this series is required by far fewer states thanis the lower level multiple-choice examination. The NES Massachusettsliteracy and communications skills examination asks more complex questionsthan do other examinations. Most tests examined were multiple choiceassessments dominated by high-school-level material. There was no evidence ofcontent at the baccalaureate level in any test. After presenting a summary ofthe tests, the article discusses who determines test content, reasons for aminimal approach to teacher licensing, and whether licensing examinationsmean anything. (SM)

Reproductions supplied by EDRS are the best that can be madefrom the original document.

NOT GOOD ENOUGH: A CONTENT ANALYSIS OF TEACHER LICENSINGEXAMINATIONS

THINKING K-16VOLUME 3, ISSUE 1, SPRING 1999

The Education Trust

BEST COPY AVNLABLE....

U.S. DEPARTMENT OF EDUCATIONOffice of Educational Research end Improvement

EDUCATIONAL RESOURCES INFORMATIONCENTER (ERIC)

0 This document has been reproduced asreceived from the person or organization

joriginating it.

Minor changes have been made toimprove reproduction quality.

Points of view or opinions stated in thisdocument do not necessarily representofficial OERI position or policy.

1

PERMISSION TO REPRODUCE ANDDISSEMINATE THIS MATERIAL HAS

BEEN GRANTED BY

F.4.4k.

TO THE EDUCATIONAL RESOURCESINFORMATION CENTER (ERIC)

Vol.. 3, Ilssue 1 A Publication of The Education Trust Spr1111

NOT GOOD ENOUGHA Conte llst Anallyete of

eachex 1T licenefuno, Exam Ina-none:

0VER THE PAST SIX MONTHS, WE HAVE

been asked on countless occasions whywe are spending so much time on teacher

quality issues these days. People seemsurprised that an organization whose missionsquarely focused on closing the achievementgap separating poor and minority students fromother young Americans would be pushing sohard for higher standards for teachers. "Sethigher standards for teachers," they say, "andminority kids will suffer because they will havefewer teachers who look like them."

Let us be clearfrom the beginningthat we rejectcategorically theassertion that higherstandards inevitablymean fewer minorityteachers. Underneaththat argument is athinly veiledsuggestion that peopleof color are somehowunable to meet highstandards. Yet all ofour experiencesuggests just the

InsideThinking K-16

Not Good Enough

How TeacherLicensing TestsFall Short

Acting on thisInformation

State Tables '2XCD

1999

SCOPE OF INTEREST NOTICE

The ERIC Facility has assignedthis documer--iTOr processingto: (In our judgment, this documentis also of interest to theClearinghouses noted to the rightIndexing should reflect theirspecial points of view. s p

opposite: minority and low-income students,including education students, can meet highstandards if they are taught to high standards.The point is not to set standards below wherethey should be out of some misguidedsympathyor equally misguided belief thatwhat minority kids need most is teachers whosimply look like them. Rather, the point is toraise the quality and intensity of the educationthey receive.

IiIiliIiliIilili!iltIiliIiIWhy We Caure

To those who arguefor reasons of diversityor because of fears about supplythat thestandards should be kept where they are, wemake a simple suggestion.0 First, go spend time sitting in the back ofclassrooms, especially classrooms in high-poverty schools. Or join our staff as they workwith teachers in those classrooms. You'll seesome stunningly good teachers, but you willalso see teachers who quite obviously cannotget their students to state or local standardsbecause they, themselves, don't meet them.0 Then, once you have a feel for the problem,take a look at the growing body of research onteacher quality and student achievement, much

3continued next page

T in ing M415

of it summarized in the Education Trust's GoodTeaching Matters or What Matters Most from the

National Commission on Teaching and America'sFuture. Both of these reports underscore a simple fact:

teacher quality is the single most important factor in

student achievement.

.° Look, too, at who gets our weakest teachers: thevery students who most need our best.

In the end, if you spend as much

Teachers oftentell us that theyare finsultecl bythe low level ofl.hcensfirat exams.

time in high-poverty schools as we do,

you can't not care about teacherqualitY. Well-educated and well-supported teachers can help all children

to soar to heights literallyunimaginable to their poorly educated

and poorly supported peers.Unfortunately', existing

mechanisms are not even close to

adequate for aSsuring teacher quality.

Seven states have no licensing

examinations fin- teachers. The remaining 44 (we

. include D.C.) require examinations, but the

combination of too-low content and too-low passingscores renders these systems effective in excluding

only the weakest of the weak.This does not mean that all teachers are poorly

prepared. On the contrary, both our own experiencesand recent research prove that many teachers are-wonderfully educated. Such teachers often tell us that

they are insulted by the low level of content exams.But the insults to children are eiien greater. Many

are being shortchanged daily by poorly prepared

teachers because we have failed to set high enoughstandards for entry into the field of teaching.

Every American should be deeply concerned

about the information in this report. But low-incomefamilies, as well as those of color, should be mostworried of all, for the simple reason that the schools

that serve them are the most likely to hire froth the

bottom of the pack.That we dramatically ratcheted up our standards

for students without inSisting on commensurateincreases in standards for teachers is a chilling

indictment of all of us: K-12 leaders, policymakers,higher educators and advocates. As they say, however,

it is never too late. There are some things that states,districts and colleges can do iffimediately to reduce

the problem. Other things will take more time.We must begin nowand we must focus on:

o kigorous preparation for intending teachers;o Higher standards for entry into the field of

teachingincluding tough academic examinatiOns;

ando Ongoing support for current teachers.

If we are relentless about these three things, no

excuses and nO exceptions, our children will succeed.

Kati HaycockDirector

IS CONTENT SUFFMENT?In both our analyses and our recommendations, we have concerned ourselves primarily with content

knowledge. We have done so not because we believe that deep content knowledge is sufficient or because wethink other things are uniMportantthings like content pedagogy, knowledge of how children learn, and beliefsystems, aniong.others. (Nobody who does as much work as.we do in higher education could possiblybelieve that deep subject matter knowledge always equals good teaching.) Nor, it may be important to pointout, do we believe that content knowledge is forever fixed when teachers complete their preparation.

Rather, we beli6re that the grasp of the core concepts and structure of a discipline with which one exitsfrom college is a eritical foundation for teaching: if that foundation is inadequate, no instructional wizardrycan make up for it. Moreover, though there are many voices within the teacher education reform communityfor the importance of pedagogy and the like, there are few such voices raised around content, even thoughteachers whO are furthest ahead in implementing their state's standards Often struggle More with content thananything else.

So, for the time being at least, we'll be a bit shrill. Until, that is, m6re faculty in the Arts and Sdiencesjoin in and there is; finally, a balande in the conversation and the reform effort.

The EclUcatiton Trust

Thin ing

How rIFIFACIHIER 41ICENSIING. TEE

FAILS SHIM':by Ruth Mitchell, Ph.D., and Patte arth

L JEACHERS HOLD A POSITION OF PUBLIC TRUST.

The community, for its part, is responsible for_ supporting the goals of education and for

allocating sufficient funds to get the job done. But

ultimately, whether or not our children succeedacademically depends on the knowledge, skill andcommitment of their teachers.

With so much riding on thequality of teachers, the publicneeds assurance that everystudent is taught byprofessionals who know a lotabout the subjects they teach.This is especially true forstudents who lack resources athome. These studentsmuchmore than their more advantagedpeersdepend almost exclusivelyon their teachers for academiccontent knowledge. Most statesadminister teacher licensing examinations as a kind ofguarantee that teachers know-enough about theirsubjects. But do these tests really certify that teachershave the breadth and depth of subject knowledge toteach all students to high standards?

The short answer is no. Over the last year, weexamined the tests most commonly used for licensingbeginning teachers. In general, state licensingrequirements place more emphasis on prospectiveteachers' pedagogical knowledge than on their contentknowledge. Moreover, the subject area tests weexamined are too weak to guarantee that teachers havethe content they need to teach students to highstandards.

° SECONDARY EDUCATION: Whereas 44states require candidates for secondary licenses to take

The Education Trust

some kind of licensing examination, only 29 requirethem to take tests in the subject area they will teach.The content in the subject tests, with a few (under-used) exceptions, is within easy reach of many of thestudents the test-takers are expected to teachaboutthe same as in high-level high school.courses.

° ELEMENTARYEDUCATION: Seven

states have no examinationrequirements for candidates forelementary certification. Theremainder require examiriationsthat cover pedagogy andrudimentary general knowledgeand skills. In general, these testsassess verbal and mathematicalliteracy at about the tenth gradelevel.'

As Lynn Steen, a nationaladviser to our study, put it: "Why

should prospective teachers go to college if this is allthey need to know?"

The long answer to our question about theadequaby of existing licensure examinations is acomplicated, tale that has its *origins in good intentions,but in the end pits students' needs against institutionalinterests and adults' right to jobs. At its core, thesystem is designed to prevent false negativejudgments (about either candidates or the institutionsthat produce them). Btit if we were truly concernedabout students, we would be more worried about thefalse positives.

Like many practices in education, the criteria forteacher licensure were established in an era that heldmodest academic expectations for the majority,of .

young people. In the last decade, howeyer, IC-12education began a transforination: high. academic

"Why shozallaiprrospective teachem

to collie eftf tM s all theyneed to know?"

Spriln.4.' 1999

E*1_1(6

standards are now the expectation for all, not somestudents. But while we raised standards for students,we have yet to make corresponding increases instandards for teaching.

Unfortunately, raising the level of teacherlicensing examinations is no simple matter. Theprocess for defining both test content and whatconstitutes "passing" takes many factors into accountthat compete directly with the goal of certifying thatcandidates have a strong command of subject matter.Projections of teacher supply and demand, protectionfor the state and university against legal challenges byunsuccessful candidates, and the authority ofuniversities are all considered in the licensingequation.

State licensure policies also rest on assumptionsabout matters that do not enter the licensing equationbut should. Many assume, for example, that the act ofmajoring or passing a certain numberof courses in accredited universitiesin itself certifies a sufficient level ofcontent knowledge. Licensure policiesalso typically assume that whatbeginning teachers don't know nowthey will learn in time. Underlyingthe whole process is the assumptionthat teachers only need to know thecontent that is expected of theirstudents, and maybe just a little bitmore. For all these reasons, licensureexaminations don't contain as muchcontent as we believe fully qualifiedteachers need in order to educate all students to highlevels of understanding.

Another wrinkle in the process is theestablishment of passing scoresthe cut-off pointbetween passing and failing the licensing examination.Passing scores are not set by the test publisher; ratherthey are established state by state. In some states,candidates can pass subject-matter exams by correctlyanswering as few as half the test items.2 In areas ofshort supply, states may still require candidates to takethe test, but will waive the requirement for minimumperformance. In such cases, any old score will do. Farfrom a guarantee of high professional standards,certification requirements often define teaching down,even while public demands for teacher performanceare being ratcheted up.

There are no bad guys to blame for this situation.Officials who make certification policies are pushed tobalance teacher shortages, growing studentenrollments, the demand to reduce class size ANDmake sure a teacher is in every classroom. Thecolleges that prepare teachers struggle to respond towhat often seem to be conflicting state mandates andconfusing messages from school districts about whatis important. At the same time, universities want toprotect both the academic freedom of professors andthe right of students to choose how to fulfill coreacademic requirements. Up and down the line there isconcern about the potential impact on the quantity anddiversity of the teacher force if the bar is raised high.The pressures on state policymakers are to minimize.In the end, test publishers respond by giving theircustomers what they say they want: a reliable methodfor measuring the lowest possible teaching

competencies that can hold up inStates Not Requiring court.

Licensing Exams: 98-99 The institutional and logisticalIdaho issues that influence the makeup ofIowa teacher exams are real. Yet they have

North_Dakota been allowed to overshadow whatSouth Dakota should be the paramount

Utah consideration of teacher certification:Vermont Can this individual teach all students

WaatiingtQn to high, not minimum, standards?Source: NASDTEC Manual on thePreparation & Certification of We looked at one aspect of thisEducational Personnel 98-99, Table question, content knowledge, becauseG-4 we find it is the most neglected in

teacher education reform. In addition,although it is by no means the only characteristic of agood teacher, we believe content knowledge is themost central. Without it, no manner of teaching skillcan possibly yield high student achievement.

WHAT WE LOOIKED FOR

"Because we were interested in how muchteachers know about their subjects, we looked only atexaminations of subject matter and generalknowledge; we did not examine tests of pedagogicalskills and knowledge. The're are too many contentexaminations for the scope of this study. We thereforelimited our study to English/language arts,mathematics and science. Within these subjects, in

4 6 The Educatkm Trust

CONEDUCTIING 7171-11E 0711)1IDY

Choosing particular tests to investigate was relatively straightforward once we limited the subjectareas. There are only two major providers of teacher examinations: Educational Testing Service (ETS)which publishes the Praxis series and National Evaluation Systems (NES) which designs state-specificexaminations. A larger number of states require Praxis, but a larger number of students take NESexaminations, since NES contracts with the big-population states such as Texas and New York. ETSand NES exams are taken by the vast majority of prospective teachers. We therefore based ouranalysis on tests published by these two providers.

As much as possible we analyzed actual exams as opposed to their widely available sample forms.ETS granted us controlled access to examine their Praxis series examinations on two occasions.Because NES contracts with individual states, access to their tests is in the hands of the appropriatestate agency. Our conclusions are based on complete examinations from a single NES state and studyguides from six others.

The content analysis of these examinations was conducted initially by a team comprised ofEducation Trust staff, represented by the authors, and outside consultants. In this method, the teamworked through the tests as if they were teacher candidates, noting relevant information about eachitem. The methodology and preliminary judgments were validated by a distinguished national reviewpanel who went through the analytical model in an abbreviated form. The national panel has endorsedthe judgments reported here.

NAT113NAL RIEVHIEW aDANEL

ean RuFrEB, past president of the National Council of Teachers of Mathematics (NCTM), now SeniorProgram Officer at the National Academy of Sciences (NAS) and associate researcher at the Universityof Wisconsin, Madison;

ecoega McF, senior lecturer emeritus of chemistry at the University of California-Irvine;

Dan Janss, Dean of the College of Liberal Arts, Towson University, Maryland;

Lynn APOTUP Mean, past president of the Mathematical Association of America (MAA) and professor atSt. Olaf College, Northfield, Minnesota;

Eugcnts Scar, executive director of the National Center for Science Education; and

GOOFZ3 PuHman., associate professor of English, Georgia State University

OUTaIDE COHE-7,311,17,6=0

EMIE And3man, science education consultant;

Larradart1 Fndcn, Program Officer, Center for Science, Mathematics, and Engineering Education at theNational Research Council; and

Amy L:',11G-taa StempaU, high school consultant with the Edison Project.

Sprzin61. 11.999 7 5

WHIAT IfS OlIMPLE OIR COMPLEX?

Test items were classified as either "simple," "moderate," or "complex." These categories refer to the level ofsophistication demanded by the test question. Here are some examples (answers appear in boldface type).

SUMP LE

A "simple" problem requires the simple recallor recognition of factual material. In math andscience, the problem can be solved in one step orby applying a simple procedure. Simple items canbe answered successfully with only a superficiallevel of understanding the concept.

An example of a "simple" question:

Which of the sales commissions shownbelow is the greatest?6

a) 1% of $1,000b) 10% of $200c) 12.5% of $100d) 15% of $100e) 25% of $40

This problem requires the application of onesimple arithmetic procedure to answersuccessfully.

COM PLEX

A "complex" item is a multi-step problem thatalso requires the development of a strategydrawing on more than one domain. Open-endeditems, such as essays and mathematical proofs,tend towards more complexity, but we occasionallyfound complex multiple-choice items. Complexitems require a deeper understanding of conceptsthan either simple or moderate questions.

Example of a "complex" problem:

Use the graph below to answer the question thatfollows:7

The graph represents a system of linearequations. For what values of m will the solution tothe system be in the first quadrant?

a) m < 1/2b) m < 5c) m > 1/2d) m > 5

Gail Burrill wrote that this Algebra 1 question iscomplex in that it requires students "to useunderstanding of slope, graphs and solutions tofind the answer and gets at more than justapplying procedure."

MODERATE

"Moderate," not surprisingly, is between simpleand complex: the item requires more than onestep, but not necessarily a strategy drawing onother domains.

An example of a "moderate" question:

Directions for Questions 5-6: the group ofquestions below consists of four lettered headingsfollowed by a list of phrases or sentences. Foreach sentence, select the one heading that ismost closely related to it. One heading may beused once, more than once, or not at all.8

a) Nephronsb) Flame cellsc) Malpighian tubulesd) Skin gills

5. Function in both arachnids and insects C6. Have cilia to guide waste products to excretorypores B

This question asks for the recall of information, butit also requires the test-taker to use thatinformation to classify the items, making this amoderately complex question.

While we hoped to find complexity, "simple recall" items were the most commonly seen types of questions inthe tests we examined.

6 8 The Edincatfton TTust

T ii ing M-E,5

order to be as fair as possible, we devoted most of ourattention to the highest level tests currently used.

The study was guided by the following questions:O What is the approximate grade level of this test?

We wanted to gauge when the content covered on thetest is normally taught and learned. Our, grade level

designation is a judgment of the test as a whole. It

represents the grade in school or the year in college atwhich a typical student would have learned enough toanswer most of the test questions correctly.

Our assignment of "grade level" does not takeinto account passing scores. However, it's importantto note again that teachers can become licensed insome states by correctly answering as few as 45% ofthe test items. While it's not possible to say exactlyhow such a low passing score affects the designationof grade level, it is likely to reduce the test's effectivedifficulty level substantially.

o How challenging are the test questions?We evaluated the degree of sophistication demandedby each test item. "Simple" items required only onestep and a simple procedure, for example, the simple

recall of factual information. On the high end were"complex" problems that required multi-step strategiesinvolving more than one domain, for example, a math

problem that draws on concepts from both algebra andgeometry. (See "What is Simple or Complex?", page 6)O Is this knowledge relevant to teaching?While we assumed that teachers should know a great

deal more than their students, we also wanted to see ifthe content is connected to what they will be expectedto teach. For example, linear algebra is typically notencountered until college. However, the depth of

understanding that linear algebra can develop isrelevant to teaching algebra in middle and highschool, and probably to algebraic concepts at theelementary level as well.

Our team of analysts went through each test itemby item, answering each question as if we wereteacher candidates ourselves. Items were classifiedaccording to the dimensions listed above. Theconclusions summarized in this report are based onthe initial documentation by the team and validationfrom our national panel of advisers (see "Conductingthe Study," page 5).

Sprilfra4.. 1999

WHAT WE FOUND

We found a few things to admire, a lot ofdisappointment and one huge gaping hole. One brightspot was the series of "essay" examinations publishedby ETS, which required candidates to demonstratetheir depth of knowledge. The essays tended to covermore sophisticated content, although not quite at thelevel of a B.A. On a discouraging note, the essays arerequired by far fewer states than the lower levelmultiple-choice versions.3

We were also impressed by the sample items forthe Massachusetts literacy and communications skillsexam published by NES. These questions, in thewords of Dan Jones on our advisory panel, were "of ahigher degree of complexity and expectation than anyof the otherswe looked at."States thatcontract withNES define thecontent andlevel of theassessmentsand there isconsiderablevariationamong NESexaminations.

7he rimajorrity of

we examhsedl werre

.1ormtnsated hflen-

E3chool1 molter/tali

Because we did not have access to the completeMassachusetts exam, we cannot make statementsabout its overall quality, particularly since the oneNES-published test we reviewed did not reflect thesame complexity as the Massachusetts sample items.But what we were able to see showed considerablepromise.

In contrast to these few bright spots, the majorityof tests we examined were multiple-choiceassessments dominated by high-school level material.A few, notably in science, devoted a significantproportion of questions to content learned in middleschool. Dan Jones found many of the English

. language arts questions "disappointing," saying thatthese tests offer "empty generalizations as the rightanswers."

According to our consultants and reviewers, mostof.the tests we examined could be ea§ily handled byadvanced high school students. Lynn Steen asserted

v. 7

IhfainALIN s E*r_b

that the math tests "could be passed by a B+ studentupon leaving high school." When one factors in thelow passing scores in some states, passing a licensingexam can mean nothing more than a high schooldiploma.

We found no evidence of content at thebaccalaureate level. Although a bachelor's degree initself may not certify that the content is relevant toteaching a K-12 curriculum, we did expect to seecontent demanding a level of sophistication acquiredthrough four years in college. Not one test was up tothe level of a graduating college senior.

More to the point, we did not find the content thatour panel believes is essential for teachers chargedwith getting all students to high standards. This"knowledge for teaching" involves the deep masteryof an academic subject that goes beyond, but is stillconnected to, the level of highest student achievementin K-12. It also equips teachers to answer theperennial question from students"Why are welearning this?"with reasons based on understandingof the discipline, rather than "Because you will need itin the next course." Many educators are familiar withthe term "pedagogical content knowledge" whichincludes being able to find cognitive bridges such asmetaphors, pictures, or manipulatives that enablestudents to understand concepts.4 For example, thePraxis II Mathematics: Pedagogy test asks candidatesto write on this question:

A small group of students in your seventh-grade math

class is unable to determine whether two fractions are

equivalent. Describe a strategy, using pictures ormanipulatives, that you could use to help foster the

students' conceptual understanding of equivalent

fractions. Your strategy should stress understanding

of what it means for fractions to be equivalent and the

development of the ability to determine whether

fractions are equivalent. 5

What this question does not require, andmathematics pedagogical content knowledge as suchdoes not encompass, is the understanding ofequivalence as an essential component ofmathematical thinking. A seventh-grade teachershould be able to inspire students by referring toequivalence as a technique in sophisticated proofs.This is the understanding that we are calling

OP,

"knowledge for teaching." In other words, middle-school teachers need to know how seventh-grade mathis foundational to very sophisticated mathematicalconcepts.

All teachers, including elementary teachers, needto understand not only the structure of the academicdiscipline, but how, by organizing knowledge inspecific forms, it contributes to understanding of theworld. They should know why our civilization valuesthe knowledge they are imparting to students, so thatthey can convey some of the passion for beauty andorder that their discipline embodies.

Knowledge for teaching is a gaping hole inlicensing examinations. For this reason alone, wecannot say that any of these tests satisfies our firstquestion: Do these tests certify that teachers havesufficient subject knowledge to teach all students tohigh standards?

While none of these tests adequately addressescontent, some of them were found to be better thanothers. A summary of the tests follows.

IELEMENTARY EDUCAT1ION ANDIt' ASI1C LI1TERACY

IR is difficult to evaluate the content knowledge inelementary licensing examinations. The tests thatelementary teachers most commonly take areconcerned largely with pedagogy, not subject matterknowledge, and therefore lie outside this study.However, many states require a test of basic literacyfor all prospective teachers, which by default becomesthe "content" test for elementary teachers.

Both ETS and NES provide basic literacy orgeneral knowledge tests. Overall, these tests werecharacterized by simple recognition or recall ofgeneral subject matter. A typical treatment of contentis seen in this literature question:

1 0

In the meanwhile there came along a single red ant on

the hillside of this valley, evidently full of excitement,

who either had dispatched his foe, or had not yet taken

part in the battle; probably the latter, for he had lost

none of his limbs; whose mother had charged him to

return with his shield or upon it. Or perchance he was

some Achilles, who had nourished his wrath apart,

and had now come to avenge or rescue his Patroclus.

The EducatfRom Trrust

virkthmilerna M-1_16

The [preceding] passage makes use of analogies thatoriginate in

a) Roman mythologyb) Elizabethan dramac) Greek epicd) the New Testamente) Arthurian legends9

In questions like this one, the test-taker eitherknows the answer or does not. It reveals nothing aboutthe candidate's ability to interpret, analyze orotherwise make use of this knowledge.

ETS publishes the widely used Praxis I, alsocalled the Pre-Professional Skills Test (PPST), whichis a test of basic skills or literacy. NES publishessimilar exams, including CBEST for the state ofCalifornia. These literacy exams are intended asqualifying tests for entry into teacher preparationprograms and are designed to be administered aroundthe second year of college. However, in many statesthese tests can be taken at any point before licensureand many prospective teachers take them aftercompleting their formal training. In these states, theliteracy test becomes by default a qualifyingexamination for teaching. Indeed, in some states it isthe only content test elementary teachers take.

Praxis I addresses only reading, writing andmathematics. None of these sections exceeded highschool level, and at least two-thirds of themathematics items were judged to be middle school.An analysis comparing the distribution of Praxis Imath items to the 1996 National Assessment forEducation Progress (NAEP) for mathematics (seechart below) seems to indicate that NAEP emphasizesa better balance of mathematics, even at the eighthgrade level, than does Praxis I.

Praxis I NAEPGrade 8

NAEPGrade 12

Number 37.5% 25% 20%Measurement 5% 15% 15%Geometry 15% 20% 20%Data Analysis 20% 15% 20%Algebra 12.5% 25% 25%

Sprin 1999

The PPST reading passages were on the level ofthe National Geographic, typically high schoolreadings but clearly accessible to middle and upper-elementary school students. In general, the questionsthat referred to these passages asked for either directrecall of information in the passage or for obviousinferences or interpretation.

Alice Fletcher, the Margaret Mead of her day, assisted

several American Indian nations that were threatened

with removal from their land to the Indian Territory.

She helped them in petitioning Congress for legal titles

to their farms. When no response came from

Washington, she went there herself to present their

case.

According to the statement above, Alice Fletcher

attempted to:

a) imitate the studies of Margaret Mead

b) obtain property rights for American Indians

c) protect the integrity of the Indian Territory

d) become a member of the United States Congress

e) persuade Washington to expand the Indian

Territory"

This question is simple on several levels. Thereading passage itself is straightforward withrelatively simple vocabulary and syntax. The questionasks only for a literal recognition of informationprovided in the passage. This question might easily befound on a middle-school reading exam.

The writing questions, or prompts, in the paper-and-pencil version were bland and general with nospecified audience or discernible purpose. Anexample: "Which of your possessions would be themost difficult for you to give up or lose?"" GeorgePullman of the national advisory panel described thewriting prompts this way: "The weakness tends to beacontextuality the kind of 'once off the top of yourhead to no one in particular for no reason except totest your writing' test that is so common. The betterprompts have a clear context and a set audience andpurpose."

The basic literacy exams showed littlecomplexity; rather the test items tended to requireonly simple recall or the application of a setprocedure. These tests are taken by students at any

11 9

ILInLfigating N-D6

point between two and four years into their collegecareers. Yet overall we found these tests to be far lessdifficult than either the SAT or ACT tests thatstudents should have performed on with some successin order to have been admitted to their universities atthe outset. These tests were mostly at the eighth totenth (sometimes seventh) grade level.

S1ECONDARY MATHEMATICS

Most of the content of the Praxis II and NESmathematics examinations can be found in a broadhigh school curriculum. Only a few questions wentbeyond calculus or addressed concepts typically notlearned until the first two years of college. Most of theitems on the exams differed little from what can befound on a test of high school mathematics.

A notable feature of most of the mathematics testswas that mathematical definitions and basic formulaswere provided up front (e.g., formulas for the area ofa triangle and the circumference of a circle). Eventhough we tended to view recall items as low level inour analysis, this was one instance when ourreviewers believed the ability to recall was important.Lynn Steen commented that by not asking candidatesto produce formulas on demand, it's as if the testpublishers "don't care whether candidates actuallyknow anything, but only whether they can carry outlearned procedures."

Of the Praxis II tests, we looked at three whichcontained the most advanced material. These wereMathematics: Content Knowledge (0061);Mathematics: Proofs, Models, and Problems, Part Iand Part II (0063 and 0064). The following tablesummarizes the content distribution of Mathematics:Content Knowledge, which is a 50-item multiplechoice test:

Content distribution by NAEP categories

Praxis IIMath (0061)

NAEPGrade 8

NAEPGrade 12

Number 10% 25% 20%Measurement 0% 15% 15%-Geometry 3-2P/0 2 % 20%Data Analysis 10% 15% 20%-Aliebra 46% 25% 25%

ILO

This distribution should be compared to the tableon page 9 for the Praxis I mathematics. Despite theobviously larger percentage of algebra items, ouranalysis concluded that only eight of 50 items wereclearly college level. In addition, 70% of the itemswere simple, and only 16% complex.

Although the vast majority of the items did notrequire complex problemsolving, more than half ofthe questions asked for some application of conceptsto problemsolving situations. Steen thought thesemultiple-choice tests contained "a significant numberof unusual questions that would exercise themetacognitive capabilities of candidates."

In contrast, he thought the Mathematics: Proofs,Models, and Problems (0063 and 0064) "includednothing that was not absolutely straightforward." Weconsidered these examinations together because theyconsist of only four and three problems respectively.We saw two forms (different years of the same test)for a total of 14 problems, ranging from geometry tolinear algebra. However, only four out of the 14problems were concerned with topics taught incollege, and just four were considered complex.

Mathematics items in NES tests vary from themostly routine questions in the complete state test wereviewed to the more complex and sophisticated itemfrom the Massachusetts test cited on page 6. Thecomplete secondary mathematics test we reviewed hada large number of items placed in a real-worldcontext, which at first sight looks like a good idea.But frequently the contexts are thinly veiledprocedural or even recall items. Many of the contextsare contrived, a few are silly, and at least two arewrong. Some items conceal a tiny mathematical topic,reached after wading through a heavy context.

This math test could be answered with ease by amathematics student after finishing AP calculus, oreven before, since the only calculus items are oftenincluded in the pre-calculus course.

In general, the tests assess mostly tenth toeleventh grade level content. Nothing in either thePraxis II tests or the NES tests probes the intellectualsubstance of college mathematics that should equiphigh school teachers with robust backgrounds fordealing with the myriad ideas that will emerge fromdiscussion with students.

12 The Educatffous Trcust

IffhLtinabn X°116

SECONDARY SCRENCIE

Un the Praxis II series, we had to choose among alarge variety of tests in several scientific disciplines.As in the case of mathematics, we wanted to look atthe tests with the highest academic challenge. But wealso had to choose those most used by states buyingPraxis II tests (for example, only 21 candidatesthroughout the entire country took the PhysicalScience: Content Essays [0482] test between 1995 and1998). The three most used Praxis II tests in sciencethat are also the most challenging are GeneralScience: Content Essays (0433), Biology: ContentKnowledge, Part I and Part II (0231 and 0232).

Like the other Praxis II essay examinations,General Science: Content Essays has only threequestions, which candidates answer in writing.Although three questions cannot measure the breadthof knowledge required at the secondary teaching level,the examination assesses the candidate's ability to useand analyze critical concepts in science covered inintroductory college courses in life science, physics,chemistry, and earth science. The three questionsbroke into six parts: the most challenging were thephysical science and general science questions; of thelife science questions, one required proceduralknowledge and the other only tapped recall. Three ofthe parts required complex, multi-step answers, butthe life science question that required recall also onlyasked for a single-step answer, surprising in a testrequiring written responses.

This example is typical of the science contentessays in that the challenge ranges from recall tocomplexity:

5-miffimeter Pipet

Water Level

Tubing

The potometer shown above is used to estimate

transpiration rate in plants.

Sprefirra..' 11999

A. Define the process of transpiration.

B. Identib) three variables that would affect

transpiration rate. Describe how the potometer canbe used to test these variables.

C. Discuss how and why a change in each variable is

expected to affect the transpiration rate. 12

The Biology: Content Knowledge, Part I (0231)test, with 75 multiple-choice items, tested knowledgeof the basic principles of science; molecular andcellular biology; classical genetics and evolution;diversity of plants and animals; ecology; science,technology, and society. Almost half of its itemscould be answered with simple recall of information;only seven required multi-step problemsolving, andonly 12 required the application of a concept.

The second part of the Biology: ContentKnowledge, Part II (0232) test also has 75 multiple-choice items, but they test knowledge that would beacquired in high school advanced or honors classesand some in first-year college biology classes. Thequestions covered molecular cellular biology; classicalgenetics and evolution; diversity in plants andanimals; and ecology. Compared with Part I, therewas a considerable increase in the number of itemsrequiring either moderate or complex problemsolving(57 in Part II), and in procedural or conceptualknowledge (62).

While all three of the tests examine knowledge ofthe "big ideas," the concepts defined in the NationalScience Education Standards and many stateframeworks for science, only Biology: ContentKnowledge Part II reflects the depth of knowledgethat one would expect of a beginning teacher ofbiology. Biology: Content Knowledge Part I does notcover a wide or deep enough selection of topics toadequately reflect what a high school science teacherneeds, even as a beginner.

The NES science tests, like the mathematics tests,were extremely variable. The complete test weexamined had tests in general science, biologicalscience, chemistry, and physics. A feature of NESexaminations which differentiates them from ETS'sPraxis series is that NES tests include pedagogicalquestions within the test. In the general science test,11% of the questions were concerned with thepedagogy of science; in the physics test, 15%; thebiological science test, 17%; and 20% in chemistry.

1 °

IsbnuilIdmo m-ficp

In no case did the academic knowledge tested gobeyond grade 12 level, and a large number of theitems in all four tests were at the grade 8, 9, or 10level (even some at grade 7). In general, the NEScomplete test reflected the techniques of multiple-choice tests from 30 years ago.

On a positive note, the NES-publishedMassachusetts sample test again provided an exampleof greater sophistication, asking for a written responseto an earth science problem on earthquakes. EugenieScott and George Miller wrote that this problem"requires both scientific and engineering/socialknowledge to answer all the parts ... If the objective ofthe sample is to have candidates prepare for science,technology and society (STS)-type items, then thisshould be effective." However, without access to thewhole exam, it's not possible to know if the testadequately addresses breadth as well as candidates'depth of knowledge of key concepts.

SECONDARY ENGLISH/LANGUAGE ARTS

Secondary English/Language Arts tests areconcerned with general knowledge about literature;knowledge of grammar, style, and etymology; andknowledge of resources (dictionaries, thesaurus, etc.).We looked at two Praxis II tests in English: EnglishLanguage, Literature, and Composition: ContentKnowledge (0041) and English Language, Literature,and Composition: Essays (0042).

We examined two forms of the ContentKnowledge test, published about five years apart. Thetest asked students to answer 150 items in two hours.The questions were mostly recall items to which astudent either knows the answer or doesn't. Forexample:

If we could only get one of them new-fangled carriages

that make no noise that Father O'Rourke told him

about, them with the rheumatic wheels.13

The underlined word in the lines above is an example of

a) a metaphor

b) a malapropism

c) an oxymoron

d) a synecdoche

112

The first 65 or so items are about literature. Therange is not wide: most of the items are concernedwith classic English literature, nineteenth centuryAmerican literature, and a smattering of worldliterature, the stuff of college survey courses. Theother questions (the majority) ask about language use,editing, etymology, and research resources(dictionaries, thesaurus, etc.)

The test was overwhelmingly concerned withbreadth of content knowledge. The result was asuperficial treatmentwho wrote what, or where doyou find this. Although taught in college surveycourses or in English education courses (wheretechnical grammatical content is often taught), thematerial is not conceptually too difficult for highschool students. There are no questions demanding adepth of knowledge that would enable "people toshow that they know how to do useful things withwhat they know," in George Pullman's words.

The story was surprisingly different with theEnglish Language, Literature, and Composition:Essays test (0042). The test is two hours long andrequires two essays. One provides two poems, usuallyon the same theme, and asks for a close criticalcomparison. The second essay presents an issue in theEnglish profession: one example asks the candidate towrite on the canon and discuss attempts to expand it;another asks the candidate to compare two schools ofcriticism, practical and response criticism, forexample. The scoring guides show that a high level ofargument, knowledge of the subject, and writingability is required to receive the highest score.

The essay test demands the knowledge of thefield and experience with close reading that istypically acquired in introductory courses to theEnglish major. It comes closer to knowledge forteaching than other tests, but its questions are notconstructed with the specificity that would requireknowledge at the level of a college graduate. Wetherefore put it within the scope of a major who hascompleted the junior year. Even so, it remains thehighest level achieved by any of the tests weexamined.

Only nine states require the Essays test (0042).Sadly more than 30 states certify secondary Englishteachers without testing any writing beyond thecontent-less essay in the basic skills tests.

1 4The EthilcatiOn ThAlst

Irkfirnileme lic°116

The NES single-subject tests combined pedagogicaland content material, but this time, within the samequestion. Here is an example from the ExCET(Texas) Preparation Manual:

Write an essay in which you:o Analyze how the speaker uses metaphor to reflect on

truth and ignorance, ando Propose learning activities for a specific secondary

grade level that would help students build an

understanding of the poem and explain why you would

use these activities.

[The poem is "Truth" by Gwendolyn Brooks]

In NES English/language arts secondary tests,most of the items concerned literature; grammar andstyle; and pedagogy. Most of the answers to theliterature questions would be found in college surveycourses, because they include examples from the bestknown Greek tragedies and traditional British andAmerican literature. Comprehension questionsinclude passages from American Indian and AfricanAmerican writers, but are all on the level of simplerecognition.

Some NES states require written essays, usuallyproviding a challenge on the level of a first or second-year college student. A Colorado prompt asks studentsto compare excerpts from the Rig Veda and a Navajomyth, and then discuss the functions of myth withanalysis of the themes and stylistic devices of the twopassages. While a college-level challenge, this promptdoes not draw on the specific knowledge of issues inliterary criticism to be expected of an upper divisioncollege student, as does the essay question cited fromthe Praxis II test, English Language, Literature andComposition: Essays (0042).

THE MATTER OF PASSING SCORES

Our judgments about the licensing exams arebased on a best case scenario. In estimating the testlevel, we assumed that all items had been answeredcorrectly. In the real world, of course, there arepassing scores, which establish the cut-off pointbetween pass and fail. Passing scores are set by theindividual states and can vary considerably for thesame test. For example, on the Praxis II test,Mathematics: Content Knowledge (0061) a test we

Praxis I Math Passing Scores By State 1998-99

State PassingScore

Estimated %Correct ToPass

% of TestTakersNationally WhoWould Fail toMake this Cut

Virginia 178 68-73% 44Hawaii 176 j 60-65% 36Oregon 175 60-65% 31Florida I 175 11 60-65% 1 31

Kansas 174 60-65% 26DC I 174 1 60-65% 1 26Delaware 174 60-65% 26Alaska I 173 1 53-58% 1 22Wisconsin 173 53-58% 22North Carolina 1 173 1 53-58% I 22Kentucky 173 53-58% 22Georgia 1 173 1 53-58% I 22West Virginia 172 53-58% 18

New Hampshire i 172 I 53-58% I 18

Maine 172 53-58% 18

South Carolina I 172 I 53-58% I 18

Arkansas 171* 53-58% 15

OklahomaJ

171 11 53-58% 1 15

Nebraska 171 53-58% 15

Texas I 171 I 53-58% ] 15

Nevada 170 45-50% 12

Montana 1 170 1 45-50% 1 12

Tennessee 169 45-50% 10

Mississippi 1 169J

45-50% 11 10

Minnesota 169 45-50% 10

* Effective July 1999See Sources Page 23

estimated to be at the advanced high-school levelrequired scores range from Oregon's high with ascaled score of 147 to Georgia's low at 124. At thehigh end, an Oregonian need only answer about 65%of the questions correctly to begin teaching high-school mathematics.14 In Georgia, a prospectivemathematics teacher can become licensed by correctlyanswering fewer than half (about 46%) of the testitems. Ironically, students would receive "Fs" forproducing such scores in the classroom, yet this is allstates require of their teachers.

Passing scores for other teacher licensing testsshow similar patterns of variation among states aswell as a tendency for dumbing down. Nowhere is thistrend more apparent than in the widely used Praxis I,

1[999 113

TrfOrfffidna

Praxis I Reading Passing Scores By State 1998-99

State PassingScore

Estimated %Correct ToPass

% of TestTakersNationally WhoWould Fail toMake this Cut

Virginia 178 71-76% 43North Carolina I 176 63-68% I 30Alaska 175 63-68% 24Delaware I 175 I 63-68% I 24Hawaii 175 63-68% 24South Carolina I 175 I 63-68% I 24Wisconsin 175 63-68% 24New Hampshire I 174 I 63-68% I 19

Oregon 174 63-68% 19

Kansas I 173 I 55-61% I 16

Kentucky 173 55-61% 16

Maine I 173 I 55-61% I 16

Minnesota 173 55-61% 16

Oklahoma I 173 I 55-61% I 16

DC 172 55-61% 12

Florida I 172 I 55-61% I 12

Georgia 172 55-61% 12

Nevada I 172 I 55-61% I 12

West Virginia 172 55-61% 12

Arkansas I 172* I 55-61% I 12

Texas 172 55-61% 12

Mississippi I 170 I 47-53% 1 8

Montana 170 47-53% 8

Nebraska I 170 I 47-53% I 8

Tennessee 169 47-53% 6*Effective July 1999See Sources Page 23

a basic literacy test that we judged to be at about thetenth to eleventh grade level. In most of the 25 statesusing this test, an elementary teacher can becomecertified by correctly answering somewhere between47 and 61% of the reading and mathematics items. Afew states have raised their Praxis I cut scores inhighly publicized efforts to raise teacher quality. InVirginia, for example, prospective teachers must nowmeet a scaled score of 178, or correctly answer about71 to 76% of the test items. However, at best, thistranslates into a mediocre performance on a high-school level exam.

Because licensing exams are reported only on apass-fail basis, there is no way of knowing ifsuccessful candidates score high or just barely make

16

the cut. Certainly, bright teacher candidates breezethrough these tests and many report feeling insultedby the tests' low level. Failing these tests sends aclear signal that the candidate is unsuitable forteaching. But passing does not tell us whetherprospective teachers know enough content to teacheffectively.

WHO DETERMIINES TESTCONTENT

The content in subject-matter licensing exams isnot, as one might expect, a deliberate, well-consideredstatement of what teachers should know in order to bequalified professionals. Licensing examinations aremeant to establish a floor. In fact, the process used todefine test specifications and validate items pushescontent levels to the basement.

Both NES and ETS are providing their customers,the states, with the product they ask for. Thesepublishers guarantee that licensing exams arepsychometrically sound. In addition, the tests haveundergone a validation process designed to assure thatthey can withstand potential legal ghallenge of the sortrecently experienced by Alabama. Such concern hasled test developers to include only content that theycan prove a beginning teacher actually uses in his orher practice. This practice reduces the likelihood thattests will contain content higher than the high schoollevel. The minimalist approach to content is justifiedby an assumption that professional growth andknowledge will occur over time. However, there is noevidence that indeed this happens among all teachers.

Tests are validated in a multi-phase process.Content specifications are provided by groups ofteachers and representatives of professionalorganizations such as the National Council ofTeachers of Mathematics, the National Council ofTeachers of English, the National Science TeachersAssociation, and so on. Because NES contractsindividually with states, these specifications areprovided by the customer.

A bank of items and/or objectives is thendeveloped by subject specialists, test experts andteachers. The tests are validated by panels comprisedmostly of novice teachers who have less than fiveyears of experience. The panels consider two

114 The Education Ta-ust

T inking ac°11S

questions about each item: "Is the knowledge in thisquestion used in my teaching?" and "What percentageof beginning teachers would answer it correctly?" Thepanel reviews are a fundamental part of the process ofvalidating test content. These findings are alsoconsidered in establishing passing scores, as areprojected supply and demand of teachers by field andimpact statements on minority candidates.

Unless a state suddenly experiences a glut ofprospective teachers (hardly likely when most areclaiming shortages) this process cannot accommodateraising the bar on licensing exams. In fact, over timethe process itself creates a downward spiral ofexpectations.

THE WRONG ANALOGY

The reason for a minimal approach to teacherlicensing is at root legal: litigation has established thatentry-level qualifications must have direct relevance tothe job. The analogy is with trade: a carpenter cannotbe required to pass an examination in calculus ifcalculus is never used in framing. Thus, testpublishers build a firewall against litigation bychecking with early-career teachers about theknowledge they claim to use on the job.

We believe the trade analogy undermines theprofessional status of teachers. Other professionalsincluding lawyers, accountants, doctors andnursesmust pass tests that are notoriously tough.Often, as these professionals advance in their fields,they must pass still other exams which specifically testtheir increasing content knowledge. Not so inteaching.

Many of those involved with licensing policywant to avoid unfairly excluding people frombecoming teachers. While conducting this study, wealso heard over and over again that the purpose oflicensing was to assure that beginning teachers woulddo no harm. But we know from research that poorlyprepared teachers do harm. And they do the mostharm to the students who have the least support to fallback on for their academic development.16

Clearly, states have an interest in preventinglengthy suits. Schools of education have an interest inshowing high success rates among their graduates.School administrators, too, have an interest in fillingvacancies.

Sprhs 11999

But all of these factors conspire to keep licensingcriteria minimal. Lost in this process is the students'interest in having teachers who have the contentknowledge needed to help them reach new and higheracademic standards.

CAN LICENSING EXAMS MEANSOMETHING?

Several states are attempting to raise licensingrequirements. A few, like Massachusetts, are makingefforts to install higher level tests. Other states,notably Maryland and Virginia, are raising passingscores. But while the attempts are laudable, raisingpassing scores on low-level examinations will produceonly modest returns in the long run. What's needed toassure a higher caliber teacher corps is a reevaluationof theassumptions andgoals uponwhich thecurrent tests arebased.

The firstassumption thatneeds to bereexamined isthat teachers'contentknowledge

'Poorly preparedteachers do harm.

And they do the moot'In_trns to the studernte

who have the leaotppoirt to fall back onfor thelr acadernfic

devekDpment

grows over time. If true, a minimal approach tolicensing is certainly less risky. Yet there is nostructure to ensure that a teacher's intellectual growthwill happen. Existing career ladders and requirementsfor continuing certification do not emphasize contentknowledge, on the whole, and they are by no meansfound in every state. States need to build suchstructures. But they also need to certify a solidacademic foundation in the beginning.

Second is the assumption that minimum contentknowledge for K-12 teachers means K-12 content, andmaybe just a little bit more. Most of the content onlicensing examinations is most typically found in high .

school curricula. On the few occasions that testsaddressed content beyond high school, it was at thelevel of the first or second year of college, never at thelevel of a bachelor's degree. Such low levels ofcontent are insufficient.

1 7115

Tri_indale.N0 Mtl6

The third assumption is about licensure and notspecifically about the exams. Many states assume thatpassing college courses assures a "college level"mastery of content. But there is ample researchshowing that this is not the case. More importantly,there is no indication that the content learned incollege courses is at all relevant to what prospectiveteachers will need to teach.

In the end, this was the most disturbing aspect tous. What we have named knowledge for teachingthe deep understanding of key concepts connected toK-12 curriculumis absent from the licensingexaminations.

The movement to draft K-12 content standardsbegan with a question: What should students knowand be able to do to be competent, literate high schoolgraduates? After long public discussions involvingeducators, subject specialists, industry and civicleaders, the answer was content that was significantlyhigher than schools were currently teaching.Moreover, it was determined that this higher-levelcontent must be mastered by each and every studentincluding those students that schools had traditionallyleft behind.

This discussion should not be over. It must extendto the next logicalquestion: What shouldteachers know and beable to do to teach theirstudents to these newstandards? When we holdsuch discussions, as somecollege faculty arebeginning to do now, weare bound to find that thecontent requirements forteachers also need to besignificantly higher. Andlicensing examinationsmust reflect this content.

Raising the level oflicensing examinations isnot without risks. Thethreat of litigation willcontinue to loom forstates and tougher testsadd to the panic ofadministrators scrambling

11.6

now to put a teacher in every classroom. Even thesimple act of raising passing scores can have animmediate impact on teacher supply.

Yet the short-term risks of shortages and ill-prepared candidates are inconveniences compared tothe long-term devastation of placing barely qualifiedteachers in charge of our students' intellectualdevelopment.

Teachers truly hold society's future in their hands.There are many gifted teachers currently practicingwho both know their subjects well and can conveythat knowledge to their students. But there is nopresent mechanism to ensure that all teachers havethese qualifications, or will attain them in time.

Everybodystudents, parents, teachersthemselves, and members of the communityholds ahigh stake in making sure teachers have theknowledge they need to teach all students to highstandards. With public support and political will,policymakers and educators can loosen thestranglehold that litigation and psychometrics have ondeveloping licensing examinations. They can makethem into instruments that signify high professionalstandards and tests that teachers will be proud to pass.

Are Teacher Candidates AdequatelyPrepared?

A Vir. haia Example

Of the 5,000 Virginia teacher candidates who took the PRAXIS I,a basic-skills test:

35% failed the writing portion35% failed the math portion20% failed the reading portion

FURTHERMORE,Nationwide, nearly half of all teacher candidates would have failedto make the Virginia cut.

Source: Data derived from the Washington Post, February 21, 1998.Notes: Passing the reading portion of the exam required that candidates answeredapproximately 71-76% of the questions correctly; passing the math portion of theexam required that candidates answered approximately 68-73% of the questionscorrectly; data not available for the writing portion of the exam.

The Educattoma Tutust

ffralidna j(c°f_6

It

ACTING ON THIS INFOIRMATHON

Some recommendations from the Education Trust

SHORT TERM ACTIONS FOR STATE, LOCAL AND EDUCATION LEADERS

All states should assess the academic knowledge of intending teachers, using the most rigorousavailable examinations.

FOR ELEMENTARY TEACHERS, assessments should measure whether the candidate has atleast the general knowledge acquired in a four-year liberal arts degree program. None of thecurrently available examinations (with the possible exception of Massachusetts') does this verywell, leaving states with two short-term options, neither good. The first is simply to raise thepassing score on whichever general knowledge exam is currently in use. The second, possibleonly in states that have rigorous, internationally-benchmarked high school exit exams (like NewYork Regents Exams), is to administer that high school exam to intending elementary schoolteachers and demand a "distinguished" or "advanced" performance level.

FOR SECONDARY TEACHERS, states should require both the essay-rich assessments (forexample, the Praxis II English Language, Literature and Composition: Essay or Mathematics:Proofs, Models and Problems, or NES' Massachusetts Science Essay Exam) and multiple-choicecontent examinations. This way, both breadth and depth of subject knowledge will be tested.

2 Minimum passing scores should be raised.

3 School districts should request from all applicants for teaching jobs their scores on relevantlicensure examinations, as well as copies of college transcripts or other evidence of contentexpertise. While all teacher hiring decisions should factor in this information, high-povertyschools, where students are especially dependent upon their teachers for content learning, shouldgive test scores and transcripts considerable weight in the hiring process.

4 University leaders should note that current state policies do not preclude them from settinghigher academic standards for graduation from their institutions than states require for licensure.Indeed, they would be well advised to consider doing what the Texas A & M System recentlydid: set higher standards for itself than did the State Licensure Board.

LONGER TERM ACTIONS FOR STATE, LOCAL AND EDUCATION LEADERS

All states should immediately initiate a process aimed at developing clear academic standards forwhat teachers need to know in the various content areas in order to teach students to the state K-12 standards. The standard-setters should start with the K-12 standards, but ask specifically whatmore a teacher needs to know both to have the deep understanding necessary to teach a conceptwell, and also the knowledge necessary to link that concept to others. At the secondary level,especially, these standards should represent the kind of knowledge that should be acquiredduring four years of intensive study at the collegiate level. This process should be led by facultyfrom the relevant disciplines, but should also include teachers and education faculty.

19§ph999 1117

'1111fortinlidin) _T,<--Tlet

2 These new standards for teachers should serve as a framework for rethinking how teachers areprepared, including what courses they are required to take. Broad general educationrequirements, which allow students to fulfill science requirements with courses like Astronomyor Human Sexuality, may not be sufficient to provide elementary teachers with the contentbackground that matches what they will be teaching. The same is true of secondary teachersand the academic major: as one group of university-based mathematicians just found, whilestudents could learn all of the content necessary to teach to that state's standards in the coursesavailable to a math major, completion of the major would by no means guarantee the rightcombination of content.

3 These standards should also guide the development and/or selection of rigorous new academicassessments for initial licensure. These should be developed, tested, and put into place as soonas possible.

4 Passing state licensure exams should not be left to chance. Rather, to ensure that institutions ofhigher education take seriously their responsibility to prepare intending teachers for theseexams, states should adopt and put into place accountability systems that hold colleges strictlyaccountable for the success of their graduates. These accountability systems should affect thearts and science departments that do most of the content preparation of future teachers, as wellas schools of education.

5 Colleges and universities should be held clearly accountable for preparing all of the students intheir teacher preparation programsincluding minority studentsto pass state licensure exams.As in the state of Texas, institutions that do not succeed with minority students should get helpimproving their success. If they fail to improve, state registration or accreditation should berevoked.

ACTIONS FOR NATIONAL DISCEPLIENARY LEADERS

Like the leaders in the field of mathematics, other disciplinary associations should take the leadin designing and carrying out a process for developing model academic standards for teachers ateach level of the education system. The process should include the relevant subject-matterassociations from both higher education and K-12. The model standards will serve as areference point for state and local academic leaders as they answer the question "What do ourteachers need to know to teach children to our standards?"

2 The disciplinary associations at both levels should also collaborate on the development ofrigorous assessments of academic content for prospective teachers. As in the examinationsdeveloped by the American Chemical Society, these might initially be used voluntarily both bycolleges (for program improvement purposes) and by individuals (to demonstrate unusualmastery). Over time, however, they might be used for initial licensure.

2 040,4) The EthacaCion Tnzst

X-f16

Notes

1 See State tables, pages 20-212 See State tables, pages 20-213 Out of the 21 states using ETS secondary English

language arts examinations, only nine have selectedthe English Language, Literature and Composition:Essays. The Mathematics: Proofs, Models andProblems exams are used in only seven of the 22states administering ETS secondary mathematicsassessments. See State tables, pps. 20-21.

4 The ability of Stephen Jay Gould in his book FullHouse (New York: Random House, 1997) to explainstatistical probabilities rests on a wide range ofmetaphors, from baseball to the sidewalk wanderingsof a drunk emerging from a bar.

5 Praxis II, Mathematics: Pedagogy (0065), 1993, p.10.This example comes from the complete tests nowreleased publicly.

6 from TAAG for Praxis I, Princeton NJ: ETS, 1998,p. 57 number 3

7 from the Massachusetts Teacher Tests I'm TestInformation booklet, Massachusetts Department ofEducation, 1998. p. 42

8 TAAG for Biology and General Science, 1998,Biology: Content Knowledge, Part 2 (0232),Princeton NJ: ETS, 1998, p. 24

9 The Praxis Series, NTE Core Battery Tests, Practiceand Review, Princeton NJ: ETS, 1992, p. 49

10 TAAG for Praxis I, Princeton NJ: ETS, 1998, p. 43,number 4

11 Interestingly, the computer version of Praxis I hadmore purposeful prompts, some of which requiredthe defense of an argument, although they stilllacked a specific audience.

12 TAAG Biology and General Science, Princeton NJ:ETS. 1998, p. 69

13 Praxis II, TAAG, English Language, Literature, andComposition: Content Knowledge (0041)

14 The Praxis II scaled scores are unweighted, makinga straight percentage of correct items a relevantindicator.

15 Richardson v. Lamar County (AL) Bd. of Educ.1989

16 see Haycock, Kati, "Good Teaching Matters,"Thinking K-16, Summer 1998, Washington DC:Education Trust.

Thinking K-16 is published by The Education Trust, Inc.1725 K Street, N.W., Suite 200Washington DC, 20006.Phone: 202/293-1217Fax: 202/293-2605Website: http://www. edtrust.org

The Education Trust was created to promote high academic achievement for all students at all levels kindergarten throughcollege. While we know that all institutions could better serve their students, our work focuses on the schools and collegesmost often left behind in efforts to improve education: those institutions serving Latino, African American, Native Americanand low-income students.The Education Trust works alongside policymakers, parents, education professionals, and community and business leaders, indistricts across the country, who are trying to transform their schools and colleges into institutions that genuinely serve allstudents.

Thinking K-16 is published with the intent to share lessons learned in these communities with policymakers as well as witheducators and members of the public concerned with the quality of education provided our neediest young people.

Co-Editors, Patte Barth, Jeanne Brennan, Amy WilkinsDesktop Editor, Karen Mora

Sp Tin

Copyright C 1999

2111999 119

Irlibuler-oPeNg F_*fiC6-)

How to Read This ChartThe chart below allows the reader to identify which states are using which content knowledge examinations for teacherlicensure in elementary education, secondary English/language arts, and secondary mathematics. Actual requirements canvary (some states,for example, require tests for out-of-state applicants only): readers should contact the appropriate stateagency with questions about requirements.

There are two major test publishers: Educational Testing Service (ETS) which publishes the Core Battery, NTE, and Praxisseries, and National Evaluation Systems (NES) which designs state-specific examinations. Tests listed on the chart include:

ETS: PRAXIS I: a three-part exam in Mathematics, Reading and WritingPRAXIS II: subject area testsCore Battery: consists of three tests: general knowledge professional knowledge and communication skills.NTE: subject area testsNES. state-specific examsOTHER: "*" denotes that licensing exams are either developed by the state or administered by the Local

Education Authority, the State Education Authority, or an Institution of Higher Education.

The chart shows, reading from left to right across rows, by state:

O The basic skills exam used for Elementary teaching.O The basic skills exam used for Secondary teaching.O The English subject area exam(s) used for Secondary teaching.O The Math subject area exam(s) used for Secondary teaching.

Please keep in mind:

O We are listing SUBJECT AREA exams only; not pedagogy exams.O Tests being phased out appear in italics. Both the iIational Teachers Exam (NTE) and the Core Battery will be

completely phased out by June 2000, and replaced with PRAXIS II. They are included in the chart because theyare in use fbr the 1998-99 year covered by the chart.

O We limited our study to English/language arts, mathematics and science. The chart includes onlyEnglish/language arts and mathematics subject area exams for secondary teaching. States may require additionalsubject area exams in other subject areas.

O Because NES contracts with individual states, access to their tests is in the hands of the appropriate state agency.We had access to complete examinations from a single NES state, which we are prohibited from naming, and study guidesfrom six others. Due to these limitations, NES exams are listed simply as NES.

The data used to build this chart are drawn from Understanding Your Praxis Scores, The Praxis Series: ProfessionalAssessments for Beginning. Teachers, 1998-99 Spring Edition, ducational Testing Service), and the NASDTEC Manual onthe Preparation and Certification of Educational Personnel, 1 98-99 (National Association of State Directors of TeacherEducation & Certification).

Content Knowied e Exams, lry State 1998-99ELEMENTARY

State All ElementaryAlabama

All SecondarySECONDARYSecondary English Secondary Math

Alaska PRAXIS I PRAXIS IArizonaArkansas PRAXIS I PRAXIS I

NES

PRAXIS ELL&C (essays)PRAXIS II: ELL&C (content)NES

PRAXIS II: Math (content)PRAXIS II: Math (proofs I & 11)NESCalifornia NES

NESPRAXIS ELL&C (essays)NES

PRAXES Math (proofs I & II)NESColorado NES

Connecticut PRAXIS I PRAXIS I

PRAXIS I

PRAXIS ELL&C (essays)PRAXIS II ELL&C (content)

PRAXIS Math (content)

Delaware PRAXIS I[DC PRAXIS I PRAXIS I

PRAXIS IPRAXIS I

PRAXIS I

PRAXIS II: ELL &C (content)

PRAXIS It ELL&C (content)PRAXIS II: ELL&C (essays)PRAXIS ELL&C (content)PRAXIS II: ELL&C (content)

PRAXIS IL Math (content)PRAXIS II: Math (proofs I)NTE: MathPRAXIS II Math (content)PRAXIS II: Math (proofs I)PRAXIS II: Math (content)

[Florida PRAXIS IGeo a PRAXIS I

Hawaii PRAXIS Ikap

20 The Educatfiora Trust

StateIllinoisIndianaowa

KansasKentucky

[1111.fimilein M4,5

ConstentlKnowiledELEMENTARY

11Al1 Elementary II All Secondary

LouisianaMaineMarylandMassachusettsMichiganMinnesotaMississippiMissouriMontanaNebraskaNevada

New HampshireNew JerseyNew MexicoNew YorkNorth Carolina

North DakotaOhiOklahoma

!Oregon

P-i WarmRhode IslandSouth CarolinaSouth DakotaTennessee

TexasUtah

ermontVuginia

ashinWest Vuginia(Wisconsin

Wyoming

e Exams, y State 11998-99SECONDARYSecondary English Secondary Math

NES NES NES NESCore Battery Core Battery NTE: E L & L NTE: Math

PRAXIS I PRAXIS IPRAXIS I PRAXIS I PRAXIS II: ELL&C (essays)

PRAXIS ELL&C (content)PRAXIS Math (Content)PRAXIS II: Math (Proofs I)

Core Battery Core Battery NTE: E L & L NTE: MathPRAXIS I1GB PRAXIS IICBCore Battery Core Battery NTE:EL&L NTE: MathNES NES NES NESNES NES NES NESPRAXIS I PRAXIS IPRAXIS I PRAXIS I NTE: E L & L NTE: Math

PRAXIS ELL&C (content) PRAXIS II: Math (content)PRAXIS I PRAXIS IPRAXIS I PRAXIS IPRAXIS I PRAXIS I PRAXIS ELL&C (essays PRAXIS II: Math (content)

PRAXIS II: Math (proofs I)PRAXIS I PRAXIS I

PRAXIS ELL&C (content) PRAXIS IL Math (content)Core Battery Core BatteryNES/CB NES/CB NES NESPRAXIS I PRAXIS I PRA)US ELL&C (essays)

PRAXIS ELL&C (content)PRAXIS II: Math (content)

Core BatteryPRAXIS I/NES

Core BatteryPRAXIS I/NES NES

NTE: MathNES

PRAXIS I/NES PRAXIS I/NES PRAXIS II: ELL&C (essays)PRAXIS IL ELL&C (content)PRAXIS ELL&C (content)

NTE: E L & L

PRAXIS IL Math (content)PRAXIS II: Math (Proofs I &PRAXIS Math (content)

N7E: Math

Core Battery Core BatteryCore Battery Core BatteryPRAXIS I PRAXES I

PRAXIS I PRAXIS I PRAXIS II ELL&C (essays)PRAXIS II: ELL&C (content)NES

PRAXIS II: Math (content)

NESPRAXIS I/NES PRAXIS I/NES

PRAXIS I PRAXIS I NTE: E L & L N7E: Math

PRAXIS IINES PRAXIS I/NES PRAXIS ELL&C (content) PRAXIS II: Math (content)PRAXIS I PRAXES I

11 *

Test 71tle Key: E L & L = "English Language & Literature (0040)"; ELL&C (essays) = "English Language, Literature & Composition: Essays(0042) "; ELL&C (content) = "English Language, Literature & Composition: Content Knowledge (0041) "; Math (Proofs V = "Mathematics:Proofs, Models & Problems, Part 1 (0063)"; Math (Proofs 11) = "Mathematics: Proofs, Models & Problems, Part H (0064) ".

Sprfun 1999 211.

Trkflidem

Content KnowIted

STATIE TABLES

e Exams: Passihn

Praxis II Eng Lang, Lit & Comp: Content (0041)State Passing

ScoreEstimated %Correct ToPass

% of TestTakersNationallywho wouldfail to makethis cut

-Connecticu 72 -79V0-- -34Florida 165

16468-71% 21

- 0HawaiiOregon 164 68-71% 20

18GeorgiaArkansas 159* 64-67% 12

Missouri 158Tennessee 157 I 64-67% I 10New Jersey 155 64-67% 8West Virginia 155 64-67% 8

t-ffiFht --Cifaiina 154 (9-12) 59-63% 7

152 (6-8) 1 59-63% 6

Pennsylvania 153 59-63% 7DC 142 I 51-55% 2

2Kentucky 138

*Effective July 1999

Praxis II Eng, Lang, Lit & Comp:Essays (0042)

State PassingScore

% of TestTakersNationallywho wouldfail to makethis cut

California 160 30Connecticut 160 30Nevada 155 21

Arkansas 150**14

14-OregonGeorgia I 135 I 4Kentucky 135 4North Carolina 135 I 4Tennessee *

4

*Test Required - Passing score not set** Effective July 1999

11, Scores by State 199 -99

Praxis II Mathematics: Content (0061)State Passing

ScoreEstimated %Correct ToPass

% of TestTakersNationallywho wouldfail to makethis cut

Oregon 147 64-66% 70Connecticut j 141 60% 60DC 1 141 60% 60Kentucky I 141 I 60% 60Missouri 137 56-58% 56Arkansas 136** 56% 51

Hawaii 136 56% I 51

Tennessee 136 I 56% I 51

North Carolina 133 52-54% 48West Virginia 133

J52-54% I 48

NewArley 130 50-52% 43Pennsylvania 1 127 I 48-50% I 38Georgia 124 46% 34Nevada

*

*

Multiple Scores** Effective

RequiredJuly 1999

Praxis II Math: Proofs, Models &Problems Part I (0063) and Part II (0064)

State PassingScore

% of TestTakersNationallywho wouldfail to makethis cut

Part ICalifornia 170 59DC 154 37Nevada f 152 1 35Arkansas 144* 26Oregon

1144 I 26

Kentucky 141 23Georgia 139 20

Part IICalifornia I 159 I 72Ore on 140 49

* Effective July 1999

2 422 The Education Txust

Content Know led

Praxis I Writing (0720)

171fil Pri)_ M-11

STATE TABLES

e Exams: Passfm

State PassingScore

% of TestTakersNationallywho wouldfail to makethis cut

1Virginia 176 56Wisconsin 174 32Alaska 174 32Texas

I 173 23Arkansas 173* 23South Carolina I 173 23North Carolina 173 23Delaware 173 1 23Tennessee 172 1 15Oklahoma I 172 1 15New Hampshire 172 15Nevada j 172

I 15Nebraska 172 15Mississippi 172 15Minnesota 172 15Kentucky 172 I 15Kansas 172 15Georgia J 172

I

171

1511West Virginia

Oregon 171 j 11

Hawaii 171 11

Florida f 171 Ir 11

DC 171 11

Montana I 170 7Maine 168 3

* Effective July 1999

40, Scorres by State199

NTE English Language &Literature (0040)

State PassingScore

% of TestTakersNationallywho wouldfail to makethis cut

Louisiana 550 20Mississippi 530 14Virginia 520 11

South Carolina 50050-0

77Indiana

Maryland 500 7Arkansas 490

NTE Mathematics (0060)State Passing

Score% of TestTakersNationallywho wouldfail to makethis cut

Virginia 580 34South Carolina 560 25Louisiana 550 20Indiana 530

5-3-o

131North Carolina

Ohio 530J

13Mississippi 520 11

Maryland 520 I 11

Kentucky 500 7Arkansas I *Florida

* Multiple scores required.

The data used to build these tables are drawn from the following documents:

Passing Score: Understanding Your Praxis Scores, The Praxis Series: Professional Assessments for Beginning Teachers,1998-99 Spring Edition, (Educational Testing Service, 1999).

Estimated % Correct to Pass: Test Analysis: Pre-Professional Skills Tests, October 28, 1989 Administration, Form3MPS1, Unpublished Report, (Educational Testing Service, 1990).

% of Test Takers Nationally Who Would Fail to Make this Cut: The Praxis Series, Professional Assessments forBeginning Teachers: 1995-96 Percentile Ranks and Summary Statistics, (Educational Testing Service, 1996).

Spirfin ]199925

-99

23

THE EDUCATION TRUST1725 K Street, N.W.Suite 200Washington, DC 20006

Th°

iftf

NonprofitU.S. Postage

PAWPermit #1241Washington

DC

duca-tions Thquiet Comferre

DG THE AAEC A0-21i(IDIM3 AND YO2

F V F

gal@ boll§(4 n't cmsg thE loszoOins kssoDwilates .Cat taT

gotoomge it mg2

NaveGlrand Hya t

flibeT __-13, 639(9Hate:, Washr1t......,otoll/a9 DOIO

26

M171117%, r

U.S. Departme t of EducationOffice of Educational Research and Improvement (OERI)

National Library of Education (NLE)Educational Resources Information Center (ERIC)

TICE

DUCTI N ASIS

This document is covered by a signed "Reproduction Release(Blanket) form (on file within the ERIC system), encompassing allor classes of documents from its source organization and, therefore,does not require a "Specific Document" Release form.

This document is Federally-funded, or carries its own permission toreproduce, or is otherwise in the public domain and, therefore, maybe reproduced by ERIC without a signed Reproduction Release form(either "Specific Document" or "Blanket").

EFF-089 (9/97)