Murnane (1985).pdf

download Murnane (1985).pdf

of 14

Transcript of Murnane (1985).pdf

  • 8/12/2019 Murnane (1985).pdf

    1/14

    Comparing Public and Private Schools: The Puzzling Role of Selectivity Bias

    Author(s): Richard J. Murnane, Stuart Newstead and Randall J. OlsenSource: Journal of Business & Economic Statistics, Vol. 3, No. 1 (Jan., 1985), pp. 23-35Published by: American Statistical AssociationStable URL: http://www.jstor.org/stable/1391686.

    Accessed: 04/04/2014 09:12

    Your use of the JSTOR archive indicates your acceptance of the Terms & Conditions of Use, available at.http://www.jstor.org/page/info/about/policies/terms.jsp

    .JSTOR is a not-for-profit service that helps scholars, researchers, and students discover, use, and build upon a wide range of

    content in a trusted digital archive. We use information technology and tools to increase productivity and facilitate new formsof scholarship. For more information about JSTOR, please contact [email protected].

    .

    American Statistical Associationis collaborating with JSTOR to digitize, preserve and extend access toJournal

    of Business &Economic Statistics.

    http://www.jstor.org

    http://www.jstor.org/action/showPublisher?publisherCode=astatahttp://www.jstor.org/stable/1391686?origin=JSTOR-pdfhttp://www.jstor.org/page/info/about/policies/terms.jsphttp://www.jstor.org/page/info/about/policies/terms.jsphttp://www.jstor.org/stable/1391686?origin=JSTOR-pdfhttp://www.jstor.org/action/showPublisher?publisherCode=astata
  • 8/12/2019 Murnane (1985).pdf

    2/14

    Journal f Business & EconomicStatistics,January1985, Vol.3, No. 1

    omparing u b l i c a n d r i v a t e Schools h ePuzzling R o l e o Selectivity i a sRichardJ. MurnaneGraduatechoolof Education, arvard niversity, ambridge,MA 2138StuartNewsteadBritish elecom,LondonEC2,UnitedKingdomRandall J. OlsenDepartmentf Economics,OhioStateUniversity,olumbus, H43120

    Recent articlesusing the same data and variantsof the same estimation echniquereportconflictingstimatesof the relativequality f educationprovided y publicandCatholic chools.Thisarticle xplains he reasonsfor the conflict. nso doing, he article larifies he assumptionsunderlyingheincreasingly opular wo-stepmethods orcontrollingelectivity ias andhighlightsthe hazards of usingthese methods when the assumptionsare not satisfied. The articlealsoillustrates n alterative method ordetecting he presenceof selectivitybias.KEYWORDS:Selectivitybias;Publicandprivate chools;HighSchool andBeyonddatabase;Distributionf residuals.

    1. INTRODUCTIONResearch on the controversialquestion of whether

    privateschoolsin the United Statesare moreeffectivethan publicschoolsin enhancingstudentachievementhas been hinderedby a varietyof conceptualproblemsand data limitations. At the center of the researchdifficulties is the problem of distinguishingstudentachievement differences due to the effectiveness ofschool programsfrom those due to the abilities ofstudents.This problem s particularly ifficultbecausethe school choices madeby American amilies,who arefaced with variedschoolingalternativesand financialconstraints,result in significantselection of studentswith particularbackgrounds nd abilitiesto particularschools.Unless the influence of studentabilityon stu-dentachievement s controlled, he estimatesof schoolprogram ffectswillbe contaminatedby what is knownin the econometric iterature s selectivitybias.In recentyearsnew techniqueshavebeen developedto deal withselectivitybias. Thesetechniquescould, inprinciple,help one develop reliable estimates of therelative effectivenessof public and private schools.Among the contributors o this new methodologyareGoldberger 1972, 1980),Gronau(1973, 1974),Mad-dala and Lee (1976), Olsen (1980, 1982), and mostimportant,Heckman(1974, 1976, 1978, 1979).These

    techniqueshave quicklycome into widespreaduse inevaluating ducationandmanpower rainingprograms(Farkas t al. 1980;Mallaret al. 1980)and in estimat-ing demandequations (McGuire1981;Willis and Ro-sen 1979) and productionfunctions (Orazem 1983).For 1981alone, the Social Science CitationIndex lists79 references o the Heckmanarticles.Until 1981lack of datapreventedapplicationof thenewtechniques o the issueof the relativeeffectivenessof publicand privateschools.In that year,however,adata setbecameavailable hatprovided nformationonthe backgroundand skill levels of large numbers ofstudentsattendingpublic and private(predominantlyCatholic)high schools. As of this writing,two sets ofpapers (Coleman, Hoffer, and Kilgore 1981b, 1982;Noell 1981, 1982)have appliedthe new techniquestothe newdata. The resultshavenot clarified he relativeschool quality issue, however.In fact, the studies re-portedconflictingestimatesof the relativeeffectivenessof publicand Catholichighschools.Since both sets ofpaperswere based on the same data and both usedvariants f thenewtechniques orcontrolling electivitybias,the conflictbetweenthe resultsposesa significantpuzzle.Theresearchdescribedn thisarticlewasundertakento solve the puzzle of the conflictingresults.As theresearch progressed, a second theme developed-

    23

    O 1985 American Statistical Association

    This content downloaded from 190.64.49.12 on Fri, 4 Apr 201409:12:39 AMAll use subject to JSTOR Terms and Conditions

    http://www.jstor.org/page/info/about/policies/terms.jsphttp://www.jstor.org/page/info/about/policies/terms.jsphttp://www.jstor.org/page/info/about/policies/terms.jsp
  • 8/12/2019 Murnane (1985).pdf

    3/14

    24 Journal f Business &EconomicStatistics,January 985namely,that the resultsfrom the new techniquescanbeextremely ensitive o a numberof assumptions,andconsequently t is important o adoptan analysisstrat-egythatpermits nvestigationof theseassumptions.To the reader nterestedonly in the substantivepuz-zle or only in selectivitybiasmethodology, he organi-zation of this articlemaybe initially rustrating ecausethe two themes are interwoven.We believe,however,that this structure s necessary or two reasons.First,understandinghe solution to the substantivepuzzlerequiresa thorough understandingof the new meth-odologyforcontrolling electivitybias and thedifferentways n which hismethodology anbeapplied.Second,the many assumptions nvolvedin applyingthe selec-tivity bias methodologyand the methods that can beused to investigate he validityof the assumptionscanbest be explained n the context of a substantiveprob-lem.

    2. THE PUZZLEIn April1981,Coleman, Hoffer,and Kilgore hence-forthCHK)completeda highlypublicizedstudyof therelativeeffectivenessof publicandprivatehighschoolsin the United States.Theirreport, Publicand PrivateSchools, used the baseline data from High Schooland Beyond (HSB), a federallyfunded longitudinalstudyof studentswho were n theirsophomoreorsenioryear in a United States high school in 1980. Of the58,728 students in the sample, 87% attendedpublicschools, 9% attended Catholic schools, and 3% at-tended other non-Catholicprivateschools. Since thenon-Catholicprivateschools in the sample formed averysmallyet exceedinglydiversegroup,attention hasfocused on differences between public schools andCatholicschools,and this articlewill addressonly thereportedpublicschool-Catholicschoolcomparisons.The most controversialaspectof CHK'sreportwasthe conclusionthat Catholicschools are more effectivethanpublicschoolsin enhancing he cognitiveskills ofstudents asmeasuredby scoreson testsof readingandmathematics).Critics attacked many aspects of thereport,but perhapsthe most common criticismcon-cerned the methodologyused to generatethe publicschool-Catholic school achievement comparisons.CHKattempted o control for differencesbetweentheattributesof public and Catholic school studentsbyincluding 17 backgroundvariables n equations pre-dictingstudentachievement.Theseequationswerees-timatedby usingordinary eastsquares. See Goldber-gerand Cain 1982 fora detaileddescriptionof CHK'soriginalmethodology.)The basic criticismof thismethodologywas that eventhe inclusionof a largenumber of familybackgroundvariablesn anequationpredicting tudentachievementdid not necessarily liminateselectivitybias(Barnowetal. 1980). This criticismraised the issues of whether

    other echniques, uchasthosedevelopedby Heckman,wereappropriateorexaminingdifferencesn the effec-tivenessof publicandCatholicschoolsandwhether heuseof suchtechniqueswouldproducedifferentresults.Later in 1981 and again in 1982, papersby Noell,based on the HSB data,reported hat the resultsweredifferentwhenpublic-Catholicschooldifferenceswereestimated n a framework hat explicitlymodeled theselectionprocess.Noell's results,basedon estimationof a Heckman-typemodel, indicated that contrary oCHK'soriginal esults, herewerealmost nostatisticallysignificantdifferencesbetween he effectiveness f pub-lic and Catholic schools in developingthe cognitiveskillsof their students.CHK respondedto the criticismsof their ordinaryleast squaresmethodologyby also bringing he Heck-mantechnique o bearon theHSB data.Theyreported,however, hat this produceda largerestimatedadvan-tageof Catholic schools overpublicschoolsthan ordi-nary eastsquares CHK 198Ib,pp. 529-530; 1982,pp.213-214).Individually,neither the CHK nor the Noell resultsareillogical,since the directionof biasin the estimatedprogrameffect producedby ordinary east squares snot known a priori (Barow et al. 1980).The conflict,however,between the results of studies that applythesamegeneralestimationstrategy o the same data baseis puzzling.3. SUMMARYOF THE TWO-STEP TECHNIQUESFOR CONTROLLINGSELECTIVITYBIAS3.1 General Framework

    This section(basedon Barnowet al. 1980)providesa brief ormaldescription f theselectivitybiasproblemin the context of public school-Catholic school com-parisons.Sections 3.2 and 3.3 describeand comparethe variants of the new methodologyfor controllingselectivitybiasadoptedby Noell andCHK.For the ith child, (i = 1, ..., n), let y, be the testscore;z;, the schooltype(1 = Catholic,0 = public);wi,the unobservedability;X\i, the exogenousvector of kbackgroundvariables, ncluding 1;X2i,the exogenousvectorof m background ariables,whereX i is a subsetof X2; and ti, the unobservedcontinuousvariablede-termining chooltype.The model thatunderliesHeck-man's method for controlling for selectivitycan beexpressedas follows:

    Yi = wi + azi + EoiWi= X,il + (liti = X2if2 + U2izi = 1, if ti - 0

    (3.1)(3.2)

    = 0, if ti < 0. (3.3)

    This content downloaded from 190.64.49.12 on Fri, 4 Apr 201409:12:39 AMAll use subject to JSTOR Terms and Conditions

    http://www.jstor.org/page/info/about/policies/terms.jsphttp://www.jstor.org/page/info/about/policies/terms.jsphttp://www.jstor.org/page/info/about/policies/terms.jsp
  • 8/12/2019 Murnane (1985).pdf

    4/14

    Murnane,Newstead,andOlsen:Comparing chools-The Role of SelectivityBiasSubstituting3.2) into (3.1) produces 3.4), which is ofcentral nterest:

    Yi= Xlil1 + azi + uli, (3.4)where uIi = Eco ,li. It is assumed that u2iis normallydistributedand E(uli IU2i) is a linear function of u2,such that

    E(u,l) = E(u2) = O,var(ui ) = a 2, var(u2i) = 1,

    COV(Uli, U2i) = paI,COV(Uli, Uij) = COV(U2i, U2j) = COV(Uli, U2j) = 0,

    if iv j.The standardizationof u2i to have unit varianceentails no lossof generality.Now define0i= -X2/i2. Barnowet al. (1980)showedthat

    E(U2i O,i,zi)= z,f(i)/(l - F(0,)) - (1 - zi)f(i)/F(0i)= hi(Oi,zi), say. (3.5)

    f(.) and F(.) representthe standard normal densityanddistribution unctions.For notationalconvenience,let hi= hi(Oi, i). It follows thatE(uli I0, zi) = palhi. (3.6)

    Equation(3.6) showsthat ordinary eastsquaresap-plied to (3.4) will produce unbiased estimates of theparametersa and ,B only if palhi = 0. Any of thefollowing hreeconditionswillsatisfy his requirement:1. Equation 3.3) predicts chool sectorchoicewith-out error u2i= 0 for all i).2. Studentsarerandomlyassigned o Catholic andpublicschools [var(u,i)? 0, var(u2i)? 0, cov(wi, ti) =0, and thereforecov(uli, u2i)= 0].3. Althoughabilityandschool choice are correlatedin the population[cov(wi,ti) 0], there is no correla-tion betweenabilityand school choiceafter condition-ing on observedX2 [cov(w,,ti X2)= cov(uli, u2i)= 0for all i].

    Typicallyresearchersustify the use of ordinary eastsquaresto estimate Equation (3.4) by assertingthateither the second or third condition is fulfilled.(Sincewi s unobserved,hese two conditionscannot bedistin-guished in practice.)If this assertionis not justified,then ordinary east squareswill produce an overesti-mate of the Catholicschool advantage f p is positive(ablestudents of a given backgroundhave a tendencyto choose Catholic schoolsover public schools)or anunderestimatef p is negative(able students are morelikelyto choosepublic schools).

    3.2 EstimationUnder he Assumption f OneStudentPopulationIf we assumethat the student bodies of public andCatholicschools are drawnfrom a singlepopulationofstudents,all of whom attend eitherpublicor Catholic

    high schools and for whom the valuesof Ai are inde-pendentof school choice, then consistent estimates ofthe parameterscan be derivedby the followingtwo-stepmethod.First, use maximum likelihood probit analysis toestimate f2 from the equationPr[zi = 1] = F(X2i,2). (3.7)

    Call this estimator f2, and employ either one of thefollowingsecond steps:(a) Replacezi in (3.4) by zi =F(X2il32),and estimate(3.8) by leastsquares,Yi = Xlifl + aZi + oi. (3.8)

    Or (b) calculate 0i = -X~i2 andh, = zif(0^)/(l - F())-(( - z,)f(ei)/F(ei),

    add the auxiliary regressor hi to (3.4), let c = pal, andestimate(3.9) by leastsquares,Yi= Xj[il + aZi + cli '+ i. (3.9)

    Noell (1982)chose the firstof the two estimationstrat-egies. (The two second steps producethe same resultsonly if the assumptionsunderlying he model arecor-rect.Thispointis oftenneglected n discussionsof thesetechniques.)Althoughthe parameter stimatesin (3.8) and (3.9)areconsistent,the standarderrorsyielded by ordinaryleast squaresare not, in general,correctbecause theerrorsoi and njare heteroscedastic unless p = 0 (Heck-man 1976).AppendixA.1 describesa simple methodfor obtainingcorrectstandarderrorson the estimatesof a, I3, and c in (3.9).3.3 EstimationUnderhe Assumption f TwoStudentPopulations

    Theestimationstrategyusedby CHK is basedon thepremisethat the structureof Equation (3.9), includingthe values of /, and c, is different for the public andCatholicschool studentpopulations. SeeMuthen andJoreskog1983for anotherdiscussionof the distinctionbetween one- and two-populationmodels.)Consistentestimatesof 1Iand c canbe derivedfor each of the twopopulationsby extendingthe methodologyof the one-populationmodel as follows:First,as in the case of theone-populationmodel,runmaximum likelihood probit on all observations andconstructhifor each student. Thus li = -f(0,)/F(tJ),for the ith public school student,and fij= f(M0)/(1F(0j)),for thejth Catholicschool student.

    25

    This content downloaded from 190.64.49.12 on Fri, 4 Apr 201409:12:39 AMAll use subject to JSTOR Terms and Conditions

    http://www.jstor.org/page/info/about/policies/terms.jsphttp://www.jstor.org/page/info/about/policies/terms.jsphttp://www.jstor.org/page/info/about/policies/terms.jsp
  • 8/12/2019 Murnane (1985).pdf

    5/14

    26 Journal f Business&EconomicStatistics,January 985Second,use leastsquares o estimate

    Yi= Xiiq + cPi + 7f, (3.10)for the publicschoolsubsample,and

    yi= XjI + c/j + ,, (3.11)for the Catholic school subsample.Correctstandarderrorscan be calculatedby a method very similar tothat usedin the one-populationcase.From the estimatesof (3.10) and (3.11), an estimateof the Catholic school advantage or a student with aparticulark vector of characteristics, *, can be calcu-latedas

    y' - yp = x**'I - x*' i,with standarderror[x*'(VP + V)x*]'/2, where VP sthe variance-covariancematrix of the estimated fpcoefficientsand V' is the variance-covariancematrixof the estimated c coefficients. n estimating he Cath-olic schooladvantage,CHKdefined x* to be the aver-age characteristics f students attending public highschools. (Becauseof softwarepeculiarities,CHK esti-matedthe two-population electivitybias modelwith amethodslightlydifferent romthe method describednthe text. The two methods, however,produceresultsthat have the same interpretationand differ only insign.)Thereare two differencesbetweenthe one- and two-population models that are importantto recognize.First,since /' and fP are not constrained o be equal,the estimate of the Catholic school advantage maydependcriticallyon x*; second,sincecPand cc are notconstrained o be equal, a findingof loss of creamfrom the publicschoolsample (cP> 0) need not implyextra ream c > 0) in the Catholicschool sample.The second differencewould be implausibleif allstudentsof high school age attended either public orCatholicschools,but someteenagers hoose non-Cath-olic privateschools and others choose not to attendschool at all. As a result, the nature and extent ofstudent selection in the public and Catholic schoolscouldbe different.As we show in Section4, the choice of a one-popu-lationor two-populationmodelplaysa rolein explain-ing the differencebetween Noell's and CHK's results.The differencesbetweenthe one- and two-populationmodels areemphasizedhere becausemany articles nthe evaluation iteraturehat discussthe applicationofthe new techniquesfor controllingselectivitybias donot clarifythe implicit assumptionsinvolved in thechoice of the one- or two-populationmodel (e.g., Bar-now et al. 1980).

    4. OUR RESEARCH STRATEGY

    by probitanalysis s computationally xpensive.Sinceour basic strategyfor unravelingthe puzzle of theconflictingresultswas to examinethe sensitivityof theresults o the manysmall differencesdistinguishinghetwo methodologies, t was importantto adopt a low-cost estimationstrategy. t is possibleto developcom-putationally nexpensivetechniquesby assumingthatinstead of being normally distributed,u2i is distrib-uted uniformlyover the interval [0, 1] for each i =1,.. . , n.4.2 Estimation Under the Assumption of OneStudent Population

    As shown in AppendixA.2 consistent estimatesoftheparameters f Equation 3.4)fortheone-populationmodel canbederivedbythefollowing wo-stepmethod.First, estimate zi, the probability that zi is 1, for eachobservation,usingthe linearprobabilitymodelZi= Pr[zi = 1] = Xi2f2. (4.1)

    Second, add the auxiliary regressor si = zi - zi toEquation(3.4) and estimate Equation (4.2) by leastsquares:yi = X'ii + az +az + v (4.2)

    As with (3.8) and (3.9), standarderrorson the esti-mates of Ai, a, and6 will, in general,be incorrect f theordinary east squaresmethod is appliedto (4.2). Ap-pendixA.1alsoshows how correct tandard rrorsmaybe derived.Thistechnique,whichwe havecalled he smethod,yieldsconsistentestimatesof thecoefficients.AppendixB showsthatthe estimatesof i1anda producedby thismethod are identicalto the estimatesprovidedby thetwo-stageeastsquaresmethod,in whichthe firststageconsists of estimation of the linearprobabilitymodel,Equation 4.1). The s methodhas the advantage,how-ever,of providinga directtest of the null hypothesisofno selectivitybias [cov(uli,u2i)= 0]. The null hypoth-esis will be rejected f the estimateof 6 is significantlydifferentfrom zero when compared to its standarderror.4.3 Estimation Under the Assumption of TwoStudent Populations

    Theonlydifferencebetweenour methodforestimat-ing the Catholic school advantageand the methoddevelopedby Heckman and used by CHK is that weestimate the selectionequationwith generalized eastsquares nsteadof probitanalysis.Thus we replacefhiin (3.10) and (3.11) with s, and estimate(4.3) for thepublic school subsampleand (4.4) for the Catholicschoolsubsampleas follows:4.1 General Framework

    One problemwith the strategiesused by CHK andNoell is that estimation of the selectionequation(3.7)

    Yi= X1if, + 65S + vI (4.3)and

    y = Xjfi' + 6 + v. (4.4)

    This content downloaded from 190.64.49.12 on Fri, 4 Apr 201409:12:39 AMAll use subject to JSTOR Terms and Conditions

    http://www.jstor.org/page/info/about/policies/terms.jsphttp://www.jstor.org/page/info/about/policies/terms.jsphttp://www.jstor.org/page/info/about/policies/terms.jsp
  • 8/12/2019 Murnane (1985).pdf

    6/14

    Mumane, Newstead, and Olsen: Comparing Schools-The Role of Selectivity Bias4.4 Comparisonf LeastSquaresand ProbitMethods

    We foundthat the predictedprobabilities f Catholicschool attendancegeneratedby leastsquaresandprobitmethods were very close-the correlation coefficientexceeded.99 for most samples.As would be expectedunder hesecircumstances,heestimatesof the Catholicschool advantagegeneratedby these alternativemeth-ods wereverysimilar.Thus ourmethodsprovidea low-cost strategyfor examiningthe sources of the puzzleposed by the conflict betweenCHK's and Noell's re-sults.The s method requiresan exclusion restrictiontoidentifythe achievementequations-(4.2) in the one-populationmodel and (4.3) and (4.4) in the two-popu-lationmodel. This is not, however,a serious disadvan-tageof the s method relative o the probit-basedmeth-ods, since applicationof the latteroften producesun-stableresultswhenidentification s madesolelythroughfunctional orm(Olsen1980).CHKreported hisinsta-bilitywhenthey appliedthe Heckmanmethodology othe HSB data with no exclusion restriction(CHK1981b,pp. 529-530). In furtherprobit-based nalyses,both CHKand Noell adopted he following dentifyingrestriction: student'sreligiousstatus(1 = Catholic,0= other) nfluenceschoiceof schooltypeandis includedin the X2vector,but does not influenceachievement,andtherefores not included n theX1vector.Wechosethe sameidentifying estriction.Inaddition o religiousstatus, CHK (1982, p. 214) assumed that two othervariables-educational expectations n the eightgradeandfamily ncome-affected school choices butdid nothave a directeffect on achievement.Wedid notexplorethe influence of these additional restrictionson theresults, n partbecausewe foundthe exclusion of thesevariables romthe achievementequationunconvincingand in part because our estimates using the singleexclusionrestriction eplicatedCHK'sreported esults.]We conducted our work with 19,213 observationsfrom the HSB sample,whichincludedall sophomoresin publicandCatholicschoolsforwhomcompletedatawereavailable.Summary tatisticsdescribinghe distri-butions of the variablesused in the empiricalworkarepresented n Table 1.

    5. WHY CHK'SAND NOELL'S RESULTS DIFFER5.1 Two Reasons

    There are many differencesbetween the specifica-tions ofCHK'sand Noell'smodels, ncluding hechoiceof backgroundvariables n the achievementand selec-tion equations X i andX2i)andthechoice of scale usedto measurethe dependentvariable.These differencesinfluence the results to some extent. Our sensitivityanalysis,however, ndicates hat the conflictstemspri-marily rom the following wo factors, istedin decreas-ing orderof importance:

    Table 1. Means and Standard Deviationsof Variables UsedVariable Pooled White Black Hispanic

    YBMATHRT 19.27 20.64 14.51 15.55(7.28) (7.17) (5.55) (6.14)YBREADRT 9.43 10.08 7.35 7.58(3.83) (3.78) (3.19) (3.40)School .11 .09 .13 .17

    Black .11Hispanic .13Parents .72 .76 .45 .72Female .54 .53 .57 .55BBSESRAW -.04 .09 -.40 -.47(.73) (.69) (.73) (.73)NEast .21 .23 .19 .15NCent .30 .35 .19 .10South .31 .26 .54 .39Cathrel .39 .37 .12 .75

    NOTE: tandard eviations regiven nparentheses; tandard eviations f 0-1 variablesare notgiven.YBMATHRTs the sum of correctanswers on two math ests (38 items);YBREADRTs the number f correctanswerson readingest (20 items); ndBBSESRAWis the compositeSES measureconstructed ythe HSBsampledesigners.Theotherninevariables re 0-1 variables hattake the value1 accordingo these definitions:chool-Catholic; lack= black but not Hispanic;Hispanic= Hispanic escent;Parents= bothparentsiveat home;Female= female;NEast= residentnNortheast;NCent= resident nNorthCentral;outh= residentnSouth;Cathrel Catholic. lack ndHispanicredefinedfollowingCHK 1981a,p. 39, footnote to Table3.1.1). The variablesBlack, .., SouthconstitutedXi. The variablesBlack, .., Cathrel onstitutedX2.As explainednthe text,the selectionequationwas estimatedon subsamplesof the observationsn each ethnicgroup.The sizes of thesubsamples N)andpercentage f students hatattendedCatholicschools ineach subsample meanof the variable chool)are as follows: orWhites,N =6,563andSchool= .08;forBlacks,N = 886 andSchool= .04;and orHispanics, = 633andSchool= .10. Thegroupsamplesizes used in estimatinghe achievement quationwere:19,213forPooled,14,464forWhites, ,206 forBlacks,and2,543forHispanics.

    1. differences n the percentageof minority stu-dentsin the two samplesand2. differentchoices about the use of a one- or atwo-populationmodel.5.2 Why he Percentageof MinorityStudentsMatters

    Noell's sample contained a smaller percentageofminoritygroupstudents hanCHK's,one reasonbeingthat his model included a largernumber of studentbackgroundvariables. Since data on student back-ground tend to be missing more often for minoritystudents han forwhitestudents, he extradatarequire-ments of Noell's model, coupled with the decision toincludein the sampleonly observationswithcompletedata, left him with a smallerpercentageof minoritystudents.A secondreasonconcerns heweightingof individualobservationsn the HSB database,which is a stratifiedrandom sample with an oversamplingof black andHispanic students. The data base includes designweights that, in principle, permit the creation of aweightedsamplethat reflects he U.S. highschool stu-dentpopulation n 1980.Noell used the designweightsto weightthe observationsn his sample.In effect thisreduced he weightgivento the oversampledblack andHispanicstudents.CHK did not use the designweightsand hence implicitly weighted every observationequally, ncluding he oversampledblackand Hispanicstudents.(CHK did use the weights in their original

    27

    This content downloaded from 190.64.49.12 on Fri, 4 Apr 201409:12:39 AMAll use subject to JSTOR Terms and Conditions

    http://www.jstor.org/page/info/about/policies/terms.jsphttp://www.jstor.org/page/info/about/policies/terms.jsphttp://www.jstor.org/page/info/about/policies/terms.jsp
  • 8/12/2019 Murnane (1985).pdf

    7/14

    28 Journal f Business &EconomicStatistics,January 985analysis,which employedleast squaresmethods with-out explicitmodelingof the school-selectionprocess.)Thus the differentweightingchoices made by Noelland CHK contributed o the difference n the minority-group representation n their samples. (We do notreport he percentagesof blackand Hispanicstudentspresentin CHK's and Noell's samplesbecauseCHKdidnot report his informationn theirpublishedwork.Thereasonforthis omission wasapparentlyhatCHKestimateda modelthatspecificallyaccounted or selec-tivity bias only in responseto criticisms of theirordi-naryleastsquaresanalysis.When the results ndicateda Catholicschooladvantage ven larger hanthatpres-ent when ordinary east squareswere used, CHK be-cameskepticalof theapplicability f thismethodology.)Clearly he percentages f black,Hispanic,andwhitestudents in a particularsample influence the resultsonly if the structureof the model is different or thesegroups-something neither Noell nor CHK reportedinvestigating.We examined this hypothesis by usingthe s method to estimate Equations(4.1) and (4.2)separatelyorwhite, black,andHispanicstudents.Bothreadingscores and mathematics scoreswere used asdependentvariablesn the achievementequation(4.2).We weightedeach observation qually(i.e., we did notuse the design weights) n estimatingthis equationtopreserve the homoscedastic properties of the errorterms.It wasnotappropriateo estimate he school selectionequation (4.1) by using the same samples, however,since Catholic school students are overrepresentednthe HSB data base. Since the choice of Catholic orpublicschool is the dependentvariable n the selectionequation,use of these sampleswould have resulted nbiasedestimates of the relationship n the populationbetween the student backgroundvariables(X2) and

    school choice. To form appropriate amplesfor esti-mationof (4.1), we used the informationon the extentof oversamplingprovided by the design weights toproducesimplerandomsamplesof the sophomoresofeachethnicgroupin publicand Catholichighschoolsin the United States.The sizes of theserandomsamplesand the percentages f students n Catholicschools arestatedin Table 1. We estimatedthe school selectionequation(4.1) by usingthese simplerandomsamples.Then we used the estimatedcoefficients to constructvalues of Si= z - zi for all sophomores of each ethnicgroupfor whom complete data were available.Theselarger ampleswerethen usedto estimate the achieve-ment equation(4.2).F test results,which are reported n the last row ofTable 2, indicate that there are significantdifferencesinthestructure f theachievement quation 4.2).Mostimportant, he estimatesof the Catholic schooladvan-tage, also presentedin Table 2, differ across ethnicgroups. For white students, there are no significantdifferencesbetweenCatholic andpublicschoolsin thereadingand mathscores. For blackand Hispanicstu-dents, however,there are large,statistically ignificantCatholicschooladvantages.The variationacrossethnic groupsin the estimatedCatholicschool advantagecontributes o an explana-tion of the conflict betweenCHK's and Noell'sresults.The smallerpercentageof minoritystudents n Noell'ssample meant that his estimatesbased on a samplepooledacrossethnicgroupsgavelessweight o the largeCatholicschooladvantages orblack andHispanicstu-dents thandid CHK'sestimates.

    Table2 alsopresents stimatesof the Catholicschooladvantage oreachethnicgroupderivedfromestimat-ing (3.4) with ordinary east squares.Comparisonofthese estimateswith those derived from (4.2), whichTable 2. Estimates of the Differences in the Reading and Math Skills of Students Attending Catholic and PublicSchools, Using a One-Population Model

    Reading MathWhite Black Hispanic Whil

    The Catholic school advan- -.28 3.31a 6.328 .6tage (a) derived rom esti- (.35) (1.05) (1.24) (.6matingEquation 4.2) withthe s methodThe extent of selectivity bias .85a -2.30a -5.05a .1(6) (.38) (1.12) (1.27) (.7R2 fromestimatingEquation .10 .07 .11 .1(4.1)withthe s methodThe Catholic school advan- .148 1.19a 1.47a .7tage (a) obtainedby-esti- (.03) (.21) (.18) (.1matingEquation 3.4) withordinaryeast squaresF (17, 19,196) statistic from 9.40atestingnullhypothesis hatthe structure of Equation(4.2)is the same for alleth-nicgroupsba Statistically significant at the .05 level.bSeparate intercepts for white students and minoritystudents were included in the pooled regressions.NOTE: Standard errors are given in parentheses. (The complete regression results are available upon request from the first author.)

    te6956)070)4r8a19)

    Black7.26a(1.72)

    -6.54a(1.83).07

    Hispanic13.58a(2.39)

    -11.368(2.44).122.67a(.32)

    1.25a(.37)

    17.558

    This content downloaded from 190.64.49.12 on Fri, 4 Apr 201409:12:39 AMAll use subject to JSTOR Terms and Conditions

    http://www.jstor.org/page/info/about/policies/terms.jsphttp://www.jstor.org/page/info/about/policies/terms.jsphttp://www.jstor.org/page/info/about/policies/terms.jsp
  • 8/12/2019 Murnane (1985).pdf

    8/14

    Murnane,Newstead,and Olsen:Comparingchools-The Roleof SelectivityBiasexplicitlyattempts to control selectivitybias, revealsthat the latterset of estimatesdiffersmuch moreacrossethnic groupsthan do the formerestimates.We willreturnlater to the question of whether either set ofestimatesaccurately eflects he relativeefficacyof pub-lic and Catholic schools in providing education tostudentsof differentethnicgroups.5.3 Why the Choice of a One- or Two-PopulationModel Matters

    Table 3 presents estimates of the Catholic schooladvantagefor each ethnic group based on the two-populationmodel. The estimateswere calculated fortwo sets of values of x*: the averagecharacteristics fstudentsattending public schools (line 1) and the av-erage characteristicsof students attending Catholicschools(line 2). (The design weightswereused in cal-culatingthe appropriatemeans but not in estimatingthe equations.)The estimates of the Catholic schooladvantageorblack studentsare included orcomplete-ness but are extremely unstable because of the lowexplanatorypowerof the predicting quations.The results in Table 3 show that for blacks andHispanics, the Catholic school advantageestimatedfrom a two-populationmodel, with x* assumingthevaluesof the characteristics f theaveragepublicschoolstudent, s larger hantheadvantage stimated n a one-populationmodel (see Table 2). The differences are

    most striking-especially for Hispanic students-whenmath scores are used as the dependent variables, as inCHK's analysis. Thus CHK's choice of the two-popu-lation model and Noell's choice of the one-populationmodel contributed to the difference in their results.It is also interesting to note that the estimates of theCatholic school advantage obtained from a two-popu-lation model are somewhat sensitive to the choice ofx*. The results would be even more sensitive if thecharacteristics of the average public school student andthe average Catholic school student of each ethnicgroup were less similar than they in fact are.6. NEW PUZZLE AND A

    PROPOSED SOLUTION6.1 The New Puzzle

    The results of estimating both the one- and two-population models for the separate ethnic groups indi-cate that the direction of selectivity bias is different forwhite and minority students. In both models, the esti-mated value of cov(u,, u2) s positive for white students,although in most cases the coefficient is not largeenough relative to its standard error to reject the nullhypothesis of no selectivity bias. The estimates ofcov(ui, u2),however, are consistently negative for blackand Hispanic students. This implies that among blackand Hispanic sophomores with the same backgroundcharacteristics, students of relatively low ability are

    Table 3. Estimates of the Differences in the Reading and MathScores of Students in Catholic and Public Schools,Using a Two-PopulationModel

    Catholic chooladvantage,based onpre-dicting chievementnpublic chools[Eq.(4.3)]and Catholic chools [Eq.(4.4)] or:Averagepublic choolstudentAverageCatholic chool student

    Extentof selectivitybiasInpublic choolsample (6P)InCatholic school sample (6C)

    R2 rompredictingchievementusingthes methodPublic chool sample [Eq. (4.3)]Catholic chool sample[Eq. (4.4)]Catholicschool advantage,based on atwo-populationmodel estimatedbyordinaryeast squareswithouta se-lectivity ias correction or:Averagepublic choolstudent

    AverageCatholic chool student*Statisticallyignificantt the .05 level.NOTE: tandardrrors regiven nparentheses.

    WhiteReading

    Black Hispanic

    -2.02(1.61)-2.24(1.58).77*(.38)3.20

    (1.94).10.06

    .68*(.12).43*(.10)

    3.56*(1.82)3.43*(1.73)-2.18(1.42)-2.43(2.05)

    .04.04

    1.38*(.41)1.20*(.23)

    White

    .14(2.94)-.79(2.88)

    .18(.72)1.72(3.54)

    .14.06

    1.57*(.22).63*(.19)

    MathBlack

    7.71*(3.26)6.78*(3.10)-6.97*(2.31)-5.81(3.61)

    .05.07

    2.45*(.72)1.34*(.40)

    7.47(5.15)7.11(4.89)-5.16*(1.23)-6.20(5.83)

    .07.07

    2.59*(.23)1.39*(.23)

    Hispanic

    20.60*(8.80)19.24*(8.80)-11.58*(2.37)-18.82

    (9.94).09.09

    3.09*(.39)2.14*(.40)

    29

    This content downloaded from 190.64.49.12 on Fri, 4 Apr 201409:12:39 AMAll use subject to JSTOR Terms and Conditions

    http://www.jstor.org/page/info/about/policies/terms.jsphttp://www.jstor.org/page/info/about/policies/terms.jsphttp://www.jstor.org/page/info/about/policies/terms.jsp
  • 8/12/2019 Murnane (1985).pdf

    9/14

    30 Journal of Business & Economic Statistics, January 1985more likely to attend Catholic schools than publicschools. The negative covariances (which are statisti-cally significant for both black and Hispanic studentsin the one-population model and for Hispanic studentsin the two-population model estimated with mathscores as the dependent variable) are somewhat coun-terintuitive.Moreover, t is not apparentwhythe direc-tion of selectivitybias for minority students shoulddifferfromthat for white students.6.2 AnAlternative est of SelectivityBiasInanattempt o solve this newpuzzle,we appliedanalternative echnique for investigatingselectivitybias(Olsen 1982).Imaginethat one knew what the popu-lation distribution of residuals from a regressionofwhite students'test scores on their backgroundchar-acteristicswould be if all whitestudentsattendedCath-olic school.Comparethis distributionwith the distri-bution of residuals of white students who do attendCatholic chools.Ifthere s a correlationbetweenabilityand choice of school type, then the Catholic schoolresidualswill not be a randomsamplefrom the under-lyingpopulation,andthesetwo distributionswillthere-fore have differentshapes.In particular,f cov(ul, u2)> 0, then the observed Catholic school residualswillunderrepresenthe left tail of the populationdistribu-tion.

    Unfortunatelyhis strategy annotusuallybe imple-menteddirectlybecausethe populationdistributionofresiduals s rarelyknown;however,an approximationis available.The achievementresidualsof white Cath-olic school studentswho have a highestimatedproba-bility of attendinga Catholic school, based on theirbackgroundcharacteristics, hould roughly representthe populationdistributionof whites'residuals.On theotherhand, heachievementresiduals fwhiteCatholicschool studentswho have a low estimatedprobabilityof attendinga Catholicschool, based on their back-ground characteristics unusually high value of u2i),shouldbeaffectedby anyselectivitybiasthatis present.If thesetwo distributionsaresufficientlydifferent, his

    Table4. X2(2) StatisticsorTestingNullHypothesesf NoSelectivity iasSector White Black Hispanic

    Public 86.956* 3.006 1.074Catholic 1.776 .198 2.180No. of students insample used forx2Publicschools 2,027 1,922 2,102Catholic schools 1,346 284 441

    *Thenullhypothesis f noselectivity ias s rejected t the .05 significanceevelNOTE: o make he size of the sampleof whitepublic choolstudentscomparableo thesizes of the blackandHispanic ublic chool student amplesusedinthe x2tests, 11,091white tudents npublic chools who hada medium robabilityf being na public choolwereexcluded rom he test sample.

    will be evidence of selectivity bias. For a more detailedexposition, see Olsen (1982).We applied this technique to the samples of white,black, and Hispanic students in public and Catholicschools, producing six tests of selectivity bias. Mathscores were used as the dependent variablein generatingthe distributions of residuals, since the differencesacross ethnic groups in the estimates of selectivity biasusing the s method were more pronounced when mathscores were used as the dependent variable than whenreading scores were used. The results of the likelihoodratio testsarereported n Table 4.Onestrikingaspectof theseresults s thatno selectiv-ity bias was found among black or Hispanic studentsamples n eitherpublicor Catholicschools.This is incontrast o the resultsobtainedfromthe s methodandsuggests hat the conclusionof significantselectionoftheless ableminoritystudents o Catholic choolsstemsfrom a specificationerror-namely, the assumptionthat Catholic religiousaffiliationdoes not influencestudentachievement. n fact,thesignificant oefficient,6, on the auxiliaryregressor,s, actually reflects theinfluenceof Catholicreligiousstatus on the achieve-ment of minoritystudents.To see this, compareEquation(6.1) belowto Equa-tions (4.1) and (4.2), whichwereestimatedwith the smethod.

    Table . TheRelationshipetweenSelectivity iasand Catholic eligious tatusWhites Blacks Hispanics

    Coefficients on Catholic religious status (r) -.02 1.26a 1.62ain Equation (6.1) estimated by ordinary (.13) (.39) (.28)least squaresCoefficient on Catholic religious status (Q) .18a .19a .14ain Equation (4.1) estimated by the linear (.01) (.03) (.02)probabilitymodel with generalized leastsquaresbExtent of selectivity bias (a) in Equation .10 -6.54a -11.36a(4.2) estimated by the s method (.70) (1.83) (2.44)The Catholic school advantage derived .78c .72 2.22afrom estimating Equation (6.1) by (.21) (.40) (.33)ordinaryleast squaresaStatisticallyignificantn a two-tailed05t test.bAsexplainednthetext,Equation4.1)was estimated n a random ampleof United tatessophomores f aparticularacelethnicroup.cThe evidenceon selectivity ias ndicateshat hetrueCatholicchooladvantages less than hispoint stimate.NOTE: tandard rrorsaregiven n parentheses.Thesamplesizes were14,464forWhites,2,206 forBlacks,and2,543 forHispanics.Theresultsreported ere used math cores as thedependent ariablenEquations4.2)and(6.1). Thecomplete egression esultsare available ponrequest rom he firstauthor.)

    This content downloaded from 190.64.49.12 on Fri, 4 Apr 201409:12:39 AMAll use subject to JSTOR Terms and Conditions

    http://www.jstor.org/page/info/about/policies/terms.jsphttp://www.jstor.org/page/info/about/policies/terms.jsphttp://www.jstor.org/page/info/about/policies/terms.jsp
  • 8/12/2019 Murnane (1985).pdf

    10/14

    Murnane,Newstead,andOlsen:Comparingchools-The Roleof SelectivityBiasyi = X,lS, + azi + rRi+ ei (6.1)

    Let R denote Catholic religiousstatus and t be thecoefficienton R in the selection equation (4.1). It iseasily shownthat 6 = -r/t. Since t is positive for allethnicgroups,6 assumesthe oppositesign from r. Asshown in Table 5, the estimatesof r are negativeforwhites and positive for blacks and Hispanics. Thissolvesthe puzzle posed in Section 6.1 concerningthedifference n the directionof selectivitybias for whiteand minoritystudents.[If the selectionequation(4.1)had been estimatedon the same sampleused in esti-mating he achievementequations 4.2) and(6.1), then6 = -r/t would hold as an identity for each ethnicgroup.Since(4.1) wasestimated,however,on a subsetof the observations or each ethnicgroupused in esti-mating (4.2) and (6.1), the equality does not holdexactly.]Thus one lesson from the Olsen test of selectivitybias is that the two-stepmethodsdevelopedby Heck-man and others-which examinewhether he meanofthe least squaresachievementresidualsshiftswith theprobability f being n a particularector-are sensitiveto specificationerror.Use of these methods led to theinference hat low ability minoritygroupstudentswereselected nto Catholicschools. The distributionsof re-siduals orminoritystudents ndicate hatthisinferenceis incorrectand that the negativevalues of b stemmedfromthe improperexclusion restriction.Taken at facevalue,the exclusionrestriction hosen0.40-

    0.35-

    0.30-

    0.25-Density at X

    0.20'

    0.15-

    0.10-

    0.05.

    0 f-0-

    I///

    /. Z61.,

    /.0 -. . oe~~~'r

    byNoellandCHKappears easonable.Thisimpressiononly points out the difficulty of properlyspecifyingmodels of human behavior and the importance offinding ways to test the validity of any assumptionmade.[Stryker1981) presented videncethatforsomebut not all ethnic groups,religiousstatusis relatedtoachievement,presumablybecauseit servesas a proxyforunobservedattributesor resources.]A secondstriking indingfromthe Olsentest is thatthe resultsindicate significantselectivitybias amongwhitestudents n publicschools.Thenatureof thisbiasis illustratedby Figure 1, which showsthe theoreticaldensitiesof the residuals orwhitestudentswho haveahighor low probability f beingin a publicschool.Thedistribution or high-probabilitytudents is skewedtotheleft,implyingeither hatthedistribution f residualsfor the underlyingpopulationof all white studentsisnonnormalor thatthe sampledistributionunderrepre-sents low-achieving students who are absent fromschool on testdays.The theoreticaldensityof residualsfor low-probability ublicschool students s skewedtothe right relativeto the high-probability istribution,indicatingunderrepresentationf high-achievingstu-dents.Thechi-squaredestreveals hatthedistributionsof the two sets of residualsare significantlydifferent,indicating electivitybiasamongwhitestudents n pub-lic schools. This finding-that high-achievingwhitestudentsare underrepresentedn public schools-im-pliesthat the populationmeanthatwouldbe observedif all whitesattendedpublicschools is underestimated

    ?.????.=

    ?...???_

    ?.?.'?.'?.

    ?._ =?,'?.

    'Q?.\ ?'??.

    I * /:?/ /? /

    ////; /* //

    ://

    /: /

    /-/

    %v.'.vv% 1--2.5 -2.0 -1.5 -1.0 -0.5 0.0 0.5 1.0 1.5 2.0 2.5XFigure 1. Theoretical Distributions of Residuals From Linear Regressions of Math Score on All Background Variables (X2):White Students inPublicSchools witha Highor LowEstimatedProbabilityfAttending PublicSchool. - - -, highestimatedprobability;--, lowprobability.

    31

    This content downloaded from 190.64.49.12 on Fri, 4 Apr 201409:12:39 AMAll use subject to JSTOR Terms and Conditions

    http://www.jstor.org/page/info/about/policies/terms.jsphttp://www.jstor.org/page/info/about/policies/terms.jsphttp://www.jstor.org/page/info/about/policies/terms.jsp
  • 8/12/2019 Murnane (1985).pdf

    11/14

    32 JournalfBusiness&Economictatistics,anuary985by the sample mean of whites who choose to attendpublicschools.Thus this findingallows us to infer thedirectionof selectivitybias,but not its size.Thereason hat thesignificant electivitybiasamongwhitestudents n publicschools did not showup whenthe s method was used is that as was the case withminoritystudents,the achievementequationwas un-identifiedand the effect of selectivitybias was con-founded with the impactof religiousstatuson studentachievement.Thus we see that an improperexclusionrestriction an leadeither to the conclusion of selectiv-ity bias when thereis in factnone or to the conclusionof no selectionwhen in fact selection s present.It is interestingto note that the result of the chi-squaredtest reveals no selectivitybias among whitestudents n Catholicschools.Thissuggestshat the highability white students, who are underrepresentednpublic schools,have not chosen Catholic schoolsbut,rather, have chosen to attend non-Catholicprivateschools.Italsoemphasizes hat theassumption mplicitin the use of the one-populationmodelto compare heeffectivenessof two programs-that loss of creamfromone programmplies extra ream n the other-shouldbe examinedcarefully.The presenceof significantselectivitybias amongwhitestudents n publicschools means that the use ofordinary eastsquareswill not produceconsistentesti-mates of the relativeefficacyof public and Catholicschools in educatingwhite students. Is there an alter-native methodologythat will produceconsistent esti-mates?The two-stepmethod basedon Heckman'sandOlsen'swork cannotbe used unless an alternativeex-clusionrestriction an bejustified,and this is doubtful.Maximum likelihood methodsare anotheralterna-tive;however, hesemethodsrequirestrongerassump-tions about the populationdistributions f the achieve-ment residuals.To illustratethis point, consider thecase in which the only regressorn the school choiceequation (3.7) is a constant.In this case the two-stepmethodscollapsebecause the auxiliaryregressor,hi, isperfectly ollinearwiththe constant n theachievementequation (3.9). Maximum likelihood methods, how-ever,will still yield estimatesof a and fA under thesecircumstances ecause dentification spossible hroughthe morecompletespecificationof the residualsdistri-bution. The resultspresented n this section indicate,however, hat the mostcommondistributional ssump-tion-that u,i and u2i are distributed bivariatenor-mal-is notjustified.

    7. LESSONSPerhaps he most significantessonconcerningcom-parisonsof schoolqualityis how difficult t is to makereliableestimates,in partbecauseof the difficultyincontrolling or studentabilityand in partbecause the

    studentswithdifferentcharacteristicss itself so varied.Withthese caveats n mind,the resultssuggest hat thebest available estimatesof the relativequality of thepublicand Catholicschools available o Hispanicandblackstudentscome from ordinary east squaresesti-mationofan achievement quation hat ncludes amilyreligiousstatusas an explanatoryvariable. Such esti-matesare presented n Table 5. These results ndicatea modest (one-thirdof a standarddeviation),statisti-cally significantCatholicschooladvantageorHispanicstudentsbut no statistically ignificantCatholicschooladvantagefor black students.This patternraises theinterestingquestion of whether Hispanic and blackstudents aredifferentlyn the samepublicand Catholicschoolsorwhether he schools availableo thesegroupsdiffersystematically.Although hepresenceof selectivitybiasamongwhitestudents n public schools createsbias in the ordinaryleastsquaresestimateof the Catholicschooladvantagefor whitestudents,we do know fromthe inferreddirec-tion of selectivitybiasforpublicschool whites thattheordinary east squaresestimatepresentedin Table 5providesan upper bound on the true advantageinimpartingmath skills.Among the lessons of this article relevant to theincreasinglyargenumberof usersof thetwo-stepmeth-ods fordealingwithselectivitybias are the following:

    1. Selectioncan work differently or differentsub-groups n the population e.g.,differentethnicgroups).When this is the case,resultscan be extremely ensitiveto the weights given to differentgroups n the sampleusedin empiricalwork.2. In comparing he effectivenessof two alternativeprograms e.g.,publicand Catholic schooleducation),usinga one-populationmodel, one makes the implicitassumptionthat nonrandom selectionof participantsto one program (e.g., cream skimming) implies thecomplementaryype of selectionof participantso theotherprogram lossof cream).In fact,this may not bea validassumptionwhen there are alternatives o thetwo programs.3. An improperexclusion restrictioncan lead toextremelymisleading nferencesabout selectivitybias.Consequently it is important to evaluate carefullywhether one reallyknows enough about the determi-nants of programchoice to model the processand toidentifya variable hat affectsprogramchoice but notprogramoutcome.4. A method basedon the distributionof residualsprovidesa usefultest of thevalidityof exclusionrestric-tions.

    ACKNOWLEDGMENTSThe research or this paperwas supportedby GrantNIE-G-79-0084 rom the NationalInstituteof Educa-

    qualityof schools,both publicandprivate,available o tion and by grantsfrom the Institution for Social and

    This content downloaded from 190.64.49.12 on Fri, 4 Apr 201409:12:39 AMAll use subject to JSTOR Terms and Conditions

    http://www.jstor.org/page/info/about/policies/terms.jsphttp://www.jstor.org/page/info/about/policies/terms.jsphttp://www.jstor.org/page/info/about/policies/terms.jsp
  • 8/12/2019 Murnane (1985).pdf

    12/14

    Murnane,Newstead,andOlsen:Comparingchools-The Roleof SelectivityBiasPolicy Studies and the Programon NonprofitOrgani-zations,YaleUniversity.We thankJohnBishop,Rich-ard Nelson, SharonOster, EdwardPauly, Jon Peck,andPaulSchultz orhelpfulcommentson earlierdrafts.

    APPENDIX A: STANDARD ERRORS FOR THEPARAMETERS OF THE SELECTIVITYBIAS MODELS

    Inthisappendix t willbe convenient n placesto usevectorsandmatrices,rather han thesubscriptedcalarsof Sections 3 and 4. To this end let y = (yl,..., y,)',and let the other variablesbe describedby appropriaten vectors, except X1 = (X1l,..., Xn)', an (n x k)matrix, and X2 = (X21,. . ., X2n)', an (n x m) matrix.A.1 Selectivity Bias and the Probit Model

    u2i is assumedto be standardnormal andE(uli/u2i)= palU2i. The arguments of this section parallel thoseof Heckman(1979) and Greene(1981).var(u,i/,i, zi) = 1 + Oihi h,

    sovar(ui/Oi, zi) = 2((1 - p2) + p2(1 + Oihi- h)).

    Combiningthis informationwith Equation(3.6), wecan rewriteEquation 3.4) in vector form:y = Xlfl + az + ch + v, (A.1)

    where E(v/0, z) = 0, E(vv'/O, z) = M, and c = pal. 0is the n vectorof zeros,and M is an (n x n) diagonalmatrixwhose(i, i)th entryisMi = a2 + c2(ihi - h2).

    Note that the errorvectorvdoes not have homoscedas-tic elements.Since f2 is unknown, we must replace h in (A. 1) with4, its estimatein Section 3.2. Equation(A.1) thereforebecomesy = Ay + n, (A.2)

    whereA =(X, z, ), an n x (k + 2) matrix;y = (fi, a, c)', a (k + 2) vector;r = c(h - i) + v.

    Performingordinary east squareson (A.2) yieldscon-sistent estimates, 7, of y.Under generalconditionson the elements of z andX2(Amemiya1973;Jennrich1969),n1/2(2 - P2) - N(0, Qf)

    as n -- oofor some positive definite (m x m) matrix t,andn'/2(/j - h) -- N(O, AX2fQXA)

    as n -- oo,conditional on z and 0, where A = dh/90 isan (n x n) diagonalmatrixsuchthatAii = ahi/aei = h2 - Oihi.

    To assignappropriatetandard rrors o theestimatesof a, /i, and c, we need to determine the asymptoticdistributionof n/1/2( - ):n/2(y - y) = n(A'A)-l'n-'/2A'[c(h - h) + v].

    Nowplim n(A'A)-' = plim n((X,, z, h)'(X1, z, h))-' = B,n--oo n--ooa positivedefinitesymmetric k + 2) x (k + 2) matrix,if the Amemiya-Jennrich onditionshold.

    Then, n'/2( - y) -> N(O, B, B) as n --- o, wherePplim (i, + 02),n---oo

    A, = n-'A'MA,and

    /2 = c2n-2(A' AX2)2(A'AX2)'.An estimatefor the variance-covariancematrixof 7is thus

    (A'A)-'[A'MA + 2(A' X2)2(XA)](A'A)-' (A.3)Here2 is the estimatedvariance-covariancematrix of/2 from the probit (limo,n2; = Q), c is the estimatedcoefficient on i from regressionequation (A.2), A =f2- Ii, Mi = A2-c2 A2,n na2 = n-' 2 + 2n-' Aii,

    i=l i=1

    and A7s the vectorof observed esiduals romregressionequation (A.2). The correlationbetweenthe achieve-mentand selectionerrors ul and u2)can be estimatedby p = c/a. M, A,, p, and a2 are consistent estimatorsofc, M, A,p, and a2.Heckman'scensoredsamplemodel canbe estimatedwithminor modifications f themethodologypresentedhere. In that model, one only observesy for thoseindividualsn the samplefor whom u2i> 0i.That is, ziis 1 if y, is observed.Supposethe first n, individualshave observations n y and the last n - n, do not. First,performmaximum likelihood probit for all n cases;then construct

    , =f()/(l -F( ,))only for the firstn, cases,and estimate(A.2) for thesen, individuals, setting XI = (X1,, ..., Xln,)', A = (X1,a), and y= (/l, c)'. Finally,construct A.3)withX2=(X21, . ., X2n,)' and n replacedby n, in the estimationof M, A, and a2.

    33

    This content downloaded from 190.64.49.12 on Fri, 4 Apr 201409:12:39 AMAll use subject to JSTOR Terms and Conditions

    http://www.jstor.org/page/info/about/policies/terms.jsphttp://www.jstor.org/page/info/about/policies/terms.jsphttp://www.jstor.org/page/info/about/policies/terms.jsp
  • 8/12/2019 Murnane (1985).pdf

    13/14

    34 JournalfBusiness&Economictatistics,anuary985A.2 Selectivity Bias Controlledby the LinearProbabilityModel

    We supposethat u2iis uniform on [0, 1] for eachi = 1,..., n, rather than being standardnormal, soE(u2i) =, and var(u2i)= i2. We also assume that

    E(u,i/U2i) = poI(12)'/2(2i - )(Olsen 1980).Then

    Pr[z I1] = Xi2B2. (A.4)HereXfiB2 = 1 - 0i,where0i= -X2if2, as defined inSection3.1. Now

    E(u2i/6i, zi) = (0i + zi)/2and

    var(u2i/Oi, i) = (0i - zi)2/12.Let 6 = pa /3. Then E(uli/i, zi) = 6(0i + zi - 1),var(ui/0i, zi) = a2 - (62/3)(1 - (,i - Zi))2 = Mii,say.Equation(3.4) thereforecan be rewritten, n vectorform,

    as n -> o, where,p= plim (,6 + t2),

    =nooA'MA,tVi= n'^A'MA,and

    '2 = 2n-2(A'X2)(A'X2).An estimateforthe variance-covariancematrixof 7is then

    (A'A)-'[A'IMA + F2(A'X2)i(X2A)](A'A)-'. (A.7)Here2 is the estimatedvariance-covariancematrixofthe coefficients from the linear probability model(lim oon2: = Q), 6 is the estimated coefficient on SfromregressionEquation A.6), M, = a2 - qi

    n na2 = n-' , VI + n-' i ^i,1=1 1=1qi = (62/3)(1 - (i - zi)2)

    = (2/3)( -(Z-(1 -z, ))2),y = Xifi + az + bs + v,where si = zi - (1 - j0) (1 is the n vector of l's); E(v/O,z) = 0 (0 is the n vector of 0's); and E(vv'/0, z) = M, adiagonal n x n) matrixwhose(i, i)th entryis Mi,.It can be seen that s, the auxiliaryregressor,has anithcomponentthat is thedifferencebetween he actualvalue of z; and the probability hat zi is 1. To avoidcollinearitybetween Xi, z, and s in (A.5), we mustidentifyat least one variable hat is in X2, but not inXI, thatis, a variable hat affectsschoolchoice but doesnot directlyaffect achievement.Use the fitted values, z, from the regression of z onX2 to construct S = z - z. Then

    y = Ay = v. (A.6)In this specification,A = (X1, z, s), an n x (k + 2)matrix; y = (fi, a, 6)', a (k + 2) vector; and v =(s - s) + v.Ifthe usualconditionsholdon X2and z (Theil 1973),

    plim n(A'A)-' = plim n[(Xlzs)'(X zs)]-l = B,n--oo n-*ooa positivedefinitesymmetric k + 2) x (k + 2) matrix,and n /2(f2 - f2) -* N(0, 2) as n -- oo,for some positivedefinite m x m) matrixQ,whichneed not be the sameas the Qof Section A.1 above.Thus

    n/2( - s) -- N(0, X2fX2)as n -- oo,and we can find the limiting distribution of

    l/2(y - y) in a manner similar to that demonstratedbefore:n/2( - y) -- N(0, BOB)

    andp = / iV.3.

    v is the vector of observedresidualsfrom regressionEquation(A.6); A, M, p, and &2are consistentesti-matorsof the trueparameter alues.The censoredsample model of Olsen can be esti-matedby running he linearprobabilitymodelon all nobservations,using least squareson (A.6) for the n,individualsfor whom y is observed,and then settings = 1 - z,,XI = (X,1,... X n)', A = (X,, s), and =(f,, 6)'. Finally, (A.7) is constructed with X2 =(X2, ..., X2n,)' and n replaced by n, in the estimationof a2, q, and M.It is a fairlystraightforwardmatterto compute thestandard rrorsgiven by the diagonalelementsof (A.3)or (A.7) once the achievementregression [Equation(A.2)or (A.6)]has been run.The matrix A'A)-' isjustthe estimated variance-covariancematrix of the coef-ficientsof this regression,multipliedby the estimatedvarianceof the regression; is the estimatedvariance-covariance matrix from the probit or least squaresregression f z on X2;and the otherestimatesareeasilyconstructed.One problem in finite samples is that there is noguaranteehat a2 will be positivein either SectionA.1or A.2. This canresult n negativediagonalelements n(A.3)or (A.7) (Greene1981).Once we conditionon X2, the ith residual romthelinearprobabilitymodel has varianceOi(1 0i). Onewayto induce homoscedastic rrors s to run the linear

    (A.5)

    This content downloaded from 190.64.49.12 on Fri, 4 Apr 201409:12:39 AMAll use subject to JSTOR Terms and Conditions

    http://www.jstor.org/page/info/about/policies/terms.jsphttp://www.jstor.org/page/info/about/policies/terms.jsphttp://www.jstor.org/page/info/about/policies/terms.jsp
  • 8/12/2019 Murnane (1985).pdf

    14/14

    Murnane,Newstead,and Olsen:Comparingchools-The Roleof SelectivityBiasprobabilitymodel using ordinaryleast squares, andthen to run weighted least squares on the sameequation,giving the ith observationa weight [zi(1 -i)]-1/2.Such a schemerequireseach predictedproba-bility,Zi, o lie between0 and 1.

    APPENDIXB: TWO-STAGELEASTSQUARESAND THE s METHODRecallthat s = z - z, wherez is the vector of fittedvalues from the regressionof z on X2. The applicationof ordinary eastsquares o the achievementequation,

    y = Xi13 + az + bs + v,yieldsa vectorof predicted cores,p,

    9 = XA3i + az + 63,where il, a, and 6arethe leastsquaresestimatesof 31,a, and b.Therefore

    y =Xfl1 + az + 6(z-z)X | 1 + a(Z + ( Z A)) A= xAl + A(z + ( ))5( z) )

    But if we use ordinary east squares o regressz onX2, then the residualvector,z - z, is orthogonal o zand X2, hence also to X1, since X2containsall of thevariables n X1. Thereforethe estimatesof f\ and afrom the s methodwillbe the sameas those derivedbyapplying wo-stageeastsquares o the model,y = XI/i + az + Ui

    andz = X232 + U2.

    Moreover, f we use two-stage eastsquares, hen thestandarderrors of the estimated coefficients will becorrect; however, application of the two-stage leastsquaresmethod does not permitan immediatetest ofselectivitybias.

    [ReceivedSeptember 983.RevisedMay 1984.]REFERENCES

    Amemiya, T. (1973), RegressionAnalysisWhen the DependentVariable sTruncatedNormal, Econometrica,1, 997-1017.Barnow,S. N., Cain,G. G., andGoldberger,A. S. (1980), Issues ntheAnalysisof SelectivityBias, n Evaluation tudies Vol.5)eds.E. W. Stromsdorfer nd G. Farkas,BeverlyHills,CA:SagePubli-cations.

    Coleman, J., Hoffer,S. N., and Kilgore,S. (1981a), PublicandPrivateSchools, reportto the National Center for EducationStatisticsby the NationalOpinionResearchCenter.(1981b), Questions nd Answers:Our Response, HarvardEducationalReview,51, 526-545.(1982),HighSchoolAchievement:ublic,Catholic ndPrivateSchoolsCompared,NewYork:BasicBooks.Farkas,G., Olsen,R. J., andStromsdorfer, . W. (1980), Reduced-

    Form and StructuralModels in the Evaluation of the YouthEntitlementDemonstration, n EvaluationStudies(Vol. 5), eds.E. W. Stromsdorfer nd G. Farkas,BeverlyHills,CA:SagePubli-cations.Goldberger,A. S. (1972), SelectionBias in EvaluatingTreatmentEffects:Some Formal Illustrations, DiscussionPaper 123-172,University f Wisconsin, nstitute orResearch n Poverty.

    (1980), AbnormalSelectionBias, DiscussionPaper8006,Universityof Wisconsin,SocialSystemsResearch nstitute.Goldberger,A. S., and Cain,G. G. (1982), The CausalAnalysisofCognitiveOutcomes n the Coleman,HofferandKilgoreReport,SociologyofEducation,55, 103-122.Greene,W. H. (1981), SampleSelectionBias as a SpecificationError:Comment, Econometrica,9, 795-798.Gronau,R. (1973), TheEffectof Children n the Housewife'sValueof Time, Journalof PoliticalEconomy,81, S168-S201.(1974), WageComparisons-A SelectivityBias, JournalofPoliticalEconomy,82, 1119-1144.Heckman,J. J. (1974), ShadowPrices,MarketWages,and LaborSupply, Econometrica, 2, 679-694.(1976), The Common Structureof StatisticalModels ofTruncation,SampleSelection,and LimitedDependentVariablesand a Simple Estimator or Such Models, Annalsof EconomicandSocialMeasurement, , 475-492.(1978), Dummy EndogenousVariablesn a SimultaneousEquationsSystem, Econometrica, 6, 931-960.(1979), SampleBiasas a SpecificationError, conometrica,47, 153-162.High School and Beyond:A NationalLongitudinalStudyfor the1980s (1981), conducted for the NationalCenter for EducationStatistics,U.S. Dept. of Education,Chicago:National OpinionResearchCenter.Jennrich,R. (1969), AsymptoticPropertiesof Nonlinear LeastSquaresEstimators, nnalsof MathematicalStatistics,40, 633-643.McGuire,T. B. (1981),FinancingPsychotherapy: osts,Effects,andPublicPolicy,Cambridge,MA:Ballinger.Maddala,G. S., and Lee,L. (1976), RecursiveModelsWithQuali-tative EndogenousVariables, Annals of Economic and SocialMeasurement, , 525-545.Mallar,C. D., Kerachsky,S. H., and Thorton, C. V. D. (1980),TheShort-TermEconomicImpactof the Job CorpsProgram.in EvaluationStudies(Vol. 5), eds. E. W. Stromsdorfer nd G.Farkas,BeverlyHills,CA:SagePublications.Muthen,B., and Joreskog,K. G. (1983), SelectivityProblems nQuasi-Experimentaltudies, EvaluationReview,7, 139-174.Noell,J. (1981), TheImpactof PrivateSchoolsWhenSelf-SelectionIs Controlled:A Critique of Coleman's 'Public and PrivateSchools', unpublishedpaper,Office of Planning, Budget, andEvaluation,U.S. Dept.of Education,Washington,DC.(1982), Publicand CatholicSchools:A Reanalysis f'PublicandPrivateSchools', SociologyofEducation,55, 123-132.Olsen,R. J. (1980), ALeastSquaresCorrection orSelectivityBias,

    Econometrica,8, 1815-1820.(1982), Distributional ests forSelectivityBiasand a MoreRobust LikelihoodEstimator, nternationalEconomicReview,23, 223-240.Orazem,P. (1983), Black-WhiteDifferences n Human CapitalInvestmentand Earnings, npublishedPh.D. dissertation,YaleUniversity,EconomicsDept.Social Science CitationIndex, Philadelphia:nstitutefor ScientificInformation.Stryker,R. (1981), Religio-EthnicEffects on Attainments n theEarlyCareer, AmericanSociologicalReview,46, 212-231.Willis,R. J., and Rosen,S. (1979), Education nd Self-Selection,JournalofPoliticalEconomy,87 (Suppl.),7-36.

    35