UC Berkeley Admissions in 2015 and 2016: An Analysis of...
Transcript of UC Berkeley Admissions in 2015 and 2016: An Analysis of...
UCBerkeleyAdmissionsin2015and2016:AnAnalysisoftheRoleofLettersofRecommendationandAugmentedReview
July13,2016
JesseRothstein,ProfessorofPublicPolicy&Economics
ExecutiveSummaryUCBerkeleymademanychangestoitsundergraduateadmissionprocessesin2016,includingtheadditionoflettersofrecommendation(LORs)formanyapplicantsandtheeliminationoftheseparate“AugmentedReview”(AR)pool.Thesechangeswereintendedinparttobetteridentifyhiddengemsintheapplicantpool–becausesomanymorestudentscouldbeaskedforLORsthanwereeverconsideredunderAR,ifLORswereanywherenearaseffectiveatidentifyingunderrepresentedstudentswhocouldsucceedatCalthenetimpactwouldbetoincreasediversity.Ontheotherhand,thechangesmighthaveraisedbarrierstoadmissionfordisadvantagedstudents,whomightnothaveaccesstoteachersorcounselorswillingandabletowritestronglettersandwhosehiddenstrengthsmightnothavebeenrecognizedwithoutAR.ViceChancellorKoshlandaskedmeinJunetoconductanindependentanalysisofBerkeleyundergraduateadmissions,focusingontheLORandARchanges.IhavedoneacademicresearchoncollegeadmissionsatUCandelsewhere,butIhaveneverbeeninvolvedwithBerkeleyadmissionsprocesses.Topreparethisreport,IconsultedextensivelywiththeadmissionsofficeandwiththeSenateCommitteeonAdmissions,Enrollment,andPreparatoryEducation(AEPE)inanefforttounderstandtheadmissionsprocess.Thisreportreflectsmyindependentanalysisandconclusions.Ireachtwomainconclusions.First,thenumberofapplicantsfromunderrepresentedgroups(low-income,first-generation,fromlow-APIhighschools,and/orunderrepresentedminorities)whowereadmittedrosein2016.Butbecausethenumberadmittedwhowerenotfromthesegroupsrosebyalargerproportion,theshareofadmittedstudentsfromunderrepresentedgroupsfellsomewhat.Second,neithertheadditionofLORsnortheremovalofARcontributedmeaningfullytothisdecline.Ifanything,askingforLORsraisedtherelativeadmissionsratesofapplicantsfromunderrepresentedgroups.IamunabletopreciselyidentifytheimpactoftheeliminationofAR–whilesomeestimatesindicatethatthisslightlyreducedadmissionsforthosewhowouldhavebeenconsideredviaARin2015,othersindicatezeroorevenapositiveeffect.Alloftheestimatesagreethattheimpactwassmallinanycase.Itisbeyondthescopeofthisreporttoidentifywhatdidcausetheshiftin2016.Mypreliminaryinvestigation,however,suggeststhatthedeclineintheshareofadmitsfromunderrepresentedgroupsisinlargepartastatisticalartifactduetotheexpandeduseofthewaitlistin2016.Therewerealsoreductionsintheadmissionschancesoftheunderrepresentedapplicantswiththestrongestnumericrecordsthatcannotbeattributedtothewaitlist.Futureinvestigationshouldfocusonunderstandingwhatinthescoringprocessharmedtheseapplicants.
2
IntroductionBerkeleymadeanumberofchangestoitsundergraduateadmissionsprocessesin2016:
• Itrequestedlettersofrecommendation(LOR)formanyapplicants.• Iteliminatedtheaugmentedreview(AR)pool.• Itshiftedfromapointsystemforscoringapplicationstoathree-category
(Yes/No/Maybe)ratingsystem.• Readersbeganscoringapplicantsonalistofholistic/non-cognitivefactors.• Everyapplicationwasreadtwice,whereinthepastmanywerereadonlyonce• Athirdread,bymembersofthefaculty,wasaddedformanyapplications.• Thewaitlistwasusedmuchmoreextensivelythaninthepast,andmanyapplicants
whoin2015wouldhavebeenadmittedorrejectedoutrightwereinsteadofferedpositionsonthewaitlistin2016.
Table1presentssimplesummariesofadmissionsoutcomesin2015and2016,intheupperpanelpoolingallin-stateandout-of-stateapplicantsandinthelowerpanelrestrictingattentiontoCaliforniaresidentsnotbeingrecruitedasathletes.OfparticularconcernisthedeclineintheshareofadmittedstudentsfromdisadvantagedbackgroundsorgroupsthatareunderrepresentedatBerkeley.InTable1andthroughoutthisreport,Iconsiderfourgroupsofsuchstudents:low-incomestudents(withfamilyincomesbelow$40,000);first-generationcollegestudents(thosewhoseparentsdidnotgraduatefromcollege);studentsfromdisadvantagedschools(withAPIindexesof5andbelow);andunder-representedminoritystudents(UREMs).Irefertothemcollectivelyas“underrepresented.”Iwasaskedtoconductananalysisofadmissionsdataforthe2015and2016cycles,focusingontheimpactoftheadditionoflettersofrecommendationandtheeliminationofaugmentedreviewontheadmissionofunderrepresentedstudents.Inalloftheanalysesbelow,IrestrictattentiontoCaliforniaresidentapplicantswhowerenotclassifiedintheadmissionsprocessasrecruitedathletes.ThelowerportionofTable1showsstatisticsfortheseapplicants.TheoverallcontextIbeginwithseveralpreliminariesnecessarytounderstandingtheimpactsoflettersofrecommendationandaugmentedreviewonadmissionsoutcomes.ThecompositionoftheapplicantpoolThefirstplacetolookforanexplanationforchangingoutcomesischangesintheapplicantpoolitself.Iftheapplicantsfromunderrepresentedgroupswereweaker,onaverage,in2016thanin2015,thiscouldaccountfortheoverallobservedchanges,evenwithoutachangeinpolicy,andwouldconfoundmyLORandARanalyses.UnderstandingthedistributionofapplicantstrengthisalsohelpfulasawayofgaugingwhichtypesofstudentswereaffectedbytheLORandARprograms.
3
Table1.Summaryofadmissionsoutcomes
Applicants
Admits
2015 2016
2015 2016
Allapplicants
Number 78,924 82,578
13,266 14,423
Admissionrate
16.8% 17.5%
Sharefromunderrepresentedgroups
Lowincome(<$40K) 24% 23%
19% 18%
Firstgeneration 17% 16%
13% 12%
LowAPI 13% 12%
12% 11%
UREM 23% 23%
18% 18%
Anyofthesefour 40% 38%
33% 31%
Californiaresidents,excludingrecruitedathletes
Number 45,570 45,626
8,570 9,610
Admissionrate
18.8% 21.1%
Sharefromunderrepresentedgroups
Lowincome(<$40K) 31% 30%
24% 22%
Firstgeneration 24% 23%
18% 15%
LowAPI 23% 22%
19% 16%
UREM 33% 34%
23% 23%
Anyofthesefour 53% 53%
42% 39%
Iconstructameasureofapplicantstrengthbyaggregatinganumberofavailablestudentcharacteristics.1My“admissionsscore”representsthepredictedprobabilitythatastudentwithagivensetofcharacteristicswouldhavebeenadmittedinthefirstround(i.e.,notoffthewaitlist,andnotthroughAugmentedReview),hadhe/sheappliedasaCaliforniaresidenttotheCollegeofLettersandSciencesin2015.2
1Thevariablesarethoseusedinthemodelusedbytheadmissionsofficetopredictapplicationreadscores,andincludemeasuresoftraditionalacademicstrength,measurescharacterizingthehighschool,(including,forexample,theextenttowhichthestudenttookadvantageoftheschool’sadvancedcourseofferings),andthreeofthefourdisadvantagemeasuresconsideredhere.Applicants’raceandethnicity,whichcannotbeconsideredinadmissions,isnotincluded.2IuseonlyL&SapplicantsbecausetheothercollegesmayweightcharacteristicsdifferentlythandoesLettersandSciences.Nevertheless,myL&S-basedscoreisnearlyperfectlycorrelatedwithascoreconstructedbasedonadmissionsintheCollegeofEngineering.
4
Twoaspectsofthisscoremustbeemphasized:First,itcapturesonlythequantitativecharacteristicsthatarecodedintheadmissionsoffice’sdatabase;readersseemoreinformation,andmayidentifyapplicantsasstrongerorweakerthanisindicatedbymyscore.Second,thecharacteristicsareweightedtobestpredictadmissionin2015.Theweightputondifferentcharacteristics–say,onhighSATscoresvs.takingalloftheAPcoursesofferedatyourhighschool–mightvaryfromyeartoyear,andindeedseemstohavechangedsomewhatin2016(asdiscussedbelow).Butevenwiththesecaveats,theadmissionsscoreneverthelesspresentsausefulsummary.Totakeoneexample,38%ofapplicantsin2016havescoresunder1%.Whileaveryfewofthesestudentsmighthavecharacteristicsnotinthedatabasethatmeritadmission,thisisquiterare;thevastmajorityofstudentsinthisgroupwouldnotbeadmittedundertheregular2015processes.Indeed,only1.8%ofthemwereadmittedin2015,and2.5%in2016.Figure1showsthedistributionofadmissionsscoresacrossCaliforniaresident,non-athleteapplicantstoallcollegesin2015.Thisisheavilyleft-skewed:Mostapplicantshaveverylowchancesofadmission,thoughthereareafewwhoaresostrongonthedimensionscapturedbymyindexthatitisrareforotherfactorstopreventthemfrombeingadmitted.
Figure1.Distributionofadmissionsscoresin2015forCaliforniaresidentapplicants
Becausetheoveralldistributionofadmissionsscoresissodominatedbyapplicantswithextremelylowchancesofadmission,Ifindithelpfultofocusonthosewhoaremorelikelytobeadmitted.Figure2showsthedistributionofadmissionsscoresforthosewhowereactuallyadmittedin2015(includingARadmitsandthoseadmittedoffthewaitlist),whileFigure3repeatsthisforthefourunderrepresentedgroupsandFigure4repeatsitforapplicantsnotfromthesegroups.
0.1
.2.3
.4Fr
actio
n
0 .2 .4 .6 .8 1Admissions score
Distribution of admissions score in 2015
5
Figure2.DistributionofadmissionsscoresforadmittedCaliforniaresidentapplicantsin2015
ItisapparentinFigure3thattheadmissionsscoredistributionisquiteskewedtotheleftforstudentsfromtheunderrepresentedgroups.Thisistrueeventhoughthepredictionmodelusedtogeneratetheadmissionsscoreincludesindicatorsforlowincome,firstgeneration,andlowAPI(butnotUREM)students.Evidently,manyofthestudentswhoareadmittedfromthesegroupsarepickedoutfromlargepoolswithsimilarobservablecredentialswhoarenotadmitted.Thisismuchlesstrueforstudentsnotfromthesegroups,forwhomthedistributionisshowninFigure4:Here,admittedstudentsaremuchmorelikelytohaveadmissionsscoresabove0.6.
Figure3.AdmissionsscoredistributionsforadmittedCaliforniaresidentapplicantsfromfourunderrepresentedgroupsin2015
0.0
1.0
2.0
3.0
4Fr
actio
n
0 .2 .4 .6 .8 1Admissions score
Distribution of admissions score in 2015For those actually admitted
0.0
2.0
4.0
6.0
8Fr
actio
n
0 .2 .4 .6 .8 1Admissions score
Low income
0.0
2.0
4.0
6.0
8Fr
actio
n
0 .2 .4 .6 .8 1Admissions score
1st Generation
0.0
2.0
4.0
6.0
8Fr
actio
n
0 .2 .4 .6 .8 1Admissions score
Low API
0.0
2.0
4.0
6.0
8Fr
actio
n
0 .2 .4 .6 .8 1Admissions score
UREM
Distribution of admissions scores in 2015For those actually admitted
6
Figure4.AdmissionscoredistributionforCaliforniaresidentsnotfromunderrepresentedgroups,2015
Table2showssummariesofthedistributionofadmissionsscoresforCaliforniaresidentapplicantsandadmittedstudentsin2015and2016,bothforallapplicantsandforapplicantsfromtheunderrepresentedgroups.Itindicatesthatthedistributionofadmissionsscoreschangedsomewhatbetweenyears,withmorestudentswithverylowandveryhighadmissionsscoresin2016thanin2015.Theseroughlyoffseteachother,however,andaverageadmissionsscores,bothoverallandforapplicantsfromunderrepresentedgroups,werequitesimilarin2016asin2015.Overall,changesinthedistributionofobservablecharacteristicsamongapplicants,onitsown,wouldnotlikelyhaveproducedsubstantialchangesinapplicationoutcomes.
Table2.Distributionofadmissionsscoresin2015and2016 Californiaresidents,excludingrecruitedathletes
Allstudents
Underrepresentedgroups
Applicants
Admits
Applicants
Admits
2015 2016
2015 2016
2015 2016
2015 2016
Mean 0.17 0.17
0.49 0.45
0.13 0.13
0.42 0.40Fractionbelow1% 33% 38%
1% 3%
39% 45%
2% 5%
Fractionbelow5% 53% 57%
7% 12%
61% 64%
11% 15%
5thpercentile 0.00 0.00
0.04 0.02
0.00 0.00
0.02 0.0110thpercentile 0.00 0.00
0.08 0.04
0.00 0.00
0.05 0.03
25thpercentile 0.00 0.00
0.22 0.14
0.00 0.00
0.15 0.1050thpercentile 0.04 0.03
0.49 0.43
0.02 0.02
0.38 0.33
75thpercentile 0.22 0.22
0.76 0.75
0.14 0.13
0.67 0.6790thpercentile 0.61 0.65
0.89 0.90
0.45 0.49
0.85 0.87
95thpercentile 0.79 0.83
0.93 0.94
0.67 0.72
0.91 0.9299thpercentile 0.94 0.95
0.97 0.98
0.90 0.92
0.96 0.98
0.0
1.0
2.0
3.0
4.0
5Fr
actio
n
0 .2 .4 .6 .8 1Admissions score
Distribution of admissions score in 2015For those actually admitted
7
Figure5showstheshareofstudentsateachadmissionsscorelevelwhowereconsideredunderARin2015(solidline)oraskedforLORsin2016(dashedline).ThefeaturethatjumpsoutthemostisthattheLORprogramwasmassivelylargerthanARhadbeen–evenatthelowestadmissionsscores,whereARstudentsareconcentrated,theshareof2016studentsfromwhomLORswererequestedgreatlyexceedstheshareconsideredunderARin2015.Thesecondthingtonoticeisthat,whileARstudentswereconcentratedaroundthe0.3mark–thatis,thesestudentswere,basedonobservablesalone,muchweakerthantheaverageapplicant–LORswereusedmostamongstudentswithmuchhigheradmissionsscores,50-80%.Inthisrange,nearlyall2016applicantswereaskedforLORs,but10%orlesswereconsideredunderARin2015.
Figure5.ShareofCaliforniaresidentapplicantsconsideredunderARoraskedforletters
ThewaitlistBeyondtheAR/LORshift,anotherimportantchangein2016wasagreatlyexpandeduseofthewaitlist.Theshareofapplicantsofferedpositionsonthewaitlistnearlydoubled(from4.5%to8.6%)in2016.Manystudentsdeclinedtheseoffers–fully4%of2016applicantsdeclinedofferedpositionsonthewaitlist,ascomparedto1.5%in2015.Thisgreatlycomplicatescomparisonsof2015and2016outcomes,assomeofthestudentswhoturneddownpositionsonthewaitlistin2016wouldhavebeenadmittedoutrightin2015.ManyBerkeleyapplicantsarechoosingbetweenBerkeleyandotherexcellentuniversities,andmanywhoareacceptedwindupgoingelsewhere.In2015,lessthanhalfofadmittedstudentscametoBerkeley,andthissharewassmallerforstrongerapplicants.Inmanycases,studentswillhavealreadydecidedtoenrollelsewherebythetimeBerkeley’sinitialadmissionsoffersaremade.Consider,forexample,astudentadmittedelsewhereunderanearlydecisionprogram.Inprinciple,thisstudentmightwithdrawherBerkeleyapplication,butthishasnotbeeneasytodo,andinanyeventsomestudentsmightnotbotherwithnothingatstake.
0.2
.4.6
.81
Prob
abilit
y
0 .2 .4 .6 .8 1Admissions score
AR in 2015 LOR in 2016
All applicants
0.2
.4.6
.81
Prob
abilit
y
0 .2 .4 .6 .8 1Admissions score
AR in 2015 LOR in 2016
From underrepresented groups
Share of applicants considered under AR and LOR
8
Ifthesestudentsarechosenforinitialadmission,theycounttowardourstatisticsonadmittedstudents.Butiftheyareofferedpositionsonthewaitlist,theyarelikelytoturndownthisoption,andthuswillnotcountasadmissionseveniftheywouldeventuallyhavebeenadmittedoffthewaitlist.Thus,ashiftofsomeadmissionsfromthefirsttothesecondround,asoccurredin2016,willreducetheshareoftheseuninterestedstudentsintheadmitpool(and,asasideeffect,raisetheenrollmentrateamongthoseadmitted).Table3showsthedistributionofadmissionsoutcomes,aggregatingnon-athleteCaliforniaresidentsacrosseachoftheseparateapplicantpools(fordifferentcollegesanddivisions)butseparatingthedifferentstages.Thisillustratesthepotentialdistortioncausedbythewaitlist:Notethattheshareofunderrepresentedapplicantswhowereadmittedrosebyonly0.7percentagepoints(andtheshareadmittedinthefirstroundfellby0.2p.p.),whilethesharewhoenrolledroseby1.9p.p.Thisisbecausethe“yield”rateforadmittedstudentsroseby4percentagepoints,from47%to51%overall,andby10p.p.,from59%to69%,foradmittedstudentsfromtheunderrepresentedgroups.Table3.Stagesoftheadmissionsprocess
Californiaresidents,excludingrecruitedathletes
Shareofallapplicants(%)
Shareofapplicantsfrom4groups(%)
2015 2016 Change
2015 2016 Change
Initialadmit 16.1 18.1 2.0
13.3 13.0 -0.2Initialdeny 79.4 73.3 -6.1
83.8 78.9 -4.8
Offerwaitlist 4.5 8.6 4.1
3.0 8.0 5.1
Declinewaitlistoffer 1.5 4.0 2.5
1.2 4.3 3.1
Acceptwaitlistoffer 3.0 4.6 1.6
1.8 3.8 1.9
AdmittedfromWL 2.7 3.0 0.3
1.6 2.6 1.0
NotadmittedfromWL 0.3 1.6 1.3
0.2 1.2 1.0
Ultimateoutcomes
Admitted 18.8 21.1 2.3
14.9 15.6 0.7
Enrolled(SIR) 8.8 10.7 1.9
8.8 10.7 1.9
Didnotenroll 10.0 10.4 0.3
6.1 4.9 -1.2
Denied 79.7 74.9 -4.8
83.9 80.1 -3.9
WithdrewafterWLoffer 1.5 4.0 2.5
1.2 4.3 3.1
Table3reinforcesmyconcernthatissuesofself-selectionarequantitativelyimportant.Manystudentswhounder2015processeswouldhavebeenadmittedbutgoneelsewherewereinsteadin2016offeredpositionsonthewaitlistonlytodeclinetheoffers–perhapsasmanyas1%of2016applicants.Thesestudentswouldhavebeencountedasadmitsin2015butnotin2016.Importantly,thisaffectsthestatisticsforunderrepresentedstudents,asthesestudents
9
weredisproportionatelylikelytodeclinepositionsonthewaitlist:47%ofallstudentsofferedpositionsonthewaitlistdeclinedthem,butthissharewas53%forstudentsfromtheunderrepresentedgroups.Unfortunately,thereisnoadmissionsmeasurethatisperfectlycomparableacrossyears–inparticular,thecompositionofboththepoolofinitialadmitsandthepoolofeventualadmitsisaffectedbytheincreaseduseofthewaitlist,evenwithnootherchanges.Inthisreport,Ipresentresultsforfourdifferentmeasures,eachimperfect:
- Initialadmissions(includingbothFallandSpringadmits)- Admittedintheinitialroundorofferedapositiononthewaitlist- Everadmitted,eitherintheinitialroundoroffthewaitlist- Admittedandenrolled(asproxiedbyfilinganSIR,eitherforFallorSpringenrollment)
Thelastofthese,ofcourse,reflectsstudentaswellascampusdecisions(asdoesthethird,whichreflectsstudentdecisionstoacceptaspotonthewaitlist).Nevertheless,inmyviewitistheclosesttocomparableacrossyears.Ifstudents’propensitytoacceptBerkeleyadmissionsoffers,ifmade,didnotchangeacrossyears,andifastudentwhowouldacceptaninitialofferisnotputoffbybeingadmittedoffthewaitlist,changesinthepoolofenrolledstudentscanbeattributedtochangesinadmissionscriteria.AugmentedReviewandLettersofRecommendationThisreportfocusesontheAugmentedReviewandLettersofRecommendationcomponentsoftheadmissionsprocess.Table4showsthenumberofstudentsconsideredunderARin2015,thenumberaskedforLORsin2016,andtheoutcomesofeachgroupofapplications.AsalreadyindicatedbyFigure5,thismakesclearthattheLORprogramwasmuch,muchlargerthantheARpool,whichIunderstandwaskeptsmallduetotheenormousstafftimerequiredtoreviewARapplications.Table4alsoshowsthat15%ofstudentswhowereaskedforlettersdidnotrequestany.Thismightreflectwhatmanywereconcernedabout,thatstudentswouldnothaveaccesstoteacherswillingtowriteletters.Buttheaboveself-selectiondiscussionpointstoanotherpotentialexplanation:StudentsadmittedEarlyDecisionelsewhere,andothersnotveryinterestedinBerkeley,mightsimplynothavebotheredtorequestletters.Forthisreason,Idonotemphasizerequestsoforreceiptoflettersasoutcomes,andfocusontheimpactofthelettersrequestonthestudent’slikelihoodofbeingadmittedorofenrolling.
10
Table4.OutcomesforARandLORstudents
Californiaresidents,excludingrecruitedathletes
Allapplicants
Underrepresentedgroups
ARin2015
LORin2016
ARin2015
LORin2016
Numberaffected 3,046 14,406
2,793 6,337
Shareofallapplicants 7% 32%
12% 26%
LORoutcomes
Anyrequested
88%
85%
Tworequested
77%
72%
Anyreceived
87%
84%
#received=#requested
83%
79%
Tworeceived
73%
67%
Admissionsoutcomes(shares)
Initialadmit 27% 40%
27% 35%
AdmitorWLoffer 33% 59%
33% 54%
Everadmit 30% 46%
30% 40%
Admitandmatriculate 17% 23%
16% 19%
Admissionsoutcomes(numbers)
Initialadmit 816 5,833
759 2,200
AdmitorWLoffer 995 8,438
923 3,414
Everadmit 907 6,672
841 2,558
Admitandmatriculate 503 3,364
459 1,211
AssessingthelettersofrecommendationcomponentofthechangeIbeginmyanalysisbyattemptingtoassesstheimpactoftheletterofrecommendationcomponentofthe2016admissionsprocess.Analysesbytheadmissionsofficehavecontrastedthosewhoprovidedletterstothosewhowereaskedforlettersbutdidnotprovidethem.Theseareusefulinunderstandingwhichstudentsmayhavetroubleobtainingletters(thoughasnotedabove,astudentwhodoesnotobtainlettersmightjusthavedecidedtogoelsewhere).Itakeadifferentapproach:ThefactorthatisunderBerkeley’scontroliswhetherapplicantsareaskedforletters,soIattempttouncovertheimpactofthisonadmissionsoutcomes,withouttryingtodistinguisheffectscomingfromdifficultyinobtaininglettersfromthosecomingfrom(forexample)thesubmissionofweakletters.Asithappens,thewaythattheLORprocesswasimplementedallowsforacompellinganalysisoftheLORrequest,basedonacomparisonofstudentsaskedforLORswithnearlyidentical
11
studentswhojustmissedbeingasked.ThereweretwowaysthatstudentswereselectedforLORs:
• Theadmissionsofficeestimatedastatisticalmodeltopredictreadscoresin2015.3Those2016applicantswhosereadscoreswerepredictedtobe2.5,2.75,or3wereautomaticallyaskedforLORs.
• Applicantswhowerescoredbythefirstreaderas“Possible”(onthe2016Yes/Possible/Noscale),whenthiswasdonebeforethedeadlineforrequestingLORs,wereaskedforLORs.Abouttwo-thirdsofinitialreadsscored“possible”werecompletedbythedeadline.
Approximately80%ofthoseaskedforLORswereidentifiedbythefirstmethod,and40%bythesecondmethod.(20%wereidentifiedbybothmethods.)Although2015readscoresuseddiscretecategories(witheachreaderassigningascoreof1,2,2.5,3,or4,andwithlowernumbersgiventostrongerapplicants),thestatisticalmodelusedforthefirstmethodgeneratedcontinuouspredictions–thatis,anapplicantmighthavebeenpredictedtogetareadscoreof2.47.Studentswithpredictedreadscoresbetween2.38and3.26wereallaskedforletters,whilestudentswithreadscoresjustoutsidethisrangewereaskedonlyiftheywerecapturedbythesecondmethod.Figure6showstheshareofapplicantsateachpredictedreadscorewhowereaskedtosubmitletters.Forapplicantswithpredictedscoresbetween2.38and3.26,thiswas100%.Butonlyabout60%ofstudentswhowerejustabitstrongerthanthisrange(predictedscoresof2.37)wereaskedforlettersduetothefirstreader’sscore,andonlyabout3%ofstudentswhowerejustabitweakerthanthisrange(predictedscoresof3.27)wereasked.Thesesharpbreakspermita“regressiondiscontinuity”analysisoftheeffectoftheLORrequestonadmissionsoutcomes.Studentswithpredictedscoresof2.37areessentiallyidentical,onaverage,tothosewithpredictedscoresof2.38,andwouldalmostcertainlyhavehadverysimilaradmissionsoutcomeshadLORsnotbeenrequestedforsomanymoreofthelatter.Thus,anydifferenceintheiroutcomescanbeattributedtotheLORrequest.43Thepredictedreadscoreissimilarinspiritto,andreliesonthesamevariablesas,myadmissionsscorediscussedabove.Theydifferbecauselowerreadscoresarebetter,andbecausethepredictedreadscoreweightscharacteristicstopredictthe2015readscore,whiletheadmissionsscoreweightsthecharacteristicstopredictfirst-round2015admissions.Thecorrelationbetweenthetwoscoresisaround-0.85.4Anotherstrategymightbetocompareadmissionsoutcomesofthosewithpredictedreadscoresbetween2.38and3.26in2015and2016,relativetothoseoutsidethisrange.Unfortunately,Idonothaveaccesstopredictedreadscoresfor2015applicants,andhavebeenunsuccessfulinre-creatingthem.Iexpecttobeabletoeventually,butasIwritethisthekeystaffperson(GregDubrow,DirectorofResearchandPolicyAnalysisintheOfficeofUndergraduateAdmissions)isonvacation.Ithusdeferthistofuturestudy.
12
Figure6.FractionofCaliforniaresidentapplicantsaskedforlettersofrecommendation,bypredictedreadscore
Figure7shows2016admissionsoutcomesasfunctionsofthepredictedreadscore.Notsurprisingly,studentswithlowerpredictedreadscoresaremorelikelytobeadmitted.Butnoticetheareaaroundtheverticallinesat2.38and3.26,whereIallowfortheaverageadmissionsoutcomestochangediscontinuously.Applicantstotheleftofthefirstline,only60%ofwhomareaskedforLORs,aresomewhatmorelikelythanstudentstotherightoftheline,allofwhomwereaskedforLORs,tobeadmitted,tobeinvitedtothewaitlist,andtoenroll.BecausethereisnoreasontoexpectdifferencesinoutcomesbetweenthesestudentsexceptforthedifferenceintheirLORtreatment,thisisclearevidencethatforstudentswithpredictedreadscoresaround2.38–strongerthan95%ofapplicantsand81%ofadmitsin2016–beingaskedforanLORreducedtheprobabilityofadmission.
Figure7.Admissionsprobabilitiesbypredictedreadscore,2016,allCaliforniaresidentapplicants
Nowturntothesecondline.Theremaybesmallerdiscontinuitieshere,generallypointingtohigheradmissionsprobabilitiesforthe3.25swhoweredefinitelyaskedforlettersthanforthe
0.2
.4.6
.81
Frac
tion
aske
d fo
r LO
R
1.5 2 2.5 3 3.5 4 4.5Predicted read score from adm. office model
Likelihood of LOR request by predicted read score
0.2
.4.6
.81
Prob
abilit
y
2 2.5 3 3.5 4Predicted read score
Initial admitAdmit or WL offerEver admitSIR
All applicants
13
3.27swhohadonlya30%chanceofbeingasked.Thediscontinuitiesaresmallerhereandmaybeentirelyattributabletostatisticalnoise.However,thereweremanymoreapplicantswithpredictedscoresinthisrangethanaround2.38,soevenasmalleffectofLORson3.25studentswouldbequantitativelyimportant.ItisnotpossibletosaywhethertheLORimpactsseeninFigure7reflectbetteradmissionsdecisionsorworse,asitisnotpossibletoidentifythespecificstudentswhowereadmittedifnotaskedforlettersbutnotadmittedotherwise(orviceversa).ButitisworthnotingthatthisisexactlywhatwewouldexpectiftheLORsprovidedusefulinformation–somestudentswhowouldhavegottenthebenefitofthedoubtduetotheirstrongnumericcredentialswithoutLORswererevealedbytheLORstobeweakerthantheyappeared,whileotherswhowouldnothavegottenthebenefitofthedoubtwererevealedbytheirLORstobeworthadmitting.Ofcourse,theresultsarealsoconsistentwiththepossibilitythatsomestrongstudentsweredeniedadmissionbecausetheywereunabletoprovideLORs.Figure8repeatstheanalysisforapplicantsfromtheunderrepresentedgroups.Thesedataarenoisier,duetothesmallernumberofobservationshere.(Dotsaroundthe2.38thresholdrepresentanaverageof60applicantsinFigure7,andonly10-15inFigure8.)ThereisnosignherethatthestrongerstudentswerehurtbytheLORrequest,onaverage:Thoseontheleftofthe2.38lineareadmittedatessentiallythesamerateasthoseontheright.Figure8.Admissionsoutcomesbypredictedreadscore,2016,Californiaresidentapplicantsfromfourunderrepresentedgroups
Table3presentsquantitativeestimatesoftheeffectofanLORrequestonadmissionsoutcomes,separatelyforthosenearthe2.38thresholdandforthosenearthe3.26threshold.55Intechnicalterms,theseareinstrumentalvariablesestimatesfromafuzzyregressiondiscontinuitydesign.TheyreflectthelocalaverageeffectoftheLORrequestonstudentsneartherelevantthreshold.
0.2
.4.6
.81
Prob
abilit
y
2 2.5 3 3.5 4Predicted read score
Initial admitAdmit or WL offerEver admitSIR
Applicants from underrepresented groups
14
Inthefullapplicantpool,strongstudentsfromwhomLORswererequestedwere8-10percentagepointslesslikelytobeadmittedthantheywouldhavebeenhadtheynotbeenselectedforLORs.Forweakerstudents,theeffectwastoincreaseadmissionschancesby2-4percentagepoints.(Notethatthereareaboutfourtimesasmanystudentsnearthe3.26thresholdasnearthe2.38threshold,sotheimpliednumberofstudentsadmittedduetolettersneartheformeriscomparabletothenumberdeniedduetolettersnearthelatter.)Forunderrepresentedapplicants,the2.38thresholdeffectissmaller,suggestingthatLORrequestswerenotharmfultostrongstudentsfromthisgroup.
Table3.RegressiondiscontinuityestimatesoftheeffectofLORrequestsontheprobabilityofadmissionCaliforniaresidents,excludingrecruitedathletes.Standarderrorsinparentheses
Initialadmit
AdmitorWLoffer
Everadmit
SIR
Allapplicants
Strongerapplicants(lowpredictedreadscores)
-10.4% -6.3% -7.6% -13.6%
(5.5%) (4.7%) (5.4%) (5.2%)
Weakerapplicants(highpredictedreadscores)
+2.0% +3.6% +4.2% +3.6%
(2.3%) (2.5%) (2.4%) (2.0%)
Applicantsfromfourunderrepresentedgroups
Strongerapplicants(lowpredictedreadscores)
-6.5% -6.5% -4.1% -2.3%
(9.4%) (8.3%) (9.3%) (7.9%)
Weakerapplicants(highpredictedreadscores)
-1.8% +0.7% +1.6% +3.2%
(3.8%) (4.1%) (3.9%) (3.2%)
Netimpactonnumberofadmittedstudents
Allapplicants
Number -304 54 40 -300
Proportion -4% 0% 0% -6%
Fromfourunderrepresentedgroups
Number -181 -60 19 123
Proportion -5% -1% 1% 5%
ThefinalrowsofthetableattempttoestimatethenetimpactofLORsonadmissionsateachstage–positivenumbersindicateapositiveneteffect,andnegativenumbersanegativenet
15
effect.6Focusingonthelastcolumn,weseethattheLORrequirementraisedthenumberofunderrepresentedstudentswhoenrolledby123,whilereducingthenumberofenrolleesfromothergroupsby423.Thesearerelativelysmallnumbers,butshownosignofnegativeeffectsofLORsondiversityandindeedimplythatLORsraisedtheunderrepresentedshareofenrolledstudentsbyseveralpercentagepoints.Asnotedabove,wecannottellwhethertheLORaspectofthe2016proceduresledtobetterorworsedecisions.Butthereisnoindicationthatitreducedadmissionschancesforunderrepresentedorweakerstudents,whoseemmostlikelytohavefacedchallengesinobtainingsuitableletters.AssessingtheAugmentedReviewcomponentofthechangeThesecondmajorquestionIaddressiswhethertheeliminationofAugmentedReviewmadeitharderforthetypesofstudentsformerlyidentifiedforARtobeadmitted,orwhetherotherchangesmadetoadmissionsprocesseswereabletocompensateforthelackofaseparateARpool.Unfortunately,thereisnoregressiondiscontinuityresearchdesignavailableforassessingtheimpactofAugmentedReview.Moreover,thereisnowaytoidentifyinthe2016dataexactlywhichapplicantswouldhavebeenreferredtoARhaditbeeninplace,sowecannotcompareoutcomesforthispoolovertime.Asanalternative,Iidentifycandidateswho,basedontheircharacteristics,wouldhavebeenlikelytobereferredtoARin2015,andexaminehowtheiroutcomeschangedovertimerelativetootherswho,basedontheircharacteristics,wouldhavebeenunlikelytohavebeenreferredtoAR.Specifically,Icreateyetanotherscorefromthesamevariablesconsideredtodate,thisonerepresentingthelikelihoodthatastudentwiththesecharacteristicswouldhavebeenreferredtoandconfirmedforARin2015.7
6Thesecalculationsrequireratherheroicassumptions.Iassumethattheestimatedeffectsfoundatthetwodiscontinuitiesextendidenticallyoutsidethem,andIlinearlyinterpolateeffectsbetweenthediscontinuities.Imakenoallowanceforsamplingerrorinthisextrapolation.7Specifically,Ifitalogisticregression,using2015data,wheretheoutcomeisanindicatorforhavingbeenreferredtoandconfirmedforARandpredictorsareaquarticinthepredictedreadscore,indicatorsforthethreedisadvantagegroupsandinteractionsamongthem;andindicatorsandseparatequadraticsinthepredictedreadscoreforthosewith1,2,or3disadvantagefactors.(Amoreflexiblemodelthatincludesalloftheunderlyingvariablesinthemodelusedtogeneratethepredictedreadscoredoesnotgeneratemeaningfullybetterpredictions.)
16
50%ofapplicantshaveARprobabilitiesbelow2%,but10%haveprobabilitiesabove20%.Figure9showshowtheARprobabilityscorerelatestothereadscore,separatelyforstudentswithdifferentnumbersofdisadvantagefactors.(Themultiplelinesineachseriesrepresentdifferentcombinationsofwhichdisadvantagefactorsthestudenthas.)Acrossalldemographicgroups,studentswithpredictedreadscoresnear3.25aremorelikelytobeconfirmedforARthanthosewithhigherorlowerpredictedreadscores.Foranygivenpredictedreadscore,ARprobabilityscoresarehigherforthosewithmoreenumerateddisadvantagefactors.Forstudentswiththreedisadvantagefactorsandpredictedreadscoresbetween2.64and3.91,ARprobabilityscoresareabove0.2,andsometimessubstantiallyso.Studentswhohaveonlytwodisadvantagefactorsmusthavepredictedreadscoresinanarrowerrange,betweenabout3.1and3.7,toachieveARprobabilityscoresthishigh,whilestudentswithzerooronedisadvantagefactorsneverhaveARprobabilitiesabove0.12.
Figure9.EstimatedprobabilityofAugmentedReviewbypredictedreadscoreandnumberofunderrepresentationfactors,Californiaresidentsin2015
Unfortunately,whilethispredictionmodelisfairlysuccessful,itdoesnotachieveasharpdistinctionbetweenARandnon-ARstudents–eventhestudentswiththeabsolutehighestARprobabilitieshaveonlya40%chanceofbeingconfirmedforAR.Inlightofthis,IconsidertwodefinitionsofstudentsmostlikelytobeconsideredintheARpool:
- StudentswithARprobabilityscoresabove0.2(10%ofapplicantsand38%ofthoseconfirmedARin2015)
- Studentswhoarelowincome,firstgeneration,andfromlowAPIschools,withARprobabilityscoresabove0.2(7%ofapplicants,and28%ofthoseconfirmedforARin2015).
Table4showstheadmissionsoutcomesforstudentsineachofthesegroupsin2015and2016,aswellasfortheircomplements(studentswithlowerARprobabilityscores).Relativechangesatallmarginsexceptinitialoutcomesarepositiveorclosetozero.(Acrosseachdefinition,therelativechangesaremostinfavorofthehigh-riskgroupwhentheoutcomeisadmissionorthe
0.1
.2.3
.4Pr
obab
ility
refe
rred
and
confi
rmed
for A
R
2.5 3 3.5 4 4.5Predicted read score
3 factors2 factors1 factor0 factors
Probability of AR confirmation, by predicted read score andnumber of disadvantage factors, 2015
17
offerofawaitlistspot–itseemsthatmanystudentswhowouldhavebeenintheARpoolin2015wereofferedwaitlistspotsin2016butnotadmitted,eitherbecausetheywerenot
Table4.Probabilityofadmission,byARprobability,2015and2016Californiaresidents,excludingrecruitedathletes
Initialadmit
AdmitorWLoffer Everadmit SIR
Definition1:PredictedARprobability>20%
HighprobabilityAR
2015 17.5% 21.5% 19.4% 9.9%
2016 18.4% 32.5% 22.5% 12.3%
Change 0.9 11.0 3.1 2.5
ChangeforlowprobabilityARgroup
2.7 6.7 2.9 2.3
Differenceinchanges -1.8 4.3 0.2 0.2
Definition2:3disadvantagefactorsandpredictedARprobability>20%
HighprobabilityAR
2015 16.3% 19.2% 17.5% 8.9%
2016 16.0% 28.5% 19.7% 11.0%
Change -0.3 9.3 2.2 2.0
ChangeforlowprobabilityARgroup
2.7 6.9 3.0 2.3
Differenceinchanges -3.0 2.4 -0.8 -0.3
ImpactofAReliminationonnumberadmitted
Definition1 -248 603 30 25
Definition2 -396 318 -105 -38
selectedfromthewaitlistorbecausetheydeclinedtheoffer.)NeitherofthedefinitionsindicatesmeaningfuleffectsofARonthenumberofstudentswhoenrolled,andingeneral,itishardtodiscernchangesofmeaningfulmagnitudeintheadmissionsoutcomesofAR-typestudentsbetweenyears,suggestingthatchangesinotheraspectsoftheadmissionsprocessenabledthesestudentstogettheextraconsiderationin2016thattheygotthroughARin2015.TowardanunderstandingoftheoverallimpactofadmissionprocesschangesTheresultsthusfarsuggestthatLORs,ifanything,increaseddiversityoftheenteringclassin2016,andthattheeliminationofARhadatrivialeffect.Buttherewereanumberofotherchangesmadein2016,andoveralltheimpactwassomewhatlessthansatisfactory–theshareofstudentsfromunderrepresentedgroupsamongadmittedstudentsfell,thoughtheshareamongstudentswhoenrolledwasstable.Inthisfinalsection,Ipresentsomeanalysesofoveralloutcomesthatpointtopossiblecontributingfactors.
18
Figure10showsestimatesoftheshareofstudentsateachapplicationscorewhoweresuccessfulin2015and2016,foreachofthedefinitionsofsuccessdefinedearlier.Here,Iadjustthe2015applicantpooltomatchthedistributionacrosscollegesseenin2016,toremovetheinfluenceofshiftsacrossadmissionsprocessesthataremoreorlesscompetitive.Weseethatweakerapplicants(asmeasuredbyadmissionsscoresbetween0.1and0.4)weremorelikelytobeadmittedin2016thanin2015,butstrongerapplicants(scoresabove0.7)weresomewhatlesslikelytobeadmitted.Thelatterchangedisappearswhenweincludewaitlistedstudentswithinitialadmits,butitreappearsandisevenlargerwhenweexaminetheshareofapplicantswhowereeveradmitted(countingasfailuresthosewhowererejectedoutrightaswellasthosewhowereofferedwaitlistspotsbuteitherdeclinedthemorwerenotadmittedoffthewaitlist).Bycontrast,thestrongestapplicantsweremorelikelyin2016thanin2015tomatriculate.
Figure10.Admissionsprobabilitiesbyadmissionsscore,Californiaresidentsin2015and2016
Thecontrastbetweenthe3rdand4thpanelsisinformative–itsuggeststhatsomeverystrongstudentswhowereadmittedin2015butmatriculatedelsewherewerereclassifiedasnon-admitsin2016,eitherbecausetheadmissionsdecisiontookaccountinsomewayofthelikelihoodofmatriculationorbecausethesestudentsdroppedoutatthewaitliststage.Inanyevent,weseethatboththeverystrongestandweakerapplicantsweremorelikelytomatriculatein2016thanin2015,whiletherewaslittlechangeforthoseinthemiddlerange(between0.4and0.8).Figure11repeatsthisexercise,thistimeonlyforapplicantsfromthefourunderrepresentedgroups.Asintheoverallpool,weseeincreasedadmissionschancesin2016forapplicantswithadmissionsscoresaround0.2.Buthereweseefairlydramaticdeclinesinadmissionsofapplicantswithscoresabove0.5thattranslateintoreducedmatriculationaswell.
0.2
.4.6
.8Pr
obab
ility
0 .2 .4 .6 .8 1Admissions score
20152016
Initial admit
0.2
.4.6
.81
Prob
abilit
y
0 .2 .4 .6 .8 1Admissions score
Admit or WL offer
0.2
.4.6
.8Pr
obab
ility
0 .2 .4 .6 .8 1Admissions score
Ever admitted
0.1
.2.3
Prob
abilit
y
0 .2 .4 .6 .8 1Admissions score
Matriculated
Admissions probabilities, 2015 and 2016All applicants
19
Figure11.Admissionsprobabilitiesbyadmissionsscore,Californiaresidentapplicantsfromunderrepresentedgroups
Evidently,somethingintheadmissionsprocessesusedin2016reducedtheadmissionschancesofthestudentsfromunderrepresentedgroupswhowere,by2015standards,thestrongestintheirobservedcharacteristics.Onecandidateexplanationistheuseofnon-cognitivescores,whichmighthavebeensubtlybiasedagainststudentsfromunderrepresentedgroups;anotheristhatreadersmighthaveputlessweightonthefactorsmeasuringstudentsrelativetotheirschoolsinevaluating2016applications.Unfortunately,inthelimitedtimeIhadtopreparethisreport,Iwasnotabletogettothebottomofthischange.Itbearsfurtherstudy.Itisworthnoting,however,thatFigures10and11constitutestrongevidenceagainsttheviewthattheeliminationofARplayedamajorrole–recallthatARstudentsareconcentratedaroundadmissionsscoresnear0.2,whereadmissionschanceswentupthemostin2016.ConclusionBerkeleyadmissionsoutcomesforunderrepresentedstudentswere,bysomemeasures,disappointingin2016:Althoughmorewereadmittedoverall,andtheirshareofenrolledstudentswassteady,theymadeupalowershareofadmissionsoffersandparticularlyoffirst-roundadmissionsoffers.ItwasnaturaltowonderwhethertheeliminationofAugmentedReviewandtheadditionofLettersofRecommendationcontributedtothischange.Myanalysisoffersnosupportforthesepossibilities.Lettersofrecommendationseemtohavehurttheadmissionschancesofotherwise-strongapplicantsnotfromunderrepresentedgroups,withsmallerornoeffectsonapplicantsfromthosegroups,andthustohaveraisedtheshareofunderrepresentedstudentsamongadmissions.ItisdifficulttoidentifyanycleareffectofAReitherway,butinanyeventitwassmall.Theexplanationforthechangeinoutcomesin2016mustlieelsewhere,inoneoftheotherchangesmadetoadmissionsprocesses.
0.2
.4.6
.8Pr
obab
ility
0 .2 .4 .6 .8 1Admissions score
20152016
0.2
.4.6
.8Pr
obab
ility
0 .2 .4 .6 .8 1Admissions score
0.2
.4.6
.8Pr
obab
ility
0 .2 .4 .6 .8 1Admissions score
0.0
5.1
.15
.2.2
5Pr
obab
ility
0 .2 .4 .6 .8 1Admissions score
Admissions probabilities, 2015 and 2016Applicants from underrepresented groups