R e p o r t Prepared for: Houston Community College Prepared by: James Hedrick Report Title: The Effectiveness of On-Line Tutoring at Houston Community College (HCC)

2

Note: AskUp, Inc. and the AskUp software referenced herein was acquired by Upswing International, Inc. in 2014. Houston Community College has since continued using similar software under the Upswing/AskUp brand, “AskUp.”

Introduction Measuringtheimpactofanystudentsupportserviceisproblematicatbest,andtutoringisnoexception.Estimatingtheimpactoftutoringonstudentachievementisparticularlydifficultwhenusingrecentlydevelopedmoderntechnologylikeonlinetutoring.Herethestudentsandtutorsnevermeetfacetoface,usersarelargelyanonymous,andit’sdifficulttodeterminetheproperwaytoquantifybothstudentachievementandusage.Accuratelymeasuringtheeffectofstudentsupportservicesliketutoringisparticularlycrucialforcommunitycolleges.Comparedtofour-yearinstitutions,communitycollegesserveamoreat-risk,disadvantagedpopulation,anddemandforbothspaceandserviceshasexpandedrapidlyoverthelastdecade.AccordingtotheDepartmentofEducation,totalenrollmentincreasedovertenpercentattwo-yearuniversitiesbetween2000and2006,withover6.5millionstudentsnowattendingtheseinstitutions.1Manyofthosestudentswillneedextrasupportandassistanceinordertocompletetheirprogramsandachievetheireducationalgoals.However,whiletheneedforcommunitycollegesandsupportservicesisseeminglyunlimited,budgetsarenot.Giventheeconomicsituation,itseemsunlikelythatcommunitycollegeswillbeabletocountonalargeinfluxofstate,local,orfederaldollarsanytimesoon.Toservetheirstudents,communitycollegesmustmakeefficientuseofthefundsthatareavailableandtodothattheyneedeffectiveandreliableestimatesoftheimpactofdifferentservices.On-linetutoringservices,likeAskUpbyUpswingatHoustonCommunityCollege,arebecominganincreasinglypopular,low-costmethodofstudentsupport.However,likemanysupportservices,ithasneverbeensatisfactorilydemonstratedthattheytrulyimproveeitherstudentachievementorretention.Thispaperattemptstoprovidereliable,quantitativeestimatesoftheimpactofonlinetutoringonstudentachievementsothatadministratorsatHCCandothercommunitycollegescanmakeinformeddecisionsabouthowandwheretospendtheirbudgetsforsupportservices.

1TheDepartmentofEducationNationalCenterforEducationStatistics(NCES)website,accessedJanuary25,2009.http://nces.ed.gov/programs/coe/2008/analysis/sa_table.asp?tableID=1054

3

Academic Theory & Review of Selected Works ThemissionofthecommunitycollegesystemintheUnitedStatesistoprovideaccesstopostsecondaryeducationandcredentialsforthosewhomightnototherwisebeabletoattendcollege.AccordingtotheAmericanAssociationofCommunityColleges(AACC),“Communitycollegesarethegatewaytopostsecondaryeducationformanyminority,lowincome,andfirst-generationpostsecondaryeducationstudents…[they]alsoprovideaccesstoeducationformanynontraditionalstudentswhoareadultsandworkingwhileenrolled.”2However,afterthataccesshasbeenachieved,itisnecessarytoensurethatthattheprogramsandservicesofferedbyHCCandothercommunitycollegesaretrulyincreasingtheskillsandacademicabilitiesoftheirstudentsratherthansimplyperpetuatingdisadvantageassomecriticshaveclaimed(Krepsetal2007).Onlinetutoringandothersupportservicesareonewaythatcommunitycollegesattempttobridgetheachievementgapwiththeirstudents.Thismissionisevenmorecrucialinthe21stCentury.Theriseofglobalizationandtheexpansionoftechnologyhavedonenothingbutincreasetheneedforskillsandeducationforindividualstocompeteeconomically.CommunitycollegestudentrepresentedapproximatelyhalfoftheundergraduatepopulationintheUnitedStates.Manyofthesestudentscomefromdisadvantagedbackgrounds.Asmentionedabove,theyaredisproportionatelylow-incomeorminoritystudents.Theyarefrequently‘non-traditional’studentsaswell,workingadultswithfamiliesandjobsthataresignificantlyolderonaveragethanthefour-yearcollegepopulation.3Theireconomicwellbeing,nottomentionthatofthenationasawhole,dependsonthecommunitycollegesystem’sabilitytohelpthesestudentsachievetheireducationalgoals.Duetothedemographicsofthecommunitycollegestudentpopulation,achievementoftendependsontheavailabilityandeffectivenessofsupportservices,everythingfromchildcaretotutoring.Asoneofthelargestandmostdiversecommunitycollegesinthecountry,HoustonCommunityCollege(HCC)hasbeenattheforefrontofofferingeducationalopportunityandsupportservicesfortheirstudents.HCCservesapopulationofover55,000predominantlyminoritystudents.Accordingtothemostrecentstatistics,HoustonCommunityCollege’sstudentbodyisapproximately28%Hispanic,25%AfricanAmerican,19%Caucasianand10%Asian,andalsocontainsthelargestinternationalstudentenrollmentamongcommunitycollegesinthenationcomprisingapproximately10%ofthestudentpopulation(HCCSQuickFacts2008).TheracialandethniccompositionoftheHCCstudentbodyoffersaprescientglimpseatthepopulationthatothercommunitycollegesandinstitutionsofhigherlearningwillincreasinglybeservinginthecomingdecades.Beingabletodesignandimplementinnovativenewacademicsupportservices,suchasonlinetutoring,thatincreasetheachievementofminoritiesandotherdisadvantagedgroupsisnecessaryforHCCandothercommunitycollegestocontinuetheirmissionofservingasthe‘gatewayto

2AmericanAssociationofCommunityCollegeswebsite,accessedJanuary25,2009.http://www2.aacc.nche.edu/research/index.htm.3AmericanAssociationofCommunityCollegeswebsite,accessedJanuary25,2009.http://www2.aacc.nche.edu/research/index.htm.

4

opportunity’fortheirstudentsTherearetwoprimarytheoreticalviewsthatrelatetotheeffectivenessofonlinetutoringandothersupportservices.First,thetraditionalviewarguesthatsupplementalinstructionandservicesliketutoringimprovebasicskillsandcorrectunderlyingacademicdeficienciesrelatedtounderpreparationforcollege.Thisimprovementinbasicskillsshouldincreasemeasureableindicatorsofacademicachievement,suchasGPA,increasingthelikelihoodofstudentretentionandsuccess(HedgesandMajer1976;Zeidenbergetal2007).Theavailabilityoftheseservicesshouldsupporttheeffortsofnon-traditionalanddisadvantagedstudents,giventheirgreaterlevelsofunderpreparation,increasingtheiracademicachievementandhelpingthemprogresstowardstheirprofessionalandeducationalgoals.Unfortunately,whiletheacademicresearchavailableontutoringandothersupportserviceshasalongifsparsehistory,theempiricalresultsconcerningtheeffectivenessofsupportservicesliketutoringhavebeenmixed.Mucholderresearchsupportedthetraditionalmodelfindingthatsupportservices,specificallyindividualizedtutoring,couldhaveapositiveeffectonachievement(HedgesandMajer1976).Atleastonemeta-analysisusing65differentstudiesfoundthatindividualizedtutoringcouldbesuccessfulatalleducationallevels(Cohenetal1982).Otherexperimentalresearchfoundevidenceforincreasedacademicsuccess,butonlyforhighachieversandinverynarrowsubjectareas(Irwin1980,1981).Inoneofthemorerigorousandrecentempiricalstudiesinthisarea,Muraskinfoundthatfederallyfundedstudentsupportprogramsincreasedbothgradepointaveragesandthelikelihoodofstudentretention(Muraskin,1997).Unfortunately,manystudiesinthisareahaveoftensufferedfromalackofappropriatecontrols,suchaspreviousacademicability,oralackofmethodologicalrigor(BaileyandAlfonso2005).Thisreportattemptstoacknowledgetheseoversightsbyusingappropriatecontrolsandtakingintoaccounttheuniqueissuespresentinthecommunitycollegesystem.Inshort,withinthetraditionalframework,supportservicesshouldbedirectedatcorrectingthespecificacademicdeficienciesofdifferentindividualsandimprovingtheirbasicskillsinreading,writing,andsubjectmatterknowledge.Servicesliketutoringthatfocusonspecificproblemareasshouldimproveunderstanding,increaseachievementandcontributetothesuccessofdisadvantagedstudentsbyofferingtargetedsupport.Figure1belowpresentsasimplediagramoftheimpactofsupportservicesinthisframework.Eachindividualstudent’scollegereadiness,definedalmost

5

entirelybyacademicpreparation,combineswithsupportserviceusagetoproduceacademicachievement.Itisthismodelofsupportservicesthatthisreportwilltest.However,muchrecentresearcharguesexactlytheoppositeofthetraditionalmodel.Inareviewoftheliteratureonpeertutoringatthecollegelevel,Maxwellfoundnoconsistentevidenceofincreasedachievementforweakerorat-riskstudentswhousedtutoringservices.Otherevidencesuggestedthatthosewhoweresuccessfulwerebetterpreparedforcollegeinitially,bothculturallyandacademically(Maxwell1990).Inaddition,whencontrollingforacademicpreparationusingSATscores,McGintyfoundthattutoringhad,atbest,amarginalpositiveimpactonGPAformoststudentsandnoimpactonthosewiththeweakestacademicpreparation(McGinty1989).Whenusingamatchedsamplesapproachtoaddresstheselectionbiasinherentintheuseofsupportservice,anissuediscussedinmoredetailbelow,Pribeshandhercolleaguesalsofoundlittleeffectontheachievementlevelsofdisadvantagedprimaryschoolstudents,althoughgeneralizingthisfindingtocollege-agestudentsisobviouslyproblematic(Pribeshetal2007).Scholarssuchasthesearguethattobeatallsuccessful,supportservicesneedtofollowthe‘studentintegration’model,originallyproposedbyTintotoaddressstudentretentionandgraduationrates(1976;1993).Accordingtothismoresociologicalview,manytypesofsupportservicesoftenperpetuatedisadvantageandfailtoimproveeithertheretentionorachievementofthemostat-riskstudents,particularlyminorities(Rosenbaumetal2006).Thestrongestsupportersofthisviewpointevenarguethattheentirestructureofthecommunitycollegesystemperpetuatesdisadvantagesandthatsupportservicessuchastutoringassistsonlythosewiththenecessary‘pre-existingsocialandculturalresourcescantakefulladvantageofthem(2008).’Withinthisframework,representedbyFigure2below,targetedacademicsupportservicessuchastutoringprimarilyassistpreviouslyadvantagedstudentsandshouldhavealimitedimpactonboththeachievementandretentionofdisadvantagedstudents.Thesestudentsaresubjecttoenvironmental,cultural,andeconomicstrainsthatcannotbeaddressedeffectivelybysimpleacademicassistance,if,indeed,theycanevenaccessthem.Thesescholarsarguethatonlyprogramsthataddressthesocialintegrationofdisadvantagedstudentswillbesuccessfulandpurelyacademicsupportservicesdonotcorrectunderlyingdifferencesincollegereadiness,whichincludesfactorsbeyondsimpleacademicpreparation.Inshort,thesamethingthatdrivesacademicachievementdrivesusageaswell,resultinginpossiblecorrelationbetweenusageandachievementbutnocausalrelationshipbetweenthetwo.Simplyput,supportserviceslikeonlinetutoringonlyassistthosewhowouldlikelyhaveprosperedanywayandanylinkbetweenusageandachievementis

6

spurious.Unfortunately,programslikethisarealmostimpossibletoimplementinthecommunitycollegesystem,whichdealsalmostexclusivelywithworkingadults,commuters,andother“non-traditional”students.

Applicability of Previous Research to Online Tutoring Tosummarize,despitetheimportanceofcommunitycollegestotheeducationalsystemasawholeandtheimportanceofsupportservicesliketutoringontheachievementofstudents,theevidencefromtheextantresearchhasbeenminimal,mixed,methodologicallysuspectattimes,andfailedtoaddressthespecificissuesofthecommunitycollegepopulation(BaileyandAlfonso2005).It’sarguablewhethersupportserviceshaveanypositiveeffectonstudentachievementandevenmoreuncertainwhethernewtechnologieslikeonlinetutoringcanhaveanyimpact.Alltold,theacademicresearchreviewedhereseemstoindicatethatwhilesupportservicescertainlycanincreaseacademicachievement,context,structure,andavailabilitymatter.Accessshouldbeasopenaspossibleandinformationshouldbeavailableaswidelyaspossibleiftheservicesaretobeeffective.Coststostudenttimeandtherequirementofpreviousknowledgeshouldbelimited.Researchersskepticaloftheeffectivenessofsupportservicedhaveapointthatbarrierstoaccesscanlimitthereachofservicesliketutoring.Certainly,accessingsupportservicescanbeadrainonstudenttime,requirespriorknowledge,canentailsimplemonetarycosts,andfrequentlyrequiresstudentstoovercomethesocialstigmaofneedingtutoring.However,therearenumerousreasonstoquestiontheapplicabilityofrecentresearchthathasdownplayedtheeffectivenessofsupportservicestoonlinetutoring.Onlinetutoringdirectlyaddressesmanyoftheseproblems,presumablydiminishingtherelatedstudentaccesscostsandbroadeningthereachoftheservicetoalargerpopulationofat-riskstudents.Firstandforemost,theonlinetutoringavailableatHoustonCommunityCollegeisentirelycostfreeforstudentsinafinancialsense,sinceitisincludedintheiradmission.Therearenorestrictionsoneitheraccessorfrequencyofuse.Secondly,itisavailable24-7andrequiresonlyasmuchtimefromthestudentasittakestologin,uploadadocument,andlogout.Whiletheremightbecertainaccessissuesrelatedtotheavailabilityoftechnologyforsomestudents,webelievethatthisisminimalanddiminishingrapidly.CertainlycomputeraccessiswidelyavailableontheHCCcampuses.Finally,thesystemisrelativelyanonymous.Studentsareidentifiedonlybyausernametheythemselvesselectandcanthereforeavoidanyreluctancetheyhavetoaskforhelpface-to-face.Communicationisdonebyemailandmessageposts.Tutorsdownloadthefilesattheirconvenienceandre-uploadthempromptlywithcommentsandsuggestedimprovementsforthestudenttoaccessattheirconvenience.Sincetheliteraturefocusesonthecostsassociatedwithaccessandusage,thelowcostsassociatedwithonlinetutoringshouldresultinimprovedachievementforthebroadestpossiblegroupofstudents.Inaddition,recentresearchhasonsupportservicesbeenalmostuniformlyqualitative,oftenduetolackofdatareliabilityandavailability.Thequantitativeworkthathasbeendoneontheimpactofsupportserviceshasoftenbeensurvey-basedorlimitedtocasestudies.However,thenatureoftheAskUptutoringsystematHoustonCommunityCollege

7

hasallowedustogatherawealthofquantitativedataonindividualstudents,theiruseofthesystem,theirpreviousacademicability,andtheirsubsequentachievementlevels.ThisprovidesustheopportunitytospecificallytesttheeffectivenessofAskUpinamorestatisticallyrigorousmannerthanhasbeenattemptedbefore.WhilethisreportisspecificallyfocusedonthequantifiableimpactoftheAskUpprogram,theresultshaveimplicationsforothersupportservicesandtheirimpactonstudentachievementmoregenerally

Research Design & Data Thebestandmostdirectwaytoobservewhat,ifany,relationshipexistsbetweenonlinetutoringandstudentachievementistomeasureandanalyzebothattheindividuallevel.AspecificstudenteitherdoesordoesnotutilizetheAskUpsystem,andtheirscoresimprove,decline,orstaythesame.Usersandnon-userscanbecomparedandtheirachievementanalyzedinlightofdifferentdemographiccharacteristicsandachievementlevels.Thisrequiresthatbothachievementandserviceusagebetrackedandcompiledforindividualstudentsratherthanintheaggregate.Therefore,itwasnecessaryforthisstudytogatherdatafromtwosources:AskUpandHCCstudentrecords.Thankfully,AskUprequiresauniqueusernametologinandtracksindividualstudentusageoftheservice.Thisinformationallowedustocompileadatasetthatcontainsanaccurateandcompletehistoryofindividualstudentusageovertime.AskUpwasinitiallyabletoprovideusagedataon5,733individualstudentusersfrom2006-2008.AllotherdatacamedirectlyfromstudentrecordsprovidedbyHCC.Thisallowedfortheinclusionofbothdemographicinformationandmeasuresofacademicabilityintheresultingdataset.Thebiggestdeficitinmostresearchonsupportserviceshasbeenthelackofanappropriatemeasuretocontrolforstudentacademicabilityandpreparation.Withoutcontrollingforstudentabilityandpreparation,itisimpossibletodeterminewhetherincreasedachievementisduedirectlytoserviceusageortoaselectionbiasintheuserpopulation.Measuresonentry-levelassessmenttestsareparticularlyusefulforcontrollingforthispossibleselectionbias(BaileyandAlfonso2005).Asluckwouldhaveit,academicreadinessexamsarearequirementforenrollmentatHoustonCommunityCollege.4Unlesstheyhavereceivedawaiver,eachincomingstudentmustprove“college-readiness”bytakingaTexasStateInitiative(TSI)approvedtest.5Whilethisrequirementdoesnotextendtotransferstudents,itwillprovideuswithabaselinefromwhichtomeasurestudentimprovement.Scoresontheirenrollmenttestwillserveasapretesttocontrolforeachstudent’sincominglevelofacademicreadiness,allowingustogetatthe

4HoustonCommunityCollegeWebsite,accessedonMay19,2009.http://www.hccs.edu/hccs/future-students/requirements/testing-requirements-to-attend-hcc.5HoustonCommunityCollegeWebsite,accessedonMay20,2009.http://www.hccs.edu/portal/site/hcc/menuitem.f6a945c7befcd217d3ef7510d07401ca/?vgnextoid=8c031de612dc5110VgnVCM100000054710acRCRD&vgnextchannel=08a4b3a3bd5f4110VgnVCM100000054710acRCRD&vgnextfmt=default).

8

independentimpactofonlinetutoring.Includingacademicreadinessinthesubsequentanalyseswillcontrolforthelargestconfoundingfactorinquantifyingtheimpactofsupportservicesasindicatedbytheliteraturereviewedabove.OtherdemographicinformationprovidedbyHCCallowedfortheexaminationofdifferentsubgroupsofHCCstudents,particularlyminorities,inthefinalanalysis.Finally,theusagedatafromAskUpweremergedwiththestudentrecordsprovidedbyHCCusingrecoveredstudentidentificationnumbers.Unfortunately,notalloftheindividualstudentswhousedAskUpcouldbematchedwiththeiracademicrecordsfromHCC.ThiswasprimarilyduetodifficultiesinmatchingdifferentstudentidentificationnumbersusedbydifferentdepartmentswithinHCCtotheidentificationnumberscapturedbyAskUp.Intheend,1,196individualuserswerematchedwiththeirenrollmentandachievementinformationfromtheHCCdatabase.WhilethissubsetofstudentscannotbecompareddemographicallytothosewhocouldnotbematchedtotheirindividualHCCrecords,diagnosticsdidnotindicatetherewasanysignificantvariationbetweentheserviceusageofmatchedandunmatchedstudents.Inaddition,thestudentswhocouldbematchedtotheirtranscriptsapproximatelymirroredcollege-wideaveragesonseveralothertraitsavailablefromtheirrecords.Finally,inordertoprovideacontrolgroupfortheseserviceusers,HCCprovidedthesameacademicinformationforarandomsampleof5,000studentsfromthesametimeperiodwhowerenotamongtheoriginal5,733usersprovidedbyAskUp.Thesecombinedsamplesofusersandnon-usersservedastheinitialdatasetforthisproject.

Analysis

Descriptive Statistics Beforemovingintoinferentialstatisticsandhypothesistesting,itseemsreasonabletooffersomedescriptivestatisticscomparingthesampleofonlinetutoringuserstothesampleofnon-users.First,Tables1and2belowofferaquicksummaryofsystemusage.Allmeasuresarepersemesteraveragesofusage.Asyoucansee,fromthestudentsthatcouldbematchedtotheirtranscriptsandwereincludedinthefinalanalysis,averageusagewasalittlemorethanthreetimespersemesterandstudentusersuploadedabouttwofilespersemesteraswell.Thisisonlyaverageusage;somestudentswerefarmorefrequentusers,othersfarless.Thisiseasilyseenintherangefrom.5usesuptoanaverageof25usespersemesterinTable1.

Table1

Table2

9

Figures3,4,and5belowoffermoredetailed,side-by-sidecomparisonsbetweentheuserandnon-usergroupsonage,gender,andrace.Whatisimmediatelyobviousisthattherearefewgenderoragedifferencesbetweentutoringusersandnon-users.Onlinetutoringusersareslightlyyoungerandmorelikelytobefemaleonaverage,comparedtothepopulationoftheHCCstudentbodyasawhole,butthedifferencesarenotparticularlysignificant.Usersarelikelytobeapproximatelyayearyounger,onaverage,thannon-users,which,whilestatisticallymeaningful,isnotparticularlysubstantivelyinteresting.Table3wasincludedtohelpclarifytherelationshipbetweenageandtutoringusage,sincetheFigure4tendstovisuallyoveremphasizethedifferencesbetweenthetwogroups.However,whiletherearenotsignificantdifferencebetweenusersandnon-usersbyageorgender,therearesignificantracialandethnicdifferencesbetweenthoseHCCstudentwhotakeadvantageofandusetheAskUpsystemandthosethatdonotasshowninFigure5.First,it’simportanttonotethatthesampleofnon-usersprovidedbyHCCusingarandomsampleof5,000roughlymirrorstheoverallracialdistributionoftheHCCstudentbodyasdescribedearlier.Thisgivesusgreaterconfidencethattheinferencesmadeusingthisdataaremorelikelytobegeneralizabletothestudentbodyasawhole.Secondly,it’sobviousthattwogroupsareoverrepresentedinthepopulationofusersoftheAskUpsystem,internationalstudentsandAsianstudents,particularlyinternationalstudents.Whitestudentsarealsoseverelyunderrepresentedintheuserpopulation,relativetothemake-upofthestudentbodyasawhole.Thisislikelyfortworeasons:onepositive,onenegative.

10

Figure3

Firstandmostobviously,AskUpisbeingusedbythosewiththemostobviousneed.Internationalstudents,likelywithlimitedEnglishproficiency,aretakingadvantageofthesystemtoimprovetheirassignments.TheserviceoriginallyofferedprimarilyEnglishtutoring,addingothersubjectsgradually.ThisoverrepresentationislikelyduetolagtimeinadoptionbystudentsforwhomEnglishistheirprimarylanguage.However,thesignificantunderrepresentationofwhitestudentsintheuserpopulation,aswellasthemoremoderateunderrepresentationofblacksandHispanics,indicatesthatknowledgeofthesystemmaynotbewidespreadamongthegeneralstudentbody.Thesethreegroupscollectivelyrepresentmorethan70%oftheHCCstudentbody.Inshort,particulargroupsareusingthesystem,likelylearningaboutitthroughword-of-mouthorfromtheirinstructors,whilethegeneralpopulationseemstobelessawareoftheservice.ThisisalsoindicatedbytherelativelysmallnumberofuniqueusersthatAskUpwasabletoidentify–5,733–comparedtothesizeoftheHCCstudentbodyasawhole,approximately55,000.Inotherwords,onlyroughly10%ofHCCstudentshaveevenutilizedtheAskUpsystemevenonce,andthoseusersarehighlyconcentratedinnon-representativesubpopulationswithintheHCCstudentbody.

11

Figure4

Table3

12

Figure5

Inshort,giventhesimplestatisticspresentedhere,itseemsthatabroaderandmorecoordinatedoutreacheffortisneededtoinformthegeneralstudentbodyabouttheAskUpsystem.AsIunderstandit,thisoutreacheffortisalreadyinprogress,withAskUptutorsattendingvariousHCCstudenteventsatdifferentcampuses,promotingthesystemandencouragingstudentuse.Ifthisoutreacheffortsucceedsandabroaderrangeofstudentslearnsaboutthesystem,itislikelythatthedisparitiesbetweenusersandnon-userswilldecline,eliminatingtheracialandethnicusagegapssummarizedabove.

Inferential Statistics & Analysis Asusefulasthestatisticsaboveare,theytellusverylittleabouttheeffectivenessofthesystem.Ifthesystemiseffective,theycanbehelpfulindeterminingwhototargetforoutreach,butthetellusverylittleaboutwhetherornotthestudentsusingthesystemareincreasingtheirachievement.Forthat,weneedtoturntoregressionmodelingandinferentialstatistics.Thisreportusesasimplecross-sectionaldesigntotesttheimpactofusageonachievementforthestudentsincludedinthefinalsample.TheprimaryexplanatoryvariablemeasuringusageisaveragenumberofAskUpusagesperlongsemesterforeachstudentuserfromfall

13

2006tospring2008,excludingsummerandwinterbreaksemesters.Thisprovidesamoreconsistentmeasureofthestudentusagethansimplyusingthetotalnumberofloginsovertheperiod.Similarly,theprimarydependentvariableisthecumulativeGPAforeachstudentforthefoursemestersincluded.Thisalsoprovidesamoreconsistentoverallmeasureoftheimpactofserviceusageonachievement,ratherthanattemptingtobreakdowntheeffectofusageonparticularsubjectareassuchasEnglish,math,etc.Finallythemodelincludesasetofcontrolsbasedonpreviousresearchinthisarea(Zeindenberg2007).Inadditiontoscoresontheentrance-examdescribedabove,themodelincludesageandgender,aswellasraceandethnicitycodedasaseriesofdummyvariablesbasedontheCensusBureaucategories.Whilethisisaveryparsimoniousmodelandcouldbeexpandedtoincludeothervariables,itmatchesmodelsusedinpreviousresearchandincludesvariablesthatrelatetothemostpertinentandrelevanttheoreticalconcepts.Subsequentregressionanalysiswasconductedusingthegenerallinearmodelprovidedbelow:Achievementi=Constant+β1CollegeReadiness+β2Ethnicity+β3Age+β4Gender+β5UsageWhilethemodelaboveincludesacontrolforacademicreadiness,therestillexiststhepossibilityofimbalancebetweenthetreatedanduntreatedgroupsinthedataset.Sinceitisnotclearwhatthetreatmentassignmentmechanismwas,otherthanself-selection,serviceusagecannotbeassumedtoberandom(Icarusetal2008),andtheevidenceindicatesquiteobviouslythatitisnot.Table4belowshowsasimplemeanestimationoftheGPAforusersandnon-usersinthesample.WhileTable4indicatesthatusersaremorelikelytohavesignificantlyhigherGPA’sthatnon-users,wecannotdeterminefromthisinformationwhethertheincreasedachievementisduetoserviceusageorpreviousacademicability.RememberFigures1&2above.Thesesimplestatisticscannotdistinguishcausefromeffect.Theeffectisequallylikelytobeanartifactthatstudentswhoaremorelikelytobesuccessfultobeginwith,choosetouseAskUpmorefrequentlythanmoredisadvantagedstudents.Inthiscase,increasedoutreachandserviceusagewouldbeunlikelytotranslateintoincreasedgainsfromnewusers.Whilecertainlyheartening,theconsiderabledifferenceinthescoreslikeltindicatesaconsiderableselectionbiasinserviceusageandtheneedtocontrol,asmuchaspossible,fortheimbalancebetweenthetreatedandnon-treatedgroups,evenaftercontrollingforcollegereadinessusingentranceexamscores.Table4Mean Estimation GPA SE Service Users 2.915 0.023 Non-Users 2.685 0.014 Tofurtherreducetheimpactoftheimbalancebetweenthetreatedandnon-treatedgroups(usersandnon-usersrespectively)inthesampleonthesubsequentanalysis,Iemployedcoarsenedexactmatching(CEM)asdescribedinIcarusetal2008.Thesamplewasmatchedonallthevariablesinthemodel,includingcollegereadiness,gender,andethnicitywithusagecodedasa0/1dummyasthetreatmentvariable.Thematchingresultedina

14

finalanalysissampleof3535cases,withmultiplecontrol(non-user)casesmatchedtoeachtreatedcase.ThemultivariateL1distance,ameasureofimbalanceinthesample,was1.3006001beforethematchingprocessand1.0648765afterward,resultinginanapproximately18%reductionintheimbalance.Moresimplyput,eachindividualstudentuserwasmatched,ascloselyaspossible,toanotherstudentorstudentsidenticaltothemselvesintermsofage,race,levelofcollegereadiness,andgender.Asmuchaspossible,theonlydifferenceallowedwasusageofAskUp.Statisticallyspeakingthisgivesusmoreconfidenceinourinferencesconcerningmagnitudeofeffectandthecausalrelationshipbetweentutoringandachievement,conditionaluponthefactthatwehavematchedonalltheappropriatevariables.Giventheresearchreviewedaboveandtheavailabledata,Ibelievethismatchingrepresentsthebestpossiblewayofreducingimbalancebetweenstudentusersandnon-users.AllsubsequentanalyseswererunonthismatcheddatasetusinglinearregressionandtheweightsprovidedbytheCEMprogramtoensureappropriatestandarderrorsandconfidenceintervals.Finally,somefewstudentshadextremevaluesontheprimaryexplanatoryvariable,averageusagepersemester,thatwerefarinexcessofmostoftherestofthesample.Forexample,onestudentusedtheserviceanaverageof115timespersemester.Thesefewoutliershadasubstantiveimpactontheresultsoftheregression,andtherefore,theanalysisonlyincludedstudentswhousedtheservicelessthananaverageof23timespersemester,resultingineightcasesbeingdroppedfromthesample.TheextremelysmallnumberofNativeAmericanstudents,twelve,werealsoexcludedfromthefinalanalysis.Inshort,thefinalsampleusesmatchingtechniquestominimizethedifferencebetweenthetreatedanduntreatedstudentsonallavailablevariablesrelatedtoconceptsthatmightimpactachievementandincludesadistributionofstudentsthatshouldberepresentativeoftheHCCstudentbody.TheresultsoftheregressionanalysiscanbeseenbelowinTable5.

15

Table5Table of Coefficients: OLS Coef SE Exam 0.010*** 0.001 White (dropped) Black -0.192*** 0.057 Hispanic 0.151*** 0.056 Asian 0.184*** 0.064 Internat’l 0.346*** 0.055 Unknown Ethnicity -0.072 0.090

Age 0.030*** 0.003 Female 0.313*** 0.033 Avg_Use 0.053*** 0.008 Constant 1.103*** 0.102 note: *** p<0.01, ** p<0.05, * p<0.1, N = 3270

Theabovetablerequiresasmallamountofexplanation.Firstandforemost,thecoefficientsindicatetheimpactofoneunitofthevariableonoverallGPA.Forexample,aoneyearincreaseinage,resultsina.03increaseinGPAoverthetimeperiod.Secondly,theasterisksrepresentstatisticalsignificance,whetherornottheimpactindicatedhasa“true”effectorwhethertheresultcouldhavehappenedbychance.Forexample,theunknownracialvariableisnotstatisticallysignificant,likelyduetotheheterogeneityofthestudentswithinthepopulation,sothenegativeimpactisinsignificant.Moreasterisksmeanthattheeffectislesslikelytohaveoccurredbychance.Almostallthevariablesarehighlysignificant,indicatingthatwecanhaveaveryhighdegreeofconfidenceintheimpactofthedifferentvariables,includingusageoftheAskUpsystem.Next,theConstantindicatesthebaselineGPAastudentwouldhaveachievedifallothervariablesweresettozero.Finally,thevariablefor‘white’wasdroppedintheanalysis,ensuringthattheimpactoftheotherracialgroupsonachievementisrelativetotheCaucasianstudents.Forexample,allelsebeingequal,theachievementofAsianstudentis.184gradepointsabovethatofwhitestudentsasagroup.Thatsaid,asyoucansee,ourprimaryvariableofinterest,averageusageoftheAskUpsystempersemester,isstatisticallysignificantinthemodelaboveandhasaconsiderablesubstantiveimpactonGPAaswell.Foreveryone-unitincreaseintutoringusagepersemester,studentGPAincreasesbyapproximately.05points.Forexample,astudentwhousedtheservicefivetimesasemesteronaveragewouldincreasetheirGPAabout.25,enoughtoturnahighCintoalowBorahighBintoanA.Increasedusage,obviously,wouldresultinincreasedimprovementinGPA,whichsupportsthetraditionalmodelofsupportservicesandindicatesthatserviceusagedoeshavebothapositiveandsignificantimpactonstudentachievement,evenfordisadvantagedstudentsandthosewithlowerlevelsofcollegereadiness.

16

Inaddition,theeffectoftheusagevariablehasmoreoverallimpactthanbothageandcollegereadiness,atleastasmeasuredbyscoresontheCOMPASSexam.Theimpactoftutoringisoverfivetimesthatofcollegereadiness,indicatingthattargeted,supplementalacademicinstructioncanhelpstudentsatalllevelsofacademicpreparationimprove,andhelpdisadvantagedstudentscompensateforinitialacademicunderpreparation.Inthiscase,theinstructionprovidesnoneofthesocialintegrationorcommunitybuildingthatisarguedforbysupportersofthestudentintegrationmodel,butstillsignificantlyandsubstantivelyincreasesstudentachievement.Finally,themagnitudeoftheusagevariableislargeenoughthatmoderate,continuoususageovertimeseemstobeenoughtocompensateforanysocio-culturaldisadvantagecapturedbytheethnicandracialvariables.Forexample,anAfrican-Americanstudentwhousedtheserviceanaverageoffourtimespersemester,approximatelytheaverageusageofthosewhousedthesystem,wouldmorethancompensateforanyracialdisadvantageinachievement.ThisisstrongevidencethattheAskUpsystematHCCcananddoesincreasetheacademicsuccessofminoritystudents,evenafteraddressingtheselectionbiasinherentintheusageoftheserviceandpreviousacademicpreparation.Thegraphsbelowshowthismoreclearlyandintuitively.TheanalysisabovewasconductedusingGaryKing’sClarifyprogramforSTATA,whichusessimulationtocreateconfidenceintervalsaroundpredictedvaluesofthedependentvariable(Kingetal2000).InFigure3below,thesubstantiveeffectoftutoringonGPAisgraphedforbothblacksandHispanics.Scoresontheentranceexamandagearesetattheirmean,whilethegendersareshownside-by-side,giventhestrongimpactofgendershowninTable5.Thebluebarsrepresent95%confidenceintervalsaroundthepredictedGPA,indicatingourfaithinthepredictedGPA’s.Allothergraphsfordifferentethnicandracialgroupsfollowasimilarpattern,differingonlybytheintercept.Inotherwords,theimpactofonlinetutoringonachievementisconstantandpositivealllevelsofcollegereadiness,differentracialgroups,ages,andgender.

17

Figure6

Obviously,thereliabilityofthepredictionsdiminishesastheusageincreasessinceveryfewstudentsusetheservicemorethananaverageofabouttentimespersemester.However,thedirectionandmagnitudeoftheeffectofincreasedusageofAskUpareobvious.PredictedGPAincreasesapproximatelyonepointasusagegoesfromzerototwentytimespersemester,theequivalentofgoingfroman85averagetoa95average,orturningaB-studentintoanA-student.Thiseffectisstableacrossbothgenderandraceandholdsforothergroups,suchasinternationalstudents,notincludedinFigure3.Inshort,supportservicesliketutoringcanbeeffectiveatsubstantivelyimprovingtheachievementofcommunitycollegestudents,particularlyminorityandotherdisadvantagedstudents,evencontrollingforacademicpreparationandlimitingtheassistancetosupplementalacademicinstruction.

Conclusion & Recommendations Toconclude,theAskUpserviceiseffectiveatincreasingstudentachievement,hasasubstantiveimpactonstudentgrades,andsignificantlyimprovestheGPA’sofdisadvantagedstudents.Thisfindingisstatisticallyrobustandcontinuesevenaftercontrollingforanumberofconfoundingfactors.Inshort,AskUpworks,anditworkswell.ThisisnottosaythattheAskUpsystemwillturnDstudentsintohonorrollstudentsby

18

itself.Instead,studentsmotivatedtousethesystemwillfindtheiroverallgradesimprovingandevenmoderateusage,approximately3-4timesasemester,isenoughtoimprovestudents’lettergrades,fromhighB’sorC’s,toA’sorB’s.Thatsaid,themainrecommendationstocomefromthisanalysisofthedataareonesofoutreach.Thesystemworksforthosethatuseit,butthestudentusersarenotrepresentativeofthestudentbodyasawhole.Whites,blacks,andHispanicsarethemostcommongroupsintheHCCstudentbody,butunderrepresentedamongusers.WhilethismightberelatedtotheearlyintroductionofEnglishtutoringandusageofAskUpbyESLstudents,italsoindicatesthatthebroaderHCCpopulationmaybeunawareoftheservice.GiventherelativeeasewithwhichincreasedtrafficcanbeaccommodatedbytheAskUpsystem,particularlycomparedtootheracademicserviceoptions,increasedoutreachseemsalow-costwaytoimprovetheachievementofstudents.Inaddition,theoutreachcantargetdifferentdemographicgroups.Forexample,blackmaleshavetheoveralllowestacademicachievementwithinthesample,asindicatedbythegraphinFigure6above.Ifusageamongthisgroupcouldbeincreasedsubstantially,theoverallimprovementforthisgroupcouldbeimprovedsignificantly,bringingthemtoparitywiththestudentbodyasawhole.Whilethisanalysishasprimarilyfocusedondifferentracialandethnicgroups,achievementgapsbetweenothergroupscouldalsobeaddressed.Forexample,GEDstudentsorpart-timestudentscouldbetargetedforintensiveoutreach.WhileobviouslynoonecanbeforcedtousetheAskUpsystem,increasedusageamongdifferentgroupsshouldresultinasubstantive,moderateincreaseinGPAandothermeasuresofachievement.Intheend,thisresearchsupportsthecontinuedandevenincreasedusageoftheAskUpsystemaswellastheideathatsimple,academicsupportservicescanincreaseachievementforHCCstudents.Therelationshipbetweenserviceusageandachievementisstrongandstableacrossavarietyofstudents.ImprovementsinGPAholdtrueevenaftercontrollingforacademicpreparationandutilizingmatching,persistsacrossallminoritygroupsincluded,aswellasacrossagegroupsandgender.HCCservesastudentbodythatrepresentswhathighereducationwilllooklikeforagrowingnumberofinstitutionsinthecomingdecadesandsimple,relativelyinexpensivesupportserviceslikeAskUpcanimprovetheachievementofusersfromdisadvantagedgroups.Asmoreandmorestudentscontinuetoenrollinhighereducationatthenation’scommunitycolleges,theevidenceheresupportstheideathatonlinetutoringandsimilarsupportservicesofferedbytheseinstitutionscananddoprovidea‘gatewaytoopportunity’forcommunitycollegestudents.

19

REFERENCES

1. American Association of Community Colleges (AACC) website, accessed April 25, 2009. http://www2.aacc.nche.edu/research/index_students.htm

2. Bailey, T., & Alfonso, M. (2005). Paths to persistence: An analysis of research on program effectiveness at community colleges. Indianapolis, IN: Lumina Foundation for Education.

3. Bailey, T., and Morest, V.S. (2006). Introduction: Defending the community college equity agenda. In T. Bailey & V. S. Morest (Eds.), Defending the community college equity agenda (pp. 1 – 27). Baltimore, MD: The Johns Hopkins University Press.

4. Bailey, T., Jenkins, D., & Leinbach, T. (2005). Is student success labeled educational failure? Student goals and graduation rates in the accountability debate at community colleges. CCRC Working Paper No. 1. New York: Community College Research Center, Teachers College, Columbia University.

5. Beal, C. R., Walles, R., Arroyo, I., & Woolf, B. P. (2007). Online tutoring for math achievement: A controlled evaluation. Journal of Interactive Online Learning, 6, 43-55.

6. Cohen, P.A., Kulik, J.A., & Kulik, C.L.C .(1982). Educational outcomes of tutoring: A meta-analysis of findings. American Educational Research Journal, 19, 237-248.

7. The Department of Education National Center for Education Statistics (NCES) website, accessed April 25, 2009. http://nces.ed.gov/programs/coe/2008/analysis/sa_table.asp?tableID=1054

8. Grubb, N. (2001). “Getting into the world”: Guidance and counseling in community college. New York: Community College Research Center, Teachers College, Columbia University.

9. Grubb, N. (2006). ‘Like, what do I do now?’ The dilemmas of guidance counseling. In T. Bailey & V. S. Morest (Eds.), Defending the Community college equity agenda (pp. 195-222). Baltimore, MD: The Johns Hopkins University Press.

10. Hedges, L.V. and Majer, K. (1976). A longitudinal comparative study of a process-oriented tutorial program. (OASIS Research Report No. 5) La Jolita, CA: Office of Academic Support and Student Services, University of California, San Diego.

11. Houston Community College Website, Texas Success Initiative Plan. Accessed on April 28, 2009.http://www.hccs.edu/portal/site/hccs/menuitem.a12520d901466b1f3227a2ced07401ca/?vgnextoid=8950badfbcecf110VgnVCM2000001b4710acRCRD&vgnextchannel=4e6ebadfbcecf110VgnVCM2000001b4710acRCRD&vgnextfmt=default.

12. Iacus, S., G. King, and G. Porro, (2008), “Matching for Causal Inference Without Balance Checking,” Unpublished Manuscript, IQSS, Harvard University.

13. Irwin, D.E. (1980). Effects of Peer tutoring on academic achievement and affective adjustment. In G. Enright (ed.), Proceedings of the Thirteenth Annual conference of Western College Reading Association (WCRA), XIII, 42-45.

14. Irwin, D.E. (1981). Final statistics grade as a function of the amount of tutoring received. In G. Enright (ed.), Proceedings of the Thirteenth Annual conference of Western College Reading Association (WCRA), XIII, 55-62.

15. Karp, Melinda, Lauren O’Gara, and Katherine L. Hughes. (2008). Do Support Services at Community Colleges Encourage Success or Reproduce Disadvantage? Community College Research Center, Teachers College, Columbia University. Working Paper No. 10.

16. King, Gary, Michael Tomz, and Jason Wittenberg. "Making the Most of Statistical Analyses: Improving Interpretation and Presentation," American Journal of Political Science, Vol. 44, No. 2 (April, 2000): 341-355, http://gking.harvard.edu/files/abs/making-abs.shtml.

17. Maxwell, M. (1990). Does Tutoring Help? A look at the Literature. Review of Research Developmental Education, V7, P4, 1990.

18. McGinty, D. (1989). A path analysis of the effects of multiple programs on student retention. Unpublished doctoral dissertation, the University of Texas at Austin.

20

19. Muraskin, L. (1997). “Best Practices” in student support services: A study of five exemplary sites. Washington, DC. U.S. Department of Education, Planning and Evaluation Service.

20. Pribesh, Shana, Potter, Allison., Nunnery, John. and Ross, Steven. "Effectiveness of Supplemental Education Services Providers: A matched samples approach" Paper presented at the annual meeting of the American Sociological Association, TBA, New York, New York City, Aug 11, 2007. http://www.allacademic.com/meta/p185011_index.html

21. Rosenbaum, J., Deil-Amen, R., & Person, A. (2006). After admission: From college access to college success. New York: Russell Sage Foundation.

22. Tinto, V. (1976). “Dropout from higher education: A theoretical synthesis of recent research.” Review of Educational Research, 45, 89-125.

23. Tinto, V. (1993). Leaving college: Rethinking the causes and cures of student attrition. Chicago, IL: University of Chicago Press.

24. Zeidenberg, M., Jenkins, D., and Calcagno, J. C. (2007). Do student success courses actually help community college students succeed? CCRC Brief No. 36. New York: Community College Research Center.