KAREN SHAKMAN EVELOPMENT CENTER

LOGICMODELSTOSUPPORTPROGRAMDESIGN,IMPLEMENTATIONANDEVALUATION

AWORKBOOKCREATEDBY

THEREGIONALEDUCATIONALLABORATORY‐NORTHEASTANDISLANDS

KARENSHAKMAN

EDUCATIONDEVELOPMENTCENTER

ISBE21STCCLCPROGRAMEVALUATIONWEBINAR

APRIL15,2014

PRESENTEDBY:KATRINABLEDSOE,JOSHUACOX,LESLIEGOODYEAR

ANDSHEILARODRIGUEZ

EDC

ProgramandPolicyEvaluationWorkshop

2

TableofContents

IntroductiontoLogicModelstoSupportProgramDesign,Implementation,andEvaluation.................................................................................................................................................................................3Whatisthepurposeofthiswork?..........................................................................................................................3Whytheworkshop?......................................................................................................................................................3

LearningaboutLogicModels........................................................................................................................4AnnotatedAgenda........................................................................................................................................................4

Purpose................................................................................................................................................................6

Agenda..................................................................................................................................................................7

IntroducingtheCase........................................................................................................................................8CaseStudyBrainstorm..............................................................................................................................................10

WhatIsaLogicModel?..................................................................................................................................11TypesofLogicModels...............................................................................................................................................12TheoryApproachModels......................................................................................................................................................12ActivitiesApproachModels..................................................................................................................................................12OutcomesApproachModels.................................................................................................................................................13

Inputs‐Strategies/Activities‐Outputs‐Outcomes.............................................................................................13

ElementsofaLogicModel...........................................................................................................................15ProblemStatement.....................................................................................................................................................15Outcomes.......................................................................................................................................................................17OutcomesChecklist..................................................................................................................................................................19

StrategiesandActivities...........................................................................................................................................20AWordAboutOutputs..............................................................................................................................................22Resources(inputs).....................................................................................................................................................22Assumptions.................................................................................................................................................................24Context............................................................................................................................................................................24

TheLogicinaLogicModel...........................................................................................................................27TheTheoryEmbeddedintheModel:If‐ThenStatements...........................................................................27

NextStepsandClosingWords....................................................................................................................29FinalThoughtsonLogicModels............................................................................................................................29

AppendixA:SimpleLogicModelTemplate...........................................................................................31

AppendixB:CollegeReadySampleLogicModel.................................................................................32

AppendixB:ResourcesforAdditionalExamples................................................................................33LogicModelResources..............................................................................................................................................33EvaluationResources................................................................................................................................................33


3

IntroductiontoLogicModelstoSupportProgramDesign,Implementation,andEvaluation

Whatisthepurposeofthiswork?Beginningin2012thetenRegionalEducationalLaboratorieshaveundertakenbuildingandsupportingresearchalliancesontopicsofimportancetostatesorlocalschooldistricts.Someexamplesofthesetopicsincludeeducatoreffectivenessandsupportingallstudentstobecomereadyforcollegeandcareers.Eachofthesealliancesestablishedaresearchagendathatincludesquestionsthatwillguidetheirworkoverthenextthreetofiveyears.TheRegionalEducationalLaboratoryNortheastandIslands(REL‐NEI)hasbeeninvolvedinworkwithasetofeightresearchalliancesthatareintendedtobesustainedcollaborationsamongresearchers,administrators,policymakers,andpractitioners.Thesealliancesfocusonaparticularpriorityforthepurposeofincreasingstateandlocalcapacitytousedataandresearchtoinformdecision‐makinginthatpriorityarea.Wedevelopedthisworkshoptobuildstateanddistrictleaders’capacitytodesign,implement,andevaluateprogramsandpoliciesthataddresssomeofthemostpressingeducationalissues.

Whytheworkshop?Thisworkshopwasdevelopedtoassistgroups,likethealliancesasawholeaswellasmembersoftheallianceswithintheirowneducationalcontexts,tolearnaboutandbuildlogicmodelstosupporteffectiveprogramdesignsandevaluations.Basedonfeedbackfromalliancemembers,REL‐NEIlearnedthatmanyofourdistrict‐andstate‐basedmemberswouldliketobuildtheircapacitytodeveloplogicmodelsthatmaybeutilizedtobothevaluatetheirownprogramsaswellastoworkmoreeffectivelywithevaluatorswhomtheyengagetoconductevaluationsontheirbehalf.Thisworkshopisdesignedtoprovideaprimeronlogicmodeling,ausefultoolforprogramdesign,implementation,andevaluation,andutilizelogicmodelsasatoolforprogramevaluation.TheworkshopwaspresentedforthreedifferentREL‐NEIresearchalliancesin2013,intwodifferentformats.ThePuertoRicoDropoutPreventionResearchAllianceparticipatedinaface‐to‐face3hoursworkshopfocusedonsupportingthealliance’sefforttogenerateacommonvisionfordropoutpreventionwork.TheUrbanSchoolImprovementAllianceandtheNortheastEducatorEffectivenessResearchAlliancebothparticipatedinvirtualwebinarsforabroadaudienceofpractitionersinterestedindevelopingskillsandcapacitytodevelopandutilizelogicmodelstoincreaseindividualandgroupcapacitytodesignandevaluateprograms.


4

LearningaboutLogicModels

AnnotatedAgenda3Minutes5Minutes

WelcomeandOverviewIntroducingtheCase

Facilitatorintroductions,followedbyoverviewofagendaandpurposesofworkshop.Purposes: Tointroducelogicmodelsasaneffectivetoolfor

programorpolicydesign,implementation,andevaluation

Toreviewtheelementsofalogicmodel Todiscussthevalueoflogicmodelsfortheir

program

Facilitatorreviewsthecasethatwillbeusedasanexamplethroughouttheworkshop.Activity:Facilitatorthenintroducesinitialactivitytobrainstormthegoalsoftheprograminthecaseandthetypesofquestionsthatmightbeappropriatetoaskaboutthecase.Participantsentercommentsinvirtualchat.

15Minutes WhatisaLogicModel?

Facilitatorintroduceslogicmodelsasausefultoolforprogramdesign,implementationandevaluation.Facilitatordiscussesoverallideaofinputs‐outputs‐outcomesthatdriveslogicmodeldevelopment.Activity:Participantsrespondtomultiple‐choicequestionsaboutinputs,activities/strategies,outputs,andoutcomesfortheCollegeReadycase.

30Minutes ElementsofaLogicModel Facilitatorwillwalkthroughallelementsofalogicmodelinthissectionofthewebinar.

(1) ProblemStatement:Includesadiscussiononwhataproblemstatementisandthequestionsthatneedtobeconsideredindevelopinganappropriateproblemstatement.

(2) Outcomes:Includesdiscussionofshort‐tolong‐termoutcomesandimpacts,aswellasreviewofwhatoutputsareandhowtheydifferfromoutcomes.

(3) Strategiesandactivitiesisthenextelementof


5

themodel.Anexamplewillbegivenonwhatthestrategiesandactivitiesareforoneofthecases.

(4) Resources.Inthissectionmaterialandintangible

resourceswillbediscussed.

(5) Assumptions:Assumptionswillbebrieflydiscussed.

(6) Context:Howexternalfactorsimpactprograms

willbediscussed

7Minutes NextSteps&Closing Facilitatorclosesworkshopwithdiscussionofsomenextstepsfordevelopingalogicmodelandareminderofhowlogicmodelscansupporteffectiveprogramdesignandevaluation.Therewillbeaninvitationtobeintouchwithfurtherquestions.Activity:Participantsindicateanextstepfortheirworkinthechat.


6

Purpose ThepurposeoftheWorkshopistointroducelogicmodelsasatoolforeffectiveprogramorpolicydesign,implementation,andevaluation.Theworkshopwill: Introducelogicmodelsasaneffectivetoolforprogramandpolicydesign,implementation,

andevaluation; Reviewtheelementsofalogicmodel; Discusstheroleandvalueoflogicmodelsforyourprogram.


Page 7

Agenda3Minutes WelcomeandOverview

5Minutes IntroducingtheCase

15Minutes WhatisaLogicModel?

30Minutes ElementsofaLogicModel

7Minutes NextStepsandClosing


Page 8

IntroducingtheCase Whatfollowshereisonecase,basedonarealprogramorpolicythatweuseasanexamplethroughouttheworkshop.Thiscaseprovidesacommonlanguagefordiscussingallaspectsofthelogicmodel.CaseStudy:CollegeReadinessHighSchoolProgramCollegeReadyisaschool‐basedcollegeaccessprogramfor9th‐12thgradestudents.StudentsareidentifiedfortheprogrambasedonFreeandReducedLunchstatus,recommendationsfromschoolguidancecounselors,and/orrecommendationsfrom8thgradeEnglishandMathteachers.StudentsparticipateinmonthlymeetingsasagroupwiththeCollegeReadystaff,areprovidedwithone‐on‐onecounselingwithCollegeReadystaff,areassignedanadultmentorandapeermentor,andparticipateinaseriesofeveningandsummerworkshops.Inaddition,familiesmakeacommitmenttotheprogramandattendaseriesofworkshopsspecificallydesignedtopreparethewholefamilyforthecollegeapplicationprocess.Thegoaloftheprogramistosignificantlyincreasecollegeattendanceamongthelow‐incomestudents.Thepurposeofthiscaseandtheactivityassociatedwithitistoprovidesomecommonexamplesforuseasyouworkthroughtheexercisesintheworkshop.


Page 9

Activity: Discussion of Case

1. Individually

Directions:Workingonyourown,considertheCollegeReadycaseabove.Whatarethegoalsoftheprogram?Whatmightwewanttoknowaboutit?

2. LargeGroup“Discussion”

Directions:Whatareyourideasaboutthegoalsoftheprogramandwhatyoumightwanttoknowaboutit?


Page 10

CaseStudyBrainstormWorkindependentlyfortwominutestolistthepotentialgoalsandquestionsthatwemighthaveabouttheprogram.Considerquestionsofimplementation,effectiveness,andimpact.Whatarethegoalsoftheprogram/policy? Whatdowewanttoknow abouttheprogram/policy?


Page 11

WhatIsaLogicModel? Inthissectionoftheworkshopweprovideaprimeronlogicmodels,includingdifferenttypesoflogicmodelsandtheirpotentialpurposes.Inmostbasicterms,logicmodelsprovideakindofmapforaprogramorinitiative,helpingtoclarifyaprogramorpolicy’sdestination,thepathwaystowardthedestinationandmarkersalongtheway.Consider: Whereareyougoing? Howwillyougetthere? Whatwilltellyouyou’vearrived?

Logicmodelsprovideasimplifiedpictureoftherelationshipsbetweentheprogramorpolicyinputs(resources,strategies,activities)andthedesiredoutcomesoftheprogram.Logicmodelspresentatheoryofactionorchangethatdrivestheprogramorpolicy,andmakesexplicitanyassumptionsaboutboththeresourcesatthedisposaloftheprogramaswellastherationalebehindtheeffort.Alogicmodelisvaluableinsupporting: ProgramPlanning ProgramImplementation ProgramMonitoring ProgramEvaluation

Whyusealogicmodel?Alogicmodel: Bringsdetailtobroadgoals; Helpsidentifygapsinprogramlogicandclarifyassumptions; Buildsunderstandingandpromotesconsensus; Makesexplicitunderlyingbeliefs; Helpsclarifywhatisappropriatetoevaluateandwhen; Summarizescomplexprogramsforeffectivecommunication.

Alogicmodelisveryusefulindesigningprogramandpolicyevaluation,asalogicmodelhelpstoclarifybothwhattheprogram,initiative,orpolicyisandwhatitisn’t.Thiskindofclarificationisveryhelpfulinbuildinganevaluationdesignthatcancapturetheprogram’sorpolicy’sinfluenceandimpact.Whatarethelimitationsofalogicmodel?


Page 12

Alogicmodelisnotafullydevelopedplanfordesigningormanagingaprogramorpolicy; Alogicmodelisnotanevaluationplan.

Whilelogicmodelsareusefultoolsforbuildingprogramplansorevaluationdesigns,additionalworkisnecessarytocreatebothprogrammaticandevaluationplans.

TypesofLogicModelsNotalllogicmodelsarethesame,noraretheydesignedforthesamepurpose.Justaslogicmodelsmayaidinprogramdesign,implementation,andevaluation,thetypeofmodeldevelopedvariessomewhatbasedonitspurpose.Therearethreemaintypesoflogicmodels: TheoryApproachModel ActivitiesApproachModel OutcomesApproachModel

Theory Approach Models Logicmodelsthatdescribetheoveralltheoryofchangeprovidea“bigpicture”oftheprogramandmaybeusefulforprogramdesignandoverallcommunicationoftheprogramtheory.Thesemodelsprovideacleardescriptionofwhythedevelopersbelievetheprogramorpolicywillbeeffectiveinachievingthegoals.Forexample,intheexampleofcollegereadiness,atheoryapproachlogicmodelmighthelpclarifytheassumptionsimplicitinthepushforprogramsforlow‐incomestudentsandfamilies,anddescribetherelationshipbetweeninitiatingthecollegegoingprocessandtheexpectedoutcomesforstudentswhoparticipate.Consider: Whatmightbethelogicinatheoryapproachmodelforyourownprogramorpolicy?(The

“bigpicture”theoryofchangeaboutyourinitiative?)

Activities Approach Models TheActivitiesApproachmodelismostfocusedonlayingoutthespecificstrategiesandactivitiesassociatedwithaprogram.Thesemodelscloselyexaminetherelationshipamongtheactivities,consideringquestionsofsequenceandtimingofimplementation,aswellashowtheseactivitieslinktooutcomes.Thistypeoflogicmodelismostusefulinprogramimplementation,monitoring,andmanagement.IntheCollegeReadyexample,thistypeoflogicmodelwouldconsiderthedifferentelementsoftheprogramandhowtheywouldbeoptimallyorderedandmanaged.Forexample,whatrolewouldthedifferentmentoringcomponentshave?Howwouldtheyrelatetooneanother?Inthistypeofmodel,relationshipsamongvariablesaremadeexplicitwitharrows,concentriccircles,andothergraphicrepresentationsofrelationships.Consider: Whyconsiderthesequenceandrelationshipamongactivitiesinalogicmodel?Howmight

thathelpyou?


Page 13

Outcomes Approach Models OutcomesApproachmodelsaremostusefulforprogramevaluation.Theyconsiderthestrategiesandactivitiesastheyrelatetothedesiredresultsofaprogramorpolicy.Inthesemodels,thefocusisonoutcomes,andtheyoftendividetheoutcomesintoshort‐termandlong‐termoutcomes,andimpacts.Atheoryofchangedrivesthesemodelsjustasitdoestheothers.However,inanoutcomesapproachlogicmodel,theemphasisisonexaminingtheoutcomesandmakingthecasethattheprogramorpolicyisresponsibleforthedesiredoutcomes.Consider: Whydivideoutcomesintoshort‐andlong‐term?Whatisthedifferencebetweenoutcomes

andimpacts?

Inputs‐Strategies/Activities‐Outputs‐OutcomesInitssimplestform,alogicmodelisagraphicrepresentationoftherelationshipamongaprogram’sorpolicy’sinputs(what’sinvestedintheprogram);thestrategies/activities;theoutputs(what’sdonewiththeseinvestments);andwhattheoutcomesare(whatresults).Takeaverysimpleexample:youhaveaheadacheandyouwantittogoaway. Whatistheinput?

o Quiettimeo Watero Ahotcompresso Twoaspirin

Whatisthestrategy/activity?(e.g.whatdoyoudowiththeinputs?)

o Sitquietlyfor5minuteso Drinkafullglassofwatero Puthotcompressono Takeaspirin

Whatistheoutput?

o Youarehydratedo Youarestartingtofeelmorerelaxed

What’stheoutcome?

o Youarerelaxedo Yourheadachegoesawayo Youareabletoreturntoyourworko Youarehappy


Page 14

Activity:Inputs‐Strategies/Activities‐Outputs‐Outcomes Nowtakethecaseexampleabove.Considertheinputs,strategies/activities,outputs,andoutcomes.Directions:ReviewtheCollegeReadycaseandincludethepossibleinputs,strategies/activities,outputs,andoutcomesbelow.

Case:___________________________________________

Inputs

Strategies/Activities Outputs Outcomes


Page 15

ElementsofaLogicModelThepurposeofthissectionistoprovideanoverviewof—andopportunitiestopractice—allelementsofalogicmodel.Theelementsofalogicmodelarelistedbelow.Inthepagesthatfollow,wewilldelineatetheseelementsanduseexamplesfromthecase. ProblemStatement Resources(inputs) StrategiesandActivities Outputs Short‐termOutcomes Long‐termOutcomes Impact Assumptions Context

ProblemStatementTheproblemstatementistheproblemorchallengeyoufacethattheprogramorpolicyisdesignedtoaddress. Consider:

o Whatistheproblem/issue?o Whyisthisaproblem?(Whatcausestheproblem?)o Forwhom(individual,household,group,community,societyingeneral)doesthis

problemexist?o Whohasastakeintheproblem?(Whocareswhetheritisresolvedornot?)o Whatdoweknowabouttheproblem/issue/peoplethatareinvolved?Whatresearch,

experiencedowehave?Whatdoexistingresearchandexperiencesay?

Finally,askyourself:istheproblemtoobig?Isittoosmall?Yourfinalproblemstatementshouldbetargetedandspecific,butitshouldnotbeasimplerestatementoftheprogramasaneed.Forexample,inthecollegereadycase,youwouldnotwanttheproblemtoread:“Lowincomestudentsarenotattendingcollegeincomparisontotheirmiddleclasspeers”Inthisexample,theproblemisreallyastatementofthelackoftheprogram.Rather,youwanttheproblemtoaddresstherealissuesunderlyingtheneedfortheprogram,suchas“studentslackknowledgeonthecollegegoingprocess”.


Page 16

Activity: Problem Statement Considertheproblemstatementmostappropriatetothechallengeyoufaceinyourwork,relatedtoaprogramyouhaveinplaceoroneyouwouldliketoinitiate.Brainstormkeyideasassociatedwiththerelevantproblem.Modelyourbrainstormaftertheexamplebelow,intermsofbrevity.Example:CollegeReadyCase

o Lowincomestudents’collegegoingratesareverylowo Lowincomestudentshavemorelimitedopportunitieso Therearehigherratesofunemploymentamonglowincomeworkerso Lowincomeworkersearnlesswagesthroughouttheirlifetimethantheirmiddleclass

counterparts.

YourBrainstorm:____________________


Page 17

OutcomesWhileoutcomesarenotthenextitemoneseeswhenonelooksfromlefttorightacrossatraditionallogicmodel,theyarealogicalnextsteptodiscusswhenexaminingtheelementsofalogicmodel.Outcomesask,“Whatdifferencedoesitmake?”Inotherwords,whatisthedifferencethattheresources,andstrategiesandactivities,takentogether,haveonthevariousparticipantsintheseefforts?Outcomesusuallycomeinstagesandfallalongacontinuumfromshort‐tolong‐termoutcomes.Thelanguagetodescribetheseoutcomesvaries,butforthepurposesofthisworkbook,weuseshort‐termoutcomes,long‐termoutcomes,andimpact.Othertermsyoumayencounterinclude:

o Short‐term:initial,immediate,proximalo Long‐term:medium‐term,intermediate,midpointo Impact:long‐term,final,ultimate,distaloutcome

Short‐termoutcomes:Short‐termoutcomesarethemostimmediateandmeasurable

resultsforparticipantsthatcanbeattributedtothestrategiesandactivities.Forexample,aprogramthatpromotesincreasedparentengagementinstudents’collegeplanningmighthaveashort‐termgoalofincreasedparentalparticipationintheprovidedparentsessions.

Long‐termoutcomes:Long‐termoutcomesarethemoredistant,thoughanticipatedresultsofparticipationinthecollectionofstrategiesandactivities.Whenitcomestoshort‐andlong‐termoutcomes,itisgoodtothinkabouttheoveralltimeframefortheprogram.Sometimes,short‐termisconsideredtobeasshortas6monthsoraslongas3years.Long‐termmightbe2yearsoraslongas6years.Theimportantpointhereistoconsidertheprogramandidentifythetimeframe,specifictotheinitiative,forshorter‐andlonger‐termoutcomes.

Impact:Whenweusetheterm“impact”,wemeanthedesiredoutcomesthatoccurasa

resultoflong‐termimplementationofthestrategiesandactivities.Thesemorelong‐rangegoalsaredependentonsomeconditionsthatgobeyondtheprogram’sscopeofstrategies.Thesemaybecalledthe“blueskies”orthebigpicturetypesofobjectivesfortheprogramandonesthataremoredistantfromtheactualstrategiesandactivities,andlesswithinthecontroloftheprogramorpolicytorealize.Oftentheseareconsideredtobe7‐10yearsoutfrominitialimplementation.


Page 18

Activity: Focus on Outcomes Beingclearaboutprogramoutcomesisessentialforbothfocusedprogramimplementationandforeffectiveevaluation.Thetablebelowisdesignedtopromoteastep‐by‐stepapproachtooutcomedevelopment.Thecolumnsare: Whoisthetarget?Whoisthegroupyouaretargetingwithyourstrategy?Isitstudents?

Parents?Aschool?Inthisexample,basedonthecollegereadycase,thetargetisparticipatinghighschoolseniorsinthreehighschoolsthatparticipateintheprogram.

Whatisthechangedesired?Heretheimportantideaistouseanactionverbtodemonstrateakindofchangeoranimpact.Forexample:increase,improve,engage…

Inwhat?Whatistheactivity,strategy,orprograminwhichthetargetpopulationisgoingtoenactthisdesiredchange?Whatistheresultingactioninwhichthetargetpopulationwillengagetoachievethegoal?

Bywhen?Hereiswhereyoubegintoclarifythetimelineforoutcomes.Isaparticularoutcomeashort‐termorlong‐termoutcome?

Directions:Foreachofthecolumnheaders(thetarget,changedesired,etc.),enteranexamplefromyourowncontext,relatedtoaprogramorpolicyinitiativeyouhaveinplaceorwouldliketodevelop.TheTarget Changedesired?(action

verb)Inwhat?(results) Bywhen?

Participatinghighschoolseniorsinthreeurbanhighschools

Increase Completedandsubmittedapplicationstopost‐secondaryinstitutions

ByJune2014


Page 19

Outcomes Checklist Considerthefollowingcriteriawhenexaminingyouroutcomes: Aretheoutcomesimportant?Aretheendoutcomesimportant?Dotheyrepresent

significantchangeorimprovementsthatarevaluedbyparticipantsandkeystakeholders?Outcomesmaybeachievablebutnotreallyworththeeffort.Applythe"Whocares?"test.

Aretheoutcomesreasonable?Aretheoutcomes(fromshort‐termtolong‐termtoimpact)connectedtooneanotherandlinkedinareasonableorder?Isitlikelythatonewillleadtothenext?Anotherwaytothinkaboutthisistoconsiderthe“if‐then”statements(orlogicstatements)embeddedinachainofoutcomes.Forexample,usingtheCollegeReadyexample,willincreasedparentparticipationinworkshopsoncollegereadinessleadtostudentscompletingmorecollegeapplications?Willmentoringstudentsincreasematriculatingincollege?Issuesofsequenceandtimingofactivitiesandintendedoutcomesisimportanttoconsider.

Aretheoutcomesrealistic?Aretheoutcomesthatyousuggestrealisticgiventhenatureoftheproblem,yourresources,andyourabilities?Willtheprogramleadtoorhelpcontributetotheseoutcomes?(Becarefultoensurethattheoutcomesarerealisticgiventhelevelofeffort.)Inotherwords,ifyoudeliveroneparenteducationclass,isitrealistictoexpectanincreaseinstudentachievement?Askhardquestionsabouttheoutcomesastheyrelatetotheactualprogramorpolicy.

Areyouattendingtounintentionalorpotentiallynegativeoutcomes?Finally,it’simportanttoanticipateandconsidertheunintendedorpotentiallynegativeoutcomesthatmightresultfromthesetofstrategiesandactivities.Whatarepotentialnegativeeffectsoftheprogramorpolicy?Whatelsemighthappenthatisdifferentfromwhatweintend?Or,howelsemightthesequenceofeventsunfold?Forexample,couldthereadinessprogramleadtostudentsnotapplyingtocollege?Consideringtheunintendedconsequencesallowsprogramandpolicydesignerstoconsiderhowtoprepareforthesepossibleoutcomes,andalsohelpsevaluatorstobeattunedtothesepossibleconsequencesintheevaluationdesign.


Page 20

StrategiesandActivitiesStrategiesandactivitiesaretheprogramcomponents,orthegameplanfortheprogramorpolicy.Thisisaninventoryofallthestrategiesandactivitiesdesignedtoachievetheoutcomes.However,itismorethanasimplelistingofactivities.Therearetwoquestionstoaskyourselfwhenyouinventorytheactivities,services,products,andeventsthatmakeuptheprogramorpolicy: Whatistheappropriatesequenceororderoftheseactivities?

ConsidertheCollegeReadycase:Itmaybeimportantthatthementoringelementoftheprogramcomepriortothedeliveryoftheparentworkshopseries.Orperhapstheseactivitiesshouldbeconcurrent.Considertheappropriateorderofactivitiesandhowtheyrelatetooneanother. Aretherecertainactivitiesthat,takentogether,adduptoakindofoverallstrategy?Do

certainactivities“bundle”or“cluster”together?

ConsidertheCollegeReadyexample:Perhapsthereareaseriesofworkshopsrelatedtothecollegegoing,suchashowtoappropriatelycompletecollegeapplications,fillingouttheFAFSA,andeducatingparentsontheprocessandwhatisrequiredofthem,thatbundletogetherasanoverarchingstrategy.Perhapsthisisthe“StudentandParenteducation”strategy.Thismaybedifferentfromotherstrategiesassociatedwiththeinitiative,suchasmentorshiporfamilyengagement.Creatingtheseclustersofactivitieshelpstostreamlinethelogicmodelandalsosupportsevaluation;theevaluationwillthenassessasetofstrategies,ratherthanindividualactivities.


Page 21

Activity: Strategies and Activities in Sequence Consideraseriesofactivitiesthatareapartofyourownwork.Listsomerelevantactivities,theirsequenceororderinwhichtheyaresupposedtooccur,andconsidertheoverarchingstrategywithinwhichtheseactivitiesfall.Inotherwords,doesyourchosenprogramorinitiativehaveacoresetofstrategiesthatguidetheactivities,events,programs,etc.thatyouprovide?(Note:thewebinarwillnotincludethisactivitybutthisissomethingyouareencouragedtodoonyourown.)Activities Sequence StrategyDevelopworkshopmaterialsforstudentreadinessprogram

1st Studentandparenteducation

Deliverstudentreadinessworkshops

2nd Studentandparenteducation

Recruitcommunitymentors

1st Mentorshipdevelopment

Activities Sequence Strategy


Page 22

AWordAboutOutputsSomelogicmodelsincludebothoutputsandoutcomesinthemodel.Outputsdifferfromoutcomesinthattheycapturedataaboutwhatwedoratherthanwhatweexpecttoachieveasaresultofwhatwedo.Outputscanbestbedescribedasactivitydataandareusefulfortrackingprogramimplementation.Outputsoftenprovidedetailaboutthebreadthandreachofthestrategiesandactivities.Outputscapturesizeandscope;theydescribeorcountstrategiesandactivities,suchasthenumberofparentsessionsdelivered,programparticipationrates,thenumberofmaterialsdevelopedordistributed,andsoforth.UsingtheCollegeReadyprogramasanexample,anotherwaytothinkaboutthedifferencebetweenoutputsandoutcomesistoconsiderthequestions:

Istheparenteducationprogrambeingdeliveredasintended?(outputquestion)

vs.Isthecollegeacceptancerateforparticipatingstudentsincreasing?(outcomequestion)

Onefinalword:itisimportantnottoconfuseoutputsforoutcomes.Aprogramthatisgoodatdeliveringactivitiesandservicesmayachieveitsoutputswithoutachievingitsoutcomes.Yet,itistheoutcomesthatmakethedifferenceinresponsetotheproblemidentified.

Resources(inputs)Resourcesincludeboththematerialandtheintangiblecontributionsthatareorcouldreasonablybeexpectedtobeavailabletoaddresstheproblem. Materialresourcesinclude:

o Moneyo Materialsandequipment

Intangibleresourcesinclude:

o Peopleo Timeo Partnershipso Other?


Page 23

Activity: Intangible Resources Consider:Whataretheintangibleresourcesatyourdisposal?Brainstormatleast5non‐monetaryresourcesthatareavailabletoyouinaprogramyouoperateormanage.Brainstorm:IntangibleResources(examplefromCollegeReadycase)

o Communitymentorso Localuniversityspaceforparentmeetingso Volunteercollegeadmissionsdirectorsforapplicationworkshopo Studentvolunteersforchildcareatparentmeetings

Brainstorm:IntangibleResources


Page 24

AssumptionsAssumptionsarethebeliefsweholdaboutparticipants,staff,andtheprogram,aswellasourassumptionsabouthowthechangeorimprovementwehopetoseemayberealized.Beingexplicitabouttheseassumptionsisoneofthefirstandmostimportantthingsyoucandoasyouconsiderprogramdesign,implementation,andevaluation.ConsidertheCollegeReadycase:theprogramassumesthatstudentswhoparticipateintheprogramwanttogotocollege,andfurtherassumesthatcollegeenrollmentwillleadtoabetterlifefortheirparticipants.Oftentheassumptionsembeddedinaprogramorpolicyiscriticaltothesuccessorfailureoftheoverallinitiative.Assumptionsmaybeinternal(assumptionsaboutparticipants,resources,andhowtheprogramwillfunction)orexternal(beliefsabouthowchangeoccurs,valuesembeddedintheprogram,orfindingsfrompriorresearch).Let’suseasimpleexample.Remembertheheadache?Youhadaheadache,youtriedafewthingstogetridofit(water,aspirin,etc.)andyoufeltbetter.Theoutcomewasthattheheadachewentaway.However,betweentheproblem(theheadache)andtheoutcome(noheadache)wereseveralassumptions.Forexample,youassumednoallergytoaspirin,thattherewasnoloudnoisepersistinginthebackground,andsoforth.Clarifyingandmakingexplicittheassumptionsbehindtheprogram,bothintermsofthespecificelementsrelatedtoimplementationandtheassumptionsembeddedinthetheorydrivingtheinitiative,iscriticaltothedevelopmentofathoughtfullogicmodel.

ContextContextisthesetofcircumstances,environments,policiesorfactsthatsurroundaparticularevent,situation,etc.Theseincludebutarenotlimitedto,history,culture,climate,socialstrata,region,andpoliticalcircumstances.Contextcandeterminehowtheprogramwillbeimplemented,howitwilloperate,howprogramparticipantswillreceiveit,andhowitwillbesustained.Let’sreturntotheCollegeReadycase.Theprogram’smainclientelearethosestudentswhoreceivefreeorreducedlunch,sosocio‐economicstatuswilllikelyeffecthowparentsandstudentsparticipate.Forinstance,iffamiliesarelowincome,theymighthavemultiplejobsthatmightimpactparticipation.Thisfactormightaffectthetimesofdaythattheprogramisoffered,aswellastheexpectationsforfamilyparticipation.Considertheaspirinexample.Itmightbeagoodideatotakeaspirintorelieveaheadache,butwhatisthecultureinyourhouseholdabouttakingtheoverthecountermedication?Isthisanormwithinyourfamily?Ifnot,youmaynotmakeuseofthatstrategytorelieveyourheadache.Whatstrategiesareculturallyappropriate?Clarifyingthecontextoftheprogramcanhelparticulateunderwhatcircumstancestheprogramwilloperateandbesuccessful.


Page 25

Activity: Uncovering Internal and External AssumptionsDirections:Consideryourprogramorinitiative.Brainstormtherangeofassumptionsembeddedintheprogramdesignandintheoveralltheoryofactiondrivingtheinitiative.InternalAssumptions ExternalAssumptions


Page 26

Activity: Identifying Contextual FactorsDirections:Consideryourprogramorinitiative.Brainstormexternalfactorsthatinfluencehowtheprogramisadministeredandhowitoperates.ContextualFactors HowOurProgramOperatestoAddressThem


Page 27

TheLogicinaLogicModelThepurposeofthissectionistounderstandthelogicembeddedinlogicmodelsandrecognizetheneedtoidentifythepossiblerelationshipsandconnectionsamongvariouselementsofthelogicmodel.

TheTheoryEmbeddedintheModel:If‐ThenStatementsUnderstandingtheseif‐thenrelationshipsisessentialtouncoveringthetheoryofactionortheoryofchangedrivingaprogramorpolicy.Ifyouhaveaccesstoandapplyresources,youwillbeabletodevelopprogramsthatwillbedesignedtoreachthetargetparticipants,andwhenyoureachthesepopulationswiththeprogramsorservicesthatyouhavedeveloped,thenyouwillbeabletomeettheunmetneedsandchangecircumstancesthatwillleadtosolvingtheproblemthatinitiatedthiswork.Tomakethismorereal,let’sconsidertheCollegeReadycase:

Ifwedevelopaseriesofstudentworkshopsfocusedonthecollegegoingprocess. Ifwedevelopaseriesofstudentworkshopsfocusedonthecollegegoingprocess,then

studentsmayattend. Ifstudentsattend,thentheywillhavemoreknowledgeonwhatittakestoapplytocollege. Iftheyhavemoreknowledgeonwhatittakestoapplytocollege,theymaybemorewillingto

considercollegeasanoption. Ifconsidercollegeasanoption,collegeapplicationsincrease.

Notethatthethenclauseinonestatementbecomestheifclausetheproceedingstatement.Thisisimportant;whenwechangethelanguagefromthethentotheifstatement,changesintheintentionofthestatementmayoccur.Insomelogicmodels,theseif‐thenstatementsarewrittenrightintothemodeltomakethetheoryofchangeexplicit.


Page 28

Activity: If‐Then Statements Directions:ConsidertheCollegeReadycase.Movethestatementsaroundtomakeaseriesoflogicalif‐thenstatementsbelow.Considerthesequencingofevents.Thestatementsbelowincludestrategiesandactivities,traditionaloutputs,andoutcomes.IF______________________________________________THEN/IF____________________________________________THEN/IF_______________________________________THEN/IF___________________________________________

THEN/IF_______________________________________THEN______________________________________________(1)Wedevelopaseriesofcollegereadinessworkshopsforparents.(2)Parentshelptheirstudentswiththeapplicationprocess.(3)Werecruitparentstoparticipateintheworkshops.(4)Parentsbetterunderstandthetimelinesanddemandsofthecollegeapplicationprocess.(5)Studentsmeetfinancialaidandcollegeapplicationdeadlines.(6)Parentsattendtheworkshops.


Page 29

NextStepsandClosingWordsInbuildingalogicmodel,itisimportanttoconsiderthefollowingquestions: DoIunderstandthedifferentelementsofthelogicmodelandhowtheydiffer? WhoshouldIconsultinthedevelopmentofthelogicmodel?Whatcolleaguesandstakeholders

shouldbeparticipantsinthedevelopmentofthelogicmodel? Whowillberesponsibleforseeingthisthrough? HowdoIknowIhavecapturedthetheoryofactionguidingtheprogram? Howwillweusethelogicmodelonceit’sdeveloped?

Activity: Discussion Directions:Takeamomenttoconsiderhowyoucanuselogicmodelsinyourwork.

Whoisusinglogicmodelsaroundyou? Howaretheybeingused? Whatmakesthemuseful?Whatdoyouseeintheoverviewoflogicmodelsherethatmightbe

helpfulorrelevanttoyourwork? Whatmoredoyouneedtomakethemusefulandeffectiveinyourwork?

FinalThoughtsonLogicModelsHereareafewquickremindersaboutwhatalogicmodelis,andwhatitisn’t.Alogicmodelis: Agraphicrepresentationofthetheoryofchangedrivingaprogramorpolicy; Aframeworkforplanning,implementation,andevaluation.

Alogicmodelisnot: Astrategicplan; Anevaluationdesign.

Whilealogicmodelisnotastrategicplanoranevaluationdesign,itcanbeusefulindevelopingeitherofthesemoredetailedresources.Alogicmodelislikelytobemuchmoreeffective,useful,andhonestiftheprocessofgeneratingthelogicmodelhasengagedabroadrangeofstakeholdersinthedesignprocess.Includingkeyvoicessuchasstaff,parents,students,funders,andothers,indiscussionsaboutprogramdesignandevaluationthatwillpromotethebuy‐inandongoingsupportoftheseparticipantsaswellasincreasetheauthenticityofthemodel.Logicmodelsshouldbelivingdocumentsthatarereferredtothroughoutthelifeoftheprogramandtheevaluation,andamendedasneeded.Theyarealsohelpfultoguideaprogramasitevolves,andensurethattheworkoftheprogramremainsfocusedonthekeygoalsandoutcomes.


Page 30

Logicmodelsareusefulforprogramevaluation,especiallywhenevaluationisconsideredinconcertwithcreatingthelogicmodelattheearlystagesofprogramdevelopment.Itismuchbettertoconsiderevaluationattheoutsetofaprogramorpolicy’sdevelopmentratherthanasanafterthoughtorhalfwaythroughprogramimplementation.Goodluckwiththisworkandpleasecontactuswithquestions!KatrinaBledsoe,EducationDevelopmentCenter,[email protected],EducationDevelopmentCenter,[email protected],EducationDevelopmentCenter,[email protected],EducationDevelopmentCenter,[email protected]


Page 31

AppendixA:SimpleLogicModelTemplateProblemStatement:Resources/Inputs Strategiesand

ActivitiesOutputs Short‐term

OutcomeLong‐termOutcomes

Impacts

Whatresourcesareorcouldreasonablybeavailable?

Whatwilltheactivities,events,etc.be?

Whatare theinitialproductsoftheseactivities?

Whatchangesareexpectedinshort‐term?

Whatchangesdoyouwanttoaftertheinitialoutcomes?

Whatarethehopedforchangesoverthelonghaul?

Assumptions:Context:


Page 32

AppendixB:CollegeReadySampleLogicModelProblemStatement:Low‐incomehighstudentsinselectedcommunitiesattendcollegeatalowerratethantheirmiddleclasspeers,leadingtomorelimitedopportunities,higherratesofunemployment,andlowerearnings.Resources/Inputs Strategiesand

ActivitiesOutputs Short‐term

OutcomeLong‐termOutcomes

Impacts

Whatresourcesareorcouldreasonablybeavailable?

Whatwilltheactivities,events,etc.be?

Whataretheinitialproductsoftheseactivities?

Whatchangesareexpectedinshort‐term?

Whatchangeswantedafterinitialoutcomes?

Whatarehopedforchangesoverlonghaul?

‐Partnershipwith3publichighschools‐Communitymentors‐Localuniversityspaceforparentmeetings‐Volunteercollegeadmissionsdirectorsforapplicationworkshop‐Studentvolunteersforchildcareatparentmeetings

‐Localcollegementorshipprogram‐Peermentors‐Studentreadinessprogram(workshops)‐Parenteducation(workshops)

‐Recruitadequate#ofmentorsforstudentcohort‐Developanddeliver12workshopsoncollegeapplicationprocess;SAT/ACT;FAFSA;collegelife‐Developanddeliver6workshopsforparents‐Highinterestandattendanceatallworkshopsforparentsandstudents.

‐Participatingstudentsapplytoatleastonecollegeontime‐Parentsreportincreasedunderstandingofthecollegeapplicationprocess‐Studentsreportincreasedreadinessforcollege‐ParticipatingstudentscompleteFAFSAformsontime

‐Participatingstudentsareacceptedtoandattendcollege,remainingenrolledintothe3rdsemesterofcollege‐ParticipatingstudentsGPAsabove3.0atcollege,intothe3rdsemester‐Increasedparentalengagementinparticipatinghighschools’studentseducation

‐Low‐incomestudentsinparticipatingcommunitiesattendcollegeatsamerateasmiddleclasspeers‐Low‐incomestudentsinparticipatingcommunitiesgraduatefromcollegeatsomerateasmiddleclasspeers‐Participatinghighschoolsseeincreaseinparentandstudentengagement‐Participatinghighschoolsstatetestscoresincreasebyx%

Assumptions:Collegeattendanceisdesiredgoalforparticipatingcommunities;highschoolleaderswillremainconsistentandsupportprogram;parentswillshowinterestandparticipateinprogram.Context:Collegepreparationisafamilyaffair;Targetingstudentsoflowsocio‐economicstatus;previouscollegereadinessprograms.


Page 33

AppendixB:ResourcesforAdditionalExamplesTherearemanyterrificresourcesavailableonlineforlogicmodelingandprogramandpolicyevaluation.Manyofthesewereusedinthedevelopmentofthisworkshop.Severaloftheresourcesbelowprovidelinkstoadditionalresources,alsoavailableonline.

LogicModelResourcesUniversityofWisconsinExtensionSchool:http://www.uwex.edu/ces/lmcourse/#TheUniversityofWisconsinExtensionSchoolhasseveralusefulresourcesrelatedtologicmodelsandevaluation.W.K.KelloggFoundationLogicModelDevelopmentGuide.Retrieved3/16/2013fromhttp://www.wkkf.org/knowledge‐center/resources/2006/02/wk‐kellogg‐foundation‐logic‐model‐development‐guide.aspxHarvardFamilyResearchProjectandCoffman,J.(1999).LearningfromLogicModels:AnExampleofaFamily‐SchoolPartnershipProgram.Retrieved3/16/2013fromhttp://www.hfrp.org/publications‐resources/browse‐our‐publications/learning‐from‐logic‐models‐an‐example‐of‐a‐family‐school‐partnership‐program InnovationNetwork,Inc.(dateunknown).LogicModelWorkbook.Retrieved3/18/2013fromhttp://www.innonet.org/client_docs/File/logic_model_workbook.pdf

EvaluationResourcesTheAdministrationforChildrenandFamilies,OfficeofPlanning,Research,andEvaluation(2010).TheProgramManager’sGuidetoEvaluation,SecondEdition.Retrieved3/16/2013fromhttp://www.acf.hhs.gov/programs/opre/resource/the‐program‐managers‐guide‐to‐evaluation‐second‐editionBond,S.,Boyd,S.,andRapp,K.(1997).TakingStock:APracticalGuidetoEvaluatingyourownPrograms.HorizonResearch,Inc.Retrieved3/16/2013fromhttp://www.gao.gov/products/GAO‐12‐208Ghttp://www.horizon‐research.com/reports/1997/taking_stock.phpCentersforDiseaseControl(August,2012).ImplementingEvaluation:LearningandGrowingThroughEvaluation.Retrieved3/16/2013fromhttp://www.cdc.gov/asthma/program_eval/guide.htm


Page 34

Chinman,M.,Imm,P.,andWandersman,A.(2004).GettingToOutcomes2004:PromotingAccountabilityThroughMethodsandToolsforPlanning,Implementation,andEvaluation.RandCorporation:SantaMonica,CA.Retrieved3/16/2013fromhttp://www.rand.org/pubs/technical_reports/TR101.html

Patton,M.Q.(1997).Utilization‐FocusedEvaluation:TheNewCenturyText.SAGEPublications:ThousandOaks,CA.UnitedStatesGovernmentAccountabilityOffice(January2012).DesigningEvaluations.Retrieved3/16/2013fromhttp://www.gao.gov/products/GAO‐12‐208GW.K.KelloggFoundationEvaluationHandbook.Retrieved3/16/2013fromhttp://www.wkkf.org/knowledge‐center/resources/2010/w‐k‐kellogg‐foundation‐evaluation‐handbook.aspxTheCDChasarangeofevaluationresources,someproducedattheCDCandsomefromothersources:http://www.cdc.gov/eval/resources/index.htmThesechecklists,createdbyTheEvaluationCenterattheWesternMichiganUniversitymaybeusefulinplanningandmonitoringevaluation:http://www.wmich.edu/evalctr/checklists/Theauthorwouldalsoliketoacknowledgethefollowingpeoplefortheirexpertiseandresourcesindevelopingthesematerials:KatrinaBledsoe,EducationDevelopmentCenter,Inc.LeslieGoodyear,EducationDevelopmentCenter,Inc.BrianLord,EducationDevelopmentCenter,Inc.AnneWang,EducationDevelopmentCenter,Inc.SheilaRodriguez,EducationDevelopmentCenter,Inc.

KAREN SHAKMAN EVELOPMENT CENTER

Documents

Transcript of KAREN SHAKMAN EVELOPMENT CENTER