Report from the Core Council to the Faculty Senate on Core...
14
Report from the Core Council to the Faculty Senate on Core Assessment 2015- 2016 Table of Contents Introduction ………………………………………………………………………. 2 What We Learned About Core Assessment in 2015-2016 …….... 3 Issues That Need Further Attention ………………………………………. 4 Recommendations for the Future ………………………………………….. 5 Appendix A - Description of UALR Core…………………………………... 7 Appendix B- Charge of the Core Council ………………………………….. 9 Appendix C-Progress on Core Assessment for Academic Year 2014-2015 and Fall 2015 ……………………………………………………… 10 Appendix D- Communication Structure for the CAACs, Liaisons, And Core Council ……………………………………………………………………. 11 Appendix E- Outline of Core Assessment Plan for 2015-2016 ……… 12 Appendix F-Liaison Reports ……………………………………………………….. separate file Appendix G - CAAC Reports ……………………………………………………….. separate file Appendix H- Timeline for Future Assessment …………………………….. separate file 1
Transcript of Report from the Core Council to the Faculty Senate on Core...
ReportfromtheCoreCounciltotheFacultySenateonCoreAssessment2015-2016
TableofContents
Introduction ………………………………………………………………………. 2WhatWeLearnedAboutCoreAssessmentin2015-2016…….... 3IssuesThatNeedFurtherAttention………………………………………. 4RecommendationsfortheFuture………………………………………….. 5AppendixA-DescriptionofUALRCore…………………………………... 7AppendixB-ChargeoftheCoreCouncil…………………………………..9AppendixC-ProgressonCoreAssessmentforAcademicYear2014-2015andFall2015……………………………………………………… 10AppendixD-CommunicationStructurefortheCAACs,Liaisons,AndCoreCouncil…………………………………………………………………….11AppendixE-OutlineofCoreAssessmentPlanfor2015-2016……… 12AppendixF-LiaisonReports………………………………………………………..separatefileAppendixG-CAACReports………………………………………………………..separatefileAppendixH-TimelineforFutureAssessment……………………………..separatefile
1
ReportfromtheCoreCounciltotheFacultySenateonCoreAssessment2015-20165-6-16
MembersoftheCoreCouncilRebeccaStreettandBrianMitchell,CALSElizabethSloanDavidsonandLindaStauffer,CEHP NickJovanovicandMichaelDeAngelis,CEITMarkFunkandJoeFelan,COBBelindaBlevins-KnabeandTustytenBensel,CSSC
Introduction
ThisreportcoversprogressonCoreAssessmentforacademicyear2015-2016.AdescriptionofthecoreisincludedinAppendixA.TheCoreCouncilisastandingcommitteeoftheFacultySenateandhasthechargeofoverseeingcoreassessment(seeAppendixBforthecharge).Workoncoreassessmentstartedin2014-2015(AppendixC).CoreAreaAssessmentcommitteeswereestablishedforeachcurricularareaandonememberoftheCoreCouncilwasappointedtoserveasaliaisonandserveasthecommunicationlinkbetweentheCAACandCoreCouncil.(SeeAppendixD)During2015-2016eachcurricularareawasaskedtopilotrubricsdevelopedinSpring2015forassessingtheirKnowledgegoal(s).RhetoricandWritingandSpeechcommunicationwereassignedthefirstskillsgoal.Thepurposeofthepilotwastotesttherubricsandtodevelopprocessesthatcouldbefollowedforfutureassessments.SeeAppendixEfortheinstructionstoeachCAACoutliningthisprocess.EachCAACwasaskedtosubmitareportdescribingthepilotresults.RepresentativesofeachCAACmetwiththeCoreCouncilinSpring2016andfollowingthosemeetingseachliaisonwroteabriefreportansweringcommonquestionstofacilitatecomparisonsacrossreports.ThesereportsalsoincludecourseenrollmentsforSpring,Summer,andFall2015.TheliaisonsreportsareincludedinAppendixFandtheCAACreportsareincludedinAppendixG.ThetimelineforfutureassessmentisincludedinAppendixH.
WhatWeLearnedAboutCoreAssessmentin2015-2016
Overall:Aremarkableamountofprogresshasoccurredoverthecourseofa2-yearperiod.ThecoreassessmentprocessisfacultyledandfacultydrivenwhichisinlinewiththechargefromtheFacultySenate.OurcoreassessmentprocessisconsistentwiththeHigherLearningCommission’semphasison“acultureofcontinuousimprovement”
2
Rubrics-Allgroupsreportedsuccessusingtherubricstoscorestudentwork.Mathmentionedmodifyingthewordingoftherubrictoaidinscoring.Speechmentionedaddingmoredetailtotherubrictofacilitatebetteridentificationofstudentstrengthsandweaknesses.Allgroupswereabletosuccessfullyrateusingthe4-pointscale(however,notecommentunderissuesbelow)Participation:Themajorityoftheprogramsparticipatedandaremakingsteadyprogressonassessment.Someprogramschosenottoparticipate(Music)andsomeprogramsparticipatedminimally(someoftheSocialSciences).PartofthechargeoftheCoreCouncilistodevelopare-approvalprocessforcorecourses.Oneofthecriteriaforre-approvalwillbesatisfactoryparticipationinthecoreassessmentprocess.Selectionofstudentwork:TheCoreCouncilrecommendedthatsectionsofcorecoursesberandomlyselectedforassessmentandthattheselectionoccurafterthestartofthesemester.Thiswastoensurethattherewasno“teachingtothetest.”Thisworkedbetterforsomegroupsthanothers.Severalgroupsusedrandomnumbergeneratorstoselectsections.However,othergroupsdidnotselectrandomly.Insomecasesitwaschallengingtogetinstructorsunfamiliarwithassessmenttosubmitartifactsthatcouldbescoredusingtherubrics.Severalgroupsexpressedconcernsoverhowthiswouldworkwithconcurrentcoursesandadjuncts.Therearealsocasesinwhichthereisonlyonesectionofacourse.TheCoreCouncilcontinuestorecommendthatsectionsofcorecoursesberandomlyselectedforassessmentandthattheselectionoccurafterthestartofthesemester.Raters:CAACsreportedawidevarietyinthetypesofindividualswhoratedthestudentworkincludingadjuncts,graduatestudents,andfull-timefaculty.Inmostcases,exceptforCAACswithonlyoneprograminthecurricular,arearaterswereoutsidethediscipline.TheCoreCouncilrequestedthattheraterscomefromoutsidethediscipline,whenpossible,basedontheassumptionthatiftherubricsandassignmentinstructionswerewellwrittenthenarateroutsidethedisciplinecouldobjectivelyratethestudentwork.CompensationforRaters:Somegroupswereabletopayasmallstipendtoratersandothersrecommendedit.SeerecommendationsbelowunderCompensationandFacultyRole.Reliability-Reliabilityofratersisnecessaryforconfidenceintheassessmentresults.Many,butnotallcurricularareas,reportedreliability.Reliabilityneedstobeapriorityinthenextassessmentcycle.
3
Typesofassignments-CommonrubricscanbeappliedtomanytypesofassignmentsandtheCAACsputthattothetest.Multiplechoice,fillintheblank,labreports,papers,problemquestions,finalessays,electronicportfoliowithmultipleassignments,andvideorecordingswereallused.Themainissuewasthealignmentoftheassignmentwiththelearningobjectivesandnotwhethertherubriccouldbeusedwithawidevarietyofassignments.Timeestimates-Timeestimatesrangedfrom2minutesto15minutestoscoretheworkofonestudent.Theestimatesdependedonthetypeofstudentwork.TheCoreCouncilanticipatesthatwithexperiencetheseestimateswilldecline,butwewanttocontinuetotrackthesenumberssowehavearoughideaoffacultytimerequired.
IssuesthatNeedFurtherAttentionConcurrentcourses:Someprogramshavecloserelationshipswiththeirconcurrentinstructorswhileothersdonot.Oneissuethatwefaceishowtomakesureconcurrentinstructorsparticipateincoreassessmentandincludetherelevantlearningobjectivesintheirsyllabi.Adjuncts:Whilesomeprogramsincludedadjunctsintheassessmentpilotothersdidnot.Inaddition,fewtonoonlinesectionswereincluded.Allmodalitieswillneedtobeincludedinthefuture.Assessmentmanagementsystem:Currentlywedonothaveanelectronicassessmentsystem.Withoutasystemissuessuchasstorage,accesstostudentwork,andrubricaggregationwillneedtobeaddressed.Rubricscale:Allrubricsarecurrentlybasedona1to4scale.Weneedtoclarifythedifferencebetweenstudentworkthatdoesnotreachthethresholdforaratingof1andworkthatdoes.Includingbothinthesamecategoryconfoundsthecomparisonswiththeotherpointsonthescale.Addinganotscorablecategorycouldsolvethisproblem.Samplesize-BoththeHumanitiesandFineArtswantdirectiononthisissue.Theansweristhatnecessarysamplesizeisdirectlydependentonthequestionsthatareasked.Lowersamplesizesareacceptableifmeasuresaremorereliable.Seerecommendationsbelow.Studentwork-Findinganassignmentorassignmentsthatalignedwiththelearningobjectiveswaschallengingforsomeareas.Manygroupsareconsideringcommonassignmentsforthefuture.Anotherpotentialsolutionisforinstructorstobemoreintentionalinconnectingassignmentstolearningobjectives.Inthefutureallgroupswillneedtohaveassignmentsthataddressthelearningobjectives.Thesedonothavetobecommonassignments.Humanitiesstatedthat
4
withoutacomprehensivefinalassignmentthelearningobjectivescouldnotbeassessedandthisproblemwillneedtobeaddressedinthefuture.Itisnecessarytoassessthelearningoutcomesandobjectives.Gradesversusassessment:Someconfusionoverdifferenceingradesversusassessment.Gradesandstudentlearningoutcomesassessmenthavedifferentpurposesandfuturefacultydevelopmentmayneedtoaddressthedifferencesbetweengradesandassessment.Pre-posttestmethodology:TwocurricularareasBiologyandHistorymentionedthevalueofpreandposttests.Thismethodologycouldprovidevaluableinformationaboutthechangethatdoesordoesnottakeplaceinthesemester.Itcanalsoprovideinformationaboutthereliablyofmeasuresthatcanbeusedtomakeestimatesaboutneededsamplesize.
RecommendationsfortheFuture
Facultydevelopment:Futuresessionsonfacultydevelopmentneedtocovertherelationshipsbetweenassignments,rubrics,andlearningobjectives;thedifferencebetweengradesandassessment,Inaddition,theCoreCouncilrecommendsafacultydevelopmentsessionaddressingtheissuesofreliability,samplesize,rubricconstructionetc.Thereareseveralwebresourcesthatarerelevantandwecouldtapintocampusexpertiseforthesession.Alternatively,wecouldadoptmetricthatiscommonlyusedwhichis20%ofthetotalsample.Theadvantageofthismetricisitgivesanimmediateanswerandthedisadvantageisthatthenumbercanbehigh.CompensationandFacultyRole:TheCoreCouncil,respondinginparttoquestionsbymembersoftheCAACs,draftedlegislationfortheFacultySenatethataddressedtheissueofhowassessmentdataisused.Thelegislationstatedthatprogrammaticstudentoutcomesassessmentdatawasnottobeusedinevaluatingindividualfacultyforpromotion,tenure,orannualreview.However,participationinassessmentactivitiescouldbeusedinevaluationindividualfacultyforpromotion,tenure,orannualreview.Thechancellorvetoedthislegislation,however,heencouragedustocontinuetoworkontheissueandthatworkwillcontinuenextyear.Thislegislationrecognizedthatparticipationinassessmentispartwhatisexpectedoffull-timefaculty.TheCoreCouncilrecognizesthatcompensationmightbeappropriateincasesinwhichadjuncts,andgraduatestudentsserveasratersofstudentwork.Smallamountsofcompensationmightalsobeappropriateforfull-timefacultywhohaveaheavyassessmentworkloadorareparticipatinginextratrainingorfacultydevelopment.However,itisimportantthatfacultyowntheassessmentprocess.Whateversystemofcollecting,analyzing,andimplementing
5
assessmentdataisadoptedineachcoreprogramistheresponsibilityoftheentirefaculty.
6
AppendixA
DescriptionofUALRCoreImplementedJanuary20159curricularareasScienceCommunication-WrittenFineArtsSocialSciencesUSHistory/GovernmentHistoryofCivilizationHumanitiesMathematicsCommunication-SpeechThecoreaddressesthefollowingeducationalgoals:Skills1. Communication(oral,written,visual,secondlanguage;professionalself- presentation).2. Criticalthinking,quantitativereasoning,andsolvingproblemsindividually andcollaboratively.3. Informationtechnology(locating,retrieving,evaluating,synthesizing).Knowledge1. Theconcepts,methodologies,findings,andapplicationsofmathematics andthesocialandnaturalsciences,engineeringandtechnology.2. Theconcepts,methodologies,andtheglobalculturalheritageoftheartsand humanities.Values1. Ethicalbehaviorandpersonalresponsibility.2. Civicresponsibility.3. Globalandculturalunderstanding
7
MapofEducationalGoalsRepresentedintheUALRCore
CurricularAreas
EducationalGoals
Communication-Written
Communication-Speech Math
FineArts
Human-ities Science
SocialSciences
HistoryofCivilization
USTraditions
Knowledge1:Math&Science
X X X x x
Knowledge2:Arts&Humanities
X X x
Skill1:Communication
X X X x x x x x x
Skill2:CriticalThinking
X X X x x x x x x
Skill3:InfoTechnology
X X X x x x x x x
Value1:Ethical
X X X x x x x x x
Value2:CivicResponsibility
X X x x x x x x
Value3:Global&Cultural
X x x x
8
AppendixBChargeoftheCoreCouncil
Leg-FS-2012-13_11CouncilonCorePoliciesAmendedMarch11,2016TheUALRCouncilonCoreCurriculumandPolicies (henceforth'Council')astanding committeeofthe FacultySenate,will beresponsiblefor:
1.Maintainingthe criteriafor determiningwhetheracourseisappropriateto satisfy acurricularareainthe UALRStandard Core;2.Maintainingasubmission andapproval processfor UALRStandardCorecourse inclusion;3.Developingandmaintainingare-approvalprocess,includingatimeline,for UALRStandardCorecoursecontinuation;4.Maintainingaplan to assesseducationaloutcomesofthe UALRStandardCore;5.Overseeing college/programcourse inclusionbeyond the UALRStandard Coretoensure thecourseworkmeets the StateMinimumCore;6.Providingadetailed,written reviewto submittersfor alldecisions;7.Grantingexemptionsto the UALRStandard Corerequirementstoprograms, whereappropriate,asoutlinedinpart C.8.Developingapplicationandexemptionformsreflectingtheseguidelines.
9
AppendixC
ProgressonCoreAssessmentforAcademicYear2014-2015andFall2015
Fall2014.Thegeneralplanofusingcommonrubricswithincurricularareaswasadopted.Eachcurricularareahascommoneducationalgoalsandlearningoutcomesforthosegoalsandrubricswillbebasedonthelearningoutcomes.Thelearningobjectivesforaparticularcoursewithinacurricularareavary.Spring2015.Eachcurricularareaformedacommitteewitharepresentativefromeachdepartmentwithacorecourseinthatcurriculararea.TheCAACs(CoreCurricularAssessmentCommittees)includedamemberoftheCoreCounciltoserveasaliaison.Thefirsttaskwastocompleteasyllabusanalysisofallcorecoursestodeterminehowwhethercoursesyllabicoveredtherelevantlearninggoalsandobjectives.Inaddition,theCAACsweretodevelopaplanforcollectingstudentartifactsacrossavarietyofassignmentsandtodesignacommonrubricbasedonthecommonlearningoutcomestoassessthem.Thisprocessproducedthefirstassessmentoutcomes.Everygroupdiscoveredthatmostsyllabididnotincludetherelevantlearningoutcomesandobjectives.TheCoreCouncilrequestedthatinthefutureallcorecoursesincludethelearningoutcomesandobjectives.Fall2015.TheCAACswereaskedtodevelopaplanforhowtocollectdataandtocollectpilotdata.EachareawasaskedtosubmitareportdueFebruary19,2016reportingbothontheprocessofdatacollectionandonhowwelltheirrubricworked.
10
CoreCouncil
LiasionsHistoryofCivilization USTraditions SocialScience Math
CommunicationWritten
CommunicationSpoken Hummanities FineArts
CAACs History History Chemistry CriminalJustice MathRhetoricandWriting Speech English MassCommunication
PoliticalScience Physics Sociology Philosophy ArtAnthropology GenderStudies Religion MusicBiology Anthropology TheatreEarthScience PsychologyAstronomy Geography
PoliticalScience
Science
Appendix D 11
AppendixE
OutlineofCoreAssessmentPlanfor2015-2016
1.LearningObjectivesAllcorecourseswillincludethelearningobjectivessubmittedtotheCoreCouncilintheirsyllabi.Syllabiwillalsoincludethegoalsandoutcomesoralinktothem.2.Pilot
Lastspring(2015)allareassubmittedarubricorbeganworkingonarubric.Thecompletedrubricisnecessaryforthisassessmentprocess.Therubricassessedtheknowledgegoal1or2andaskillsgoalinthecaseofSpeechandRhetoricandWriting.
Eachcurricularareawilldevisearandomizedwaytoselectasectionofeachcourseinthecore.Theirmethodofselectionwillbecoveredintherequiredreport.
Theinstructorsintheselectedsectionswillprovide3studentartifacts,onehigh(Awork),medium(CWork),andlow(Fwork)fromanassignmentthatmatchesthegoalandoutcomesassessedbytherubric.Weanticipatethatthiswilloccurtowardtheendofthesemester.
Theinstructorfromthesectionsendstotheassessors(membersoftheCAAC)theworkofallstudentsinthatsectiononthatassignmentdividedinto3categories:high(Awork),medium(Cwork),andlow(Fwork).
Assessorsrandomlypickonefromeachcategoryandscore.WiththeexceptionofMath,Speech,andRhetoricandWritingassessorswillnotscoretheartifactsfromtheirowndiscipline.Inallcases,assessorswillnotscoreartifactsfromtheirowncourses.Thismeansthatthedisciplinerepresentativewillneedtoprovideadetailedscoringkeyforthecontentoftheassignment.
Atthispointwearetestingtherubricandtheprocessofassessment,nothowwellthestudentsdid.
Forreliabilitytwopeoplewillscoreeachartifact.Percentagreementwillbecalculated.Thereportwillcoverreliabilityandhowdisagreementswereresolved.
Thisisaniterativeprocess.
Iftherubricischangedthentheassessorscangobacktothestudentworkthatwasnotscoredandtestthenewrubric.
12
Continuousimprovementmeansitneverends.Thesecondroundofassessmentbeginsinthespringwiththeimmediategoaloftestinganychangesandthelong-termgoalofassessmenttolookatstudentlearningbeginninginFall2016.
c.Timeline.
Fall2015:Eachcurriculararea:Planhowtocollectdata,includingawaytorandomlysectionwhichsectionsofcoursestoincludeiftherearemultiplesectionsdevisingandfine-tuningtherubric,collectingartifacts,andscoringdata.Scoringthedatamaytakeplaceinearlyspringifartifactsarecollectedattheendofthefallsemester.
Reportdue:February19,2016
Thenrevisetherubricinthespringsemester.
4.Concurrentsectionsofcorecourses
TheCoreCouncilhasrequestedcopiesofsyllabifromconcurrentcoursesfromtheProvost’soffice.Copieswillbeprovidedtotherelevantdepartments.
GuidelinesforSpring2016CAACReportsDue:February19,2016SubmitreporttotheCoreCouncilLiaisonIncludeinthereports:1.DatainthespreadsheetprovidedbytheCoreCouncil.Pleaseincludeanysuggestionsforimprovements.2.Describetheprocessfollowedtorandomlyselectsections.Includesuggestionsforimprovements.3.Howdoyouknowtherubricworked?Provideexamples.4.Describeproblemsandsolutionsfortheproblemswiththerubricscoringprocessandinterpretationsofthedata.5.Providetimeestimateofhowlongittakestoscoretheworkofonestudentforonelearningoutcome.
13
6.Explainhowthestudentworkthatwascollectedcoversthelearningobjectives,andassociatedlearningoutcomes.Pleaseincludeanyrecommendationsforfuturechanges.
14
