CISEREUEvaluationToolkitAudrey Rorrer, PhD

University of North Carolina at CharlotteCollege of Computing & Informatics

audrey.rorrer@uncc.edu

NSFCISEREUNewPIMeeting,Arlington,VA,March2017

Evaluation&AssessmentConsiderations

NSFCISEREUNewPIMeeting,Arlington,VA,March2017

ToolkitResources

Evaluation101

Recruiting

StudentOutcomes

Etc.

IntentionalEvaluation• Whatisthevision?

• Projectgoals• Studentimpact• Resources&support• Milestones

• Whatevidenceisneeded?

NSFCISEREUPIMeeting,Orlando,FL,March2017

Eval 101:REUProgramOutcomes

• Applicants• AppliedvsAccept• Demographics• Limitedresearchaccess

• Participants• Attitudes,Behaviors,Cognition(skills,knowledge)

• Experiences,Activities• Longitudinaloutcomes

• ProgramOutcomes• Publications• Dissemination• Products

• Specialty• Mentoring?• Graduatestudentdevelopment?

• OtherContributions

NSFCISEREUPIMeeting,Orlando,FL,March2017

RecruitingResources:CommonApplication

NSFCISEREUPIMeeting,Orlando,FL,March2017

• 56itemscoveringcontactinformation,academicpreparation,graduateschoolplans,demographicinformation

• Customizable(theme,items,nextstepsinformation)• Immediateaccesstoyourapplicants• Managesstudentdatainaform

• Notoolforreferences,resumes,ortranscripts

WhatPISees

NSFCISEREUNewPIMeeting,Arlington,VA,March2017

RecruitingResources:CommonApplicationWhatApplicant

Sees

1.AdvertiseonGoogleDocStudentsaredirectedtoyou2.SharedApplicantPoolYoucontactstudentsfromothersites

NSFCISEREUPIMeeting,Orlando,FL,March2017

RecruitingResources:ApplicantSharing

Fall:advertise

Winter:select

Spring:share

Results:CommonApplicationTrends

Site Descriptors2010

(N=13) 2011 (N=20)2012

(N=22)2013

(N= 26)2014

(N=25)2015

(N= 23)2016

(N= 27)

Range of # Applicants 29-152 4-176 18-212 20-299 29-232 15- 349 27- 286

Avg Applications per Site 77 79 93 112 102 122 108

Largest # of Sites Applied to by

Individual 30 (n=1) 6 (n=1) 7 (n=2) 18 (n=1) 14 (n=1) 16 (n=1) 9 (n=3)

% of Female Applicants 27% 26% 24% 28% 27% 26% 32%

% of URM Applicants 28% 34% 36% 31% 31% 35% 33%

NSFCISEREUNewPIMeeting,Arlington,VA,March2017

StudentOutcomes:AlaCarteStudentSurvey

NSFCISEREUPIMeeting,Orlando,FL,March2017

Post-TestOnlyPreandPost-Test

Self-Efficacy

GraduateSchoolIntent

Attitudes

Grit

Help-Seeking

ProfessionalIdentity

Leadership

ResearchSkills

Mentoring

ProgramSatisfaction

StudentOutcomes:HowSurveyWorks

NSFCISEREUNewPIMeeting,Arlington,VA,March2017

DeploymentOptions

ToolkitAdministered➕ Datacollectionandanalysisdone

foryou➕ Datapackage=yoursitedata(raw

andcoded),codebook,results➕ Anonymizeddata

PIAdministered• Sitecollectsandanalyzes• Identified

StudentOutcomes:DataPackage

• RawandCodeddata ofpreandpostresponses• MatchedPairsdata• CodeBook• Analysisofpre/postdata• Dataisde-identified

NSFCISEREUPIMeeting,Orlando,FL,March2017

Results:AlaCarte2016

Construct Pre (SD)N= 305

Post (SD)N= 268

Self-Efficacy* 3.67 (.61) 4.21 (.53)Intent to Grad School* 3.83 (.75) 3.7 (.88) Attitudes* 4.33 (.51) 4.09 (.74) Help-Seeking/Coping

4.27 (.61) 4.35 (.64)

Research Knowledge* 3.15 (.66) 3.92 (.59)

Grit 3.36 (.35) 3.39 (.40)

Scientific Identity* 3.42 (.91) 3.68 (1.0)

Leadership* 4.18 (.60) 4.30 (.65)

Mentorship Not applicable 4.38 (.84)

Note: 5 pt Likert scale

Significant Effects forTime(pre- andpost- surveys,*(p<.05)

Descriptives:• 43% Female • 35% multi-ethnic

minority groups

• Predominantly Rising Juniors and Seniors

NSFCISEREUNewPIMeeting,Arlington,VA,March2017

Etc.:AdditionalConsiderations

NSFCISEREUPIMeeting,Orlando,FL,March2017

IRB LongitudinalTracking

OutcomesofInterest

CampusResources

•EvaluationToolkit:reu.uncc.edu/cise-reu-toolkit•HowTovideos•EvaluationmaterialsandresourcestailoredtoCISEREU

OnlineEvaluationToolkit

•GoogleFormapplicationUNIQUEtosite•Standardized&customizable•AggregatedescriptivestatisticsacrossCISEdirectorate

CommonApplication

•ManagedviaGoogleDrive&CommonApplications•SitePI“releases”unselectedcandidates•AllPIshaveaccesstoonlinefolder

SharedApplicantPool

•StudentOutcomes:modulated,valid/reliable•FacultyImpact:Surveydeployingsummer2016

Surveys:AlaCarteand Faculty

•Underconstruction•JointheworkinggroupAlumniTracking

NSFCISEREUNewPIMeeting,Arlington,VA,March2017

reu.uncc.edu/cise-reu-toolkitThankyou

audrey.rorrer@uncc.edu

Appendices

NSFCISEREUNewPIMeeting,Arlington,VA,March2017

FutureToolkit:AlumniTracking

WorkingGroup

March2016

Discussion&Ideas

DevelopPrototypes

Feedback &Launch

WorkingGroupMembers:• Lazaros Gallos,Rutgers• ClaireDuggan,NortheasternU.• BonnieSwan,U.CentralFlorida• StephenGilbert,IowaState• TiffanyReardon,Berkeley• Huirong Fu,Oakland• JamiePayton,UNCCharlotte• RajaKushalnagar,RIT

NSFCISEREUNewPIMeeting,Arlington,VA,March2017

ApplicantCounts:SitesAppliedTo

=1006

=1105

=1934

=2905

=2445

=2827

=2335

NSFCISEREUNewPIMeeting,Arlington,VA,March2017

ApplicantDemographicTrends:UnderrepresentedMinorityDistribution

28%

34%

36%

31%

31%

35%

33%

NSFCISEREUNewPIMeeting,Arlington,VA,March2017

ApplicantDemographicTrends:GenderDistribution

27%

26%

24%

28%

27%

26%

32%

NSFCISEREUNewPIMeeting,Arlington,VA,March2017

CommonApplications:GraduateSchoolPlans

• Majorityplantopursuegraduatedegrees• 79%in2013• 80%in2014• 84%in2015• 79%in2016

• Fewarefirstgenerationcollegestudents• Lessthan20%acrossall6years• 17%arefirstpersoninfamilytoattendcollege(2015)• 19%arefirstpersoninfamilytoattendcollege(2016)

NSFCISEREUNewPIMeeting,Arlington,VA,March2017

2016CommonApplicants:Unique

NSFCISEREUNewPIMeeting,Arlington,VA,March2017

2016CommonApplicants:Consent

NSFCISEREUPIMeeting,Arlington,VA,March2017

• Percentageofapplicantswhoanswered'Yes'tohavetheirapplicationconsideredbyadditionalCISEREUprograms

¨ Percentageofapplicantswhoanswered'yes'tooptoutofparticipatingintheresearchstudy

AlaCarteStudentSurveyModular, pre/post assessment of student outcomes

• Self Efficacy – I can formulate a research problem

• Intent to attend graduate school – I plan to apply to graduate school in a

computing discipline• Attitudes towards computing

– I like to use computer science to solve problems

• Help seeking/coping skills – When I do poorly on an exam,

typically I….skip class• GRIT

• I am a hard worker

• Research Skills – Formulate a research hypothesis– Write a research paper for

publication• Leadership in Science

- I know how to be good team member

- I know how to encourage team performance

• Professional Identity as Scientist- I feel like I belong in science

• Mentoring Satisfaction (post-only)• My mentor was helpful in providing

direction and guidance on research project issues

NSFCISEREUNewPIMeeting,Arlington,VA,March2017

SurveyInstrumentsAvailable

• SURE- SurveyofUndergraduateResearchExperiences

• D.Lapatto atWashingtonUniversity

• URSSA- UndergraduateResearchStudentSelfAssessment

• UniversityofColoradoBoulder,NSF

• Botharefreeforresearchprogramsandcourses• AdaptationmadefortheAlaCarte

NSFCISEREUNewPIMeeting,Arlington,VA,March2017

HistoryofEvaluationProject2009 Working Group Members

Guy Alain Amousou Chris Aberson Wendy Cooper Teresa Dahlberg Andy Fagg Stephen Gilbert Manfred Huber Niels Lobo Sanjay Madrias Joan Peckham Eric Wong Yu-Dong Yoa

Kevin Zeng

NeedsAssessment&Study

EstablishedCommonIndicators&Tools

ResearchModuleInclusiveTerminologySharedApplicantPool

NewModulesFacultyStudyAlumniTracker

InstructionalVideos

2009Humbolt State

2010UNCCharlotte

20132014-16Toolkit

ExpansionProject

NSFCISEREUNewPIMeeting,Arlington,VA,March2017