Using Measurable Outcomes to Evaluate Tutor Programs

Jan Norton, Presenter

IntroductionJan NortonEmail: snowfanjan@gmail.comPhone: (319) 499-1372Master’s in Educational Research& Psychology

25+ years as director of multi-discipline tutoring centers with CRLA certifications

Former reviewer for CRLA tutor training certification; current NADE certification reviewer for tutoring, SI, and developmental education programs

NCLCA Lifetime Learning Center Leadership certification

Webinar ContentsTerminologyThe Black BoxTimetablesData/MeasurementData/Measurement CautionsProgram Outcomes & ExamplesStudent Outcomes & ExamplesClosing

TerminologyVisionsMissionsValuesGoalsObjectivesOutcomesProgram vs. Student

OutcomesLearning Outcomes

The Black Box

Something

Happens

The Black Box

Something

Happens

What is going on in there? How do we measure it?

TimetablesScope of assessment

◦≥ 6 years◦2 years◦1 year◦1 semester◦1 course◦1 assignment◦1 tutoring session

TimetablesPlanning for Assessment

◦Continual processes◦Multiple times per

semester◦Once or twice per

semester◦Two or three times per year◦Once per year◦Every two years◦Every 4 years

Data/MeasurementWhy Measure

What to measure◦Before (Benchmarks)◦During◦After (Outcomes)

Data/MeasurementHow to measure

◦Program-generated data◦Course-specific data◦Campus data◦Campus surveys◦Standardized tools◦Locally developed tools

Data/MeasurementMore about locally developed

tools◦Surveys◦Likert-type scales◦Explicated scales◦Behavioral rubrics◦Self assessment by student◦Tutor assessment◦Observer assessment

Data/Measurement Cautions

Direct vs. Indirect MeasuresData access and consistencyRealistic limitationsCooperationThe Baseline ChallengeTime & money

(and staff and comfort and ……)

Chat Time for Questions & Contributions

Finished the foundation: what we can and can’t measure, when & how to measure, and what data we will gain

Moving on to examples of program-level outcomes

Program OutcomesPossible Areas of Assessment

Focus◦Establishing baselines◦Usage◦Satisfaction or quality◦Level/variety of services◦Staff development◦Reaching certification or meeting

standards

Program Outcomes ExamplesOutcome Statement

◦As a result of the new marketing strategy, the number of students attending group tutoring sessions will increase by 10% in Spring 2013.

Timetable◦One semester

Starting Measure (Benchmark)◦Average of past attendance

Ending Measure ◦Workshop attendance records

Program Outcomes ExamplesOutcome Statement

◦During 2012-13, the number of tutors who earn CRLA Level I certification will increase by 5%.

Timetable◦One academic year

Starting Measure (Benchmark)◦Number of tutors achieving Level I in

previous year(s)Ending Measure

◦Number of tutors achieving Level I in 2012-13

Program Outcomes ExamplesOutcome Statement

◦Outside reviewers using the NADE self-study process will score the tutoring program at 85% or above on all sections.

Timetable◦One year

Starting Measure (Benchmark)◦ Internal review & self-study

Ending Measure◦External review

Program Outcomes ExamplesOutcome Statement

◦As a result of tutor training sessions, tutors will demonstrate an improved understanding of Bloom’s Taxonomy.

Timetable◦One semester or year, depending on

training scheduleStarting Measure (Benchmark)

◦Pre-testEnding Measure

◦Post-test

Chat Time for Questions & Contributions

Finished program-level outcomes

Moving on to examples of student outcomes

Student Outcomes: MeasuresQualitative

◦Interviews & focus groups◦Opinion surveys◦Observations or self reports

Quantitative◦Assignment & course grades◦Instrument scores◦Observation rubrics

Student Outcomes: TypesAffective

◦Build confidence◦Improve attitude◦Reduce anxiety

Cognitive◦Mastery of material◦Conscious strategies use

Student Outcomes ExamplesOutcome Statement

◦After being tutored for at least one hour, clients will be more confident in their ability to accomplish the academic task they focused on in tutoring.

Timetable◦One tutoring session

Starting Measure (Benchmark)◦Client self-report

Ending Measure ◦Client self report

Student Outcomes ExamplesOutcome Statement

◦ Students who work with the study skills tutors will demonstrate an increased ability to use memorization strategies.

Timetable◦One tutoring session

Starting Measure (Benchmark)◦Tutor or observer checklist

Ending Measure◦Tutor or observer checklist

Student Outcomes ExamplesOutcome Statement

◦75% of the students who work with a reading tutor for at least two sessions will score at least 75% on the main ideas reading test.

Timetable◦Within three weeks

Starting Measure (Benchmark)◦Reading quiz at start of tutoring

Ending Measure◦Chapter reading test in class

Student Outcomes ExamplesOutcome Statement

◦Students who attend at least 10 group tutoring sessions for Algebra Review 090 will earn an average grade of B or above in the course.

Timetable◦One semester, length of course

Starting Measure (Benchmark)◦Placement score into course

Ending Measure◦Course grade

Student Outcomes ExamplesOutcome Statement

◦After attending mandatory tutoring for all 5 of their developmental education courses, students will be retained until the start of their junior year at a rate at least 25% higher than developmental students for whom tutoring was not required.

Timetable◦Two years

Starting Measure (Benchmark)◦Developmental course placement

Ending Measure◦Enrollment level

Student Outcomes ExamplesOutcome Statement

◦Students attending SI for World Religions will “[apply] previously understood information, concepts, and experiences to a new situation or setting.” (CAS)

Timetable◦Within one semester

Starting Measure (Benchmark)◦Pre-survey

Ending Measure ◦Post-survey

ClosingCheck with your institutional

research or assessment officeReview early drafts of

outcomes statements with your supervisor

Pre-test any locally created measurement tool

Be prepared for outcomes that are not met or wildly exceeded

Closing

Remember the black box: we do not own or control the box, and our ability to see into it is very limited

Questions & Contributions?

Thank You for Participating !

Jan Norton

snowfanjan@gmail.com(319) 499-1372