It’s All About The Data: Increasing Student Success Via Partnerships (Quality Matters) And Professional Development

S.SUZAN J. HARKNESS

Assistant Dean & Director Center for Academic Technology

University of the District of Columbia

Institutional Statistics Spring 2011

2

Student population – 5,667 (FTE – 3819)48% attending Flagship Institution44% attending Community College8% attending Law School

Faculty population (Fulltime) – 253 (Adjunct)- 262

Courses per semester – 1,700 (traditional)

Helpdesk Support – 24-7-365 + on-campus

Center for Academic Technology – 7 FT Staff

This Session Will Cover:

3

How the University of the District of Columbia used astrategic triangulated approach (research anddata, professional development initiatives, andpartnerships) to impact:

Blackboard use across campus;

Faculty capacity to expand functional use and applications;

A quality approach to distance education and;

Student outcomes and success.

Three Drivers

• Strategic Planning

• Needs Assessment

• Bench Marking- early online experiences

• 360 Assessment to understand capacity and motivation

• Professional Development – data driven

• Data - Content Analysis (2010, 2011, 2012)

• Design and Findings– Targeted Training

• Partnerships

• Quality Matters, Blackboard Learn/Analytics

4

Strategic Planning

5

•Needs Assessment•Gap Analysis - Survey Faculty skills, Needs,

Motivations

•360 Assessment Capacity

•Benchmark early online experiences

•Content Analysis- Design and implement

Nominal in scale/binary

Pretest Spring 2010

Posttest Spring 2011

Posttest II Spring 2012

•Data Driven Professional Development

•Skills Workshops

•Online Instruction Certification (OLA & Track II)

•Continuous Assessment

F09 S10 F10 S11 F11 S12Totals

Total Workshops

25 24 55 36 50 47 237

Total Hours 53 149 151 127.5 118.5 79.5 678.5

Webcasts 2 0 0 0 1 2 5

Gross seat occupancy

147 313 356 180 176 257 1429

Unique Faculty attending

91 64 181 105 79 125 403

Cost per faculty/term

$319.00 $163.00 $65.00 $77.00 $157.00 $62.00 $197.00/faculty

Professional Development

Assessment Feedback

7

•83% response rate on faculty assessments

• 89% concluded that professional development workshops were effective in providing tools and applications to assist them

•100% reported that would deploy the new application(s)

•100% reported that the workshops were a valuable use of time

•77% reported that the workshops were excellent training

•88% reported that the workshop instruction was excellent

•89% felt that the trainings provided them sufficient comfort and experiences to use the new applications

Content Analysis

2010 Examination

Banners Buttons Color Scheme Announcements Course Info Syllabus Faculty Info Course

docs/content Assignments Discussion

Boards External Links Additional Tools

2011 Examination

Banners

Buttons

Color Scheme

Announcements

Course Info

Syllabus

Faculty Info

Course docs/content

Assignments

Discussion Boards

External Links

Additional Tools

Grade center

Groups

Additional Resources

2012 Examination

Banners

Buttons

Color Scheme

Announcements

Course Info

Syllabus

Faculty Info

Course docs/content

Assignments

Discussion Boards

External Links

Additional Tools

Grade center

Groups

Additional Resources

S2010 S2011 Percent Change S2012 Percent Change

Total Users 209(58%)

270(55%)

+ 29 % users -5 proportion of

users

225 (47%)

-17% users-15 %

proportion of users

Heavy Users

131 (35%) 180 (37%) 6% 141 (29%) -22%

Moderate Users

78 (21%) 90 (18%) -14% 84 (18%) null

Non-Users 153 (41%) 223 (45%) 10% 253 (53%) 18%

TotalFaculty

FT Adjunct

362

241121

493

231262

36%

-4%117%

478

182296

-3%

-21%13%

Blackboard Use Among Faculty

11

Let’s Drill Down

12

•Bimodal distribution –tells us where we need to focus

•11% of faculty who attended workshops shifted their categorization (2010 to 2011)

•3% shifted from a nonuser to a moderate user (2010-2011)

•4% shifted from a nonuser to a heavy user (2010-2011)

•4% shifted from a moderate to a heavy user (2010-2011)

•<1% shifted from a minimum user to a moderate user (2010-2011)

2012 & 2013: New Challenges New Opportunities

QUALITY MATTERS

• Beyond the pilot - 17 Courses Peer Reviewed

• Mean QM score 74.0 (met QM standards).

• 73 percent increase over QMR of courses pre QM

• Standard deviation 7.05 (Increased rater reliability)

• More importantly – Positive Student impact

• 1 Master Reviewer and a Train the Trainer on staff

BLACKBOARD: Learn, Analytics, Support Services

• Partner approach to support and services

• Analytics – data mining

Partnerships: QM, Bb Learn, Bb Analytics, Bb Student Services

Descriptive Overview of Online Courses 2007-2012 Pre/Post QM Partnership

14

Statistical Impact 2007-2012

Pre/Post QM

(N=1570)

% Change

WDL -23.53

PASS Class +19.74

FAIL CLASS -66.66

15

Percent Change Pre/Post QM 2007-2012

Statistical Impact 2011-2012

16

2011-2012 Data

points – Traditional

v online/hybrid

courses

Percent

All Classes

(N=5716)

Percent QM

Online/Hybri

d Classes

(n=935)

Percent

Difference

Traditional v

Online

WDL .071 .13 58.7

PASS Class .80 .91 12.9

FAIL CLASS .20 .09 75.9

Percent Difference: Traditional: Online

Lessons Learned

17

• Faculty who engaged in PD increased and expanded Blackboard use• Upward drift of expanded use and engagement • Triangulated data-driven approach yields better student experiences• Fewer students WDL from online courses that meet QM standards• Students in peer-reviewed courses earn higher grades/more likely to

pass courses• New policies and procedures ensure consistency • Quality and integrity over quantity – instrumentalism • Collegiality and collaboration sustain our strategic plan, drive

institutional effectiveness, and expand capacity across campus

18

We value your feedback!

Please fill out a session evaluation.