Enterprise-Based E-Learning Planning and Evaluation in a

Large-Scale Operation

© Copyright Nicola Martinez and Susan Oaks 2003. This work is the intellectual property of the authors. Permission is granted for this material to be shared for non-commercial, educational purposes, provided that this copyright statement appears on the reproduced materials and notice is given that the copying is by permission of the authors. To disseminate otherwise or to republish requires written permission from the authors.

Presenters

Susan Oaks, Ph.D., Academic Area Coordinator, Writing, Literature and Communications; Chair, Curriculum Committee

Nicola Martinez, MA, Coordinator of Instructional Design and Curriculum Development

Center for Distance Learning SUNY Empire State CollegeSaratoga Springs, New York

Objectives

Demonstrate an integrated planning & tracking database for course development, review, and evaluation

Discuss context/issues leading to database Discuss collaborative approach—and multiple

uses of database—by faculty, instructional designers, term prep staff

Focus on quality assurance and applicability to other institutions

Who We Are: Empire State College

Locations around New York State

The Center for Distance Learning (CDL) serves adult students around the world

Mission Statement

Empire State College is dedicated to:

enabling motivated adults regardless of geography or life circumstance to manage and master a rigorous academic program and earn a degree

Areas of Study

Areas of Study:

The Arts | Business, Management & Economics | Community & Human Services | Communications, Humanities & Cultural Studies | Educational Studies | Historical Studies | Human Development | Labor Studies | Science, Math & Technology | Social Theory, Structure & Change

Center for Distance Learning

offers asynchronous online courses in five, 15-week semesters per yearleading enroller in the SUNY Learning Networkpartner with several institutions such as eArmyU & Navy, public & private organizations, labor unions

Center for Distance Learning Course Statistics

approximately 5,000 students annually200 distinct online coursesmore than 350 course sectionsover 6000 enrollments per term46 courses developed for fall web delivery50+ courses in development for Jan 04existing courses scheduled for cyclical delivery

Context: Rapid Online Growth

Increase in f/t faculty from 8 to 15 (and growing) within past two years

Hire two Coordinators of Instructional Design and Curriculum Development to shepherd faculty developers through course development process and coordinate instructional support

Move from online business degrees to online degrees in many areas of study

Move all existing courses online (over 200)

Courses in development

Some courses in development this term:

Artistic Expression in a Multicultural AmericaIntroduction to Visual LiteracyDance Across World CulturesFood and Drink in Cultural ContextReflective LearningBiology of the BrainCognitive PsychologyMinds and MachinesUrban StudiesChemistry in Context

Previous Course Development System

Elected Curriculum committee (4 faculty) + 1 instructional designer

All faculty review & comment on course proposals Curriculum committee review, vote, and respond E-mail records of issues/outcomes

Problems with Previous System

Not representative of all broad disciplinary areas Sparse review & feedback on proposals E-mail records time consuming; inconsistent

depending on committee chair Course development relates to other systems

(instructor training, book ordering, quality evaluation); maintenance of multiple, discrete systems

New System: CourseTrak

A Collaboration Between:

The Center for Distance LearningDean: Meg Benke

The Center for Learning and TechnologyDirector: Evelyn Ting

The Office of Educational TechnologyVice President: Patricia Lefor

New System: CourseTrak

Integrates information for all players: instructional designers, faculty, support staff

Creates a public record from course proposal through review and offerings

Fosters collaboration through its public nature Facilitates evaluation & review

CourseTrak: Faculty Perspective

Reminds course developer/Area Coordinator of required information,objectives, content and learning activities

Requires rationale for new courses in terms of academic skills, student population, etc.

Invites other faculty collaboration Facilitates quality review process Enlists staff help with materials ordering,

scheduling, etc.—integrates systems

CourseTrak: Instructional Design Perspective

For large scale development of pedagogically and academically sound courses infused with:

Active learningMedia-rich contentStudent interactionOpportunities for Teaching, Social and Cognitive

Presence

Constraints

WHILE EXPERIENCING A RAPID VOLUME INCREASE IN ENROLLMENTS, COURSES, AREA COORDINATORS, AND ADJUNCT FACULTY.

Increased volume of courses developed per term:

from 5-8 for Fall 02to 46 for Fall 2003to 50 + for Spring 2004

Our Approach

Develop a new standard for course information documents and implement them in the courses

Create a new design for CDL web courses to increase navigability and overall consistencies in look, feel, and format

Enrich courses with multimedia components and other graphic enhancements

Include library referenced materials in all courses, with at least one library-based learning activity per course

Identify and implement best practices in the pursuit of academic and pedagogical excellence in web course development for adult learners taking distance learning courses

A Consultative Approach

Strategies for the development and achievement of these objectives included consulting with:

Curriculum Committee CDL Faculty The Curriculum Development Team The L-Guide Focus Group The Center for Learning and Technology Library

And Multimedia Instructional Design Staff.

Course Ratings

We established a three-tiered categorization of courses based on the inclusion of the above recommendations and the application of a new course design.

All new and revised courses are reviewed an assigned the following ratings:

Course Rating SystemRating Legend

   

Tier 3 Exemplary course

Tier 2 Acceptable (some criteria not met, revisions needed)

Tier 1 Not Acceptable

NR Not Rated

Course Rating System

Tier information and classification is generated as a result of the information captured in The Empire State College Online Course Evaluation Checklist.

All courses are considered unrated until they have been reviewed with this instrument.

The evaluation piece associated with The Empire State College Online Course Evaluation Checklist happens at several levels.

Online Course Evaluation Checklist

Administered during and after a new course has been developed, before delivery

Before an old course is revised to identify specific areas targeted for improvement

After an old course has been revised to ensure that it is complete and meets all baseline criteria for a Tier 3 course

This will also be used, in the future, to evaluate instruction

The Course Resource Needs Analysis

is used to establish the resources, support, services, and document creation needed for a course in development or revision.

An instructional designer conducts the needs analysis in consultation with the course developer (s). 

The Course Resource Needs Analysis

Information captured in the Course Resource Needs Analysis Form is used to generate work requests for

technical assistance library assistance multimedia learning object creation digital media research and implementation and instructional design/pedagogical support, among other things.

 

Focus on Quality Assurance

Establish Baseline Criteria for Courses Evaluate and Review Courses Track Performance Capture Data Make Necessary Adjustments to Courses Create a Data Driven Analysis Develop a Report Make Recommendations Improve Process and Performance Using Results

Applicability to Other Institutions

This is an example of an in-house, web based system

designed to meet our course planning, revision, and evaluation needs

and tailored to the culture and goals of our institution

Applicability to Other Institutions

Points to Consider in the Development of such a system:

Level of Institutional Support# of faculty developing courses# of new faculty needing intense supportHow to integrate the system with other relevant

systemsAccess Control and Security IssuesAnalyze your assumptions

Contact Information

Nicola.Martinez@esc.edu

Susan.Oaks@esc.edu

Empire State College: www.esc.edu

Center for Distance Learning: www.esc.edu/cdl